Today we have the option to create some kind of dead letter table for 'Kafka engine" tables using kafka_handle_error_mode=stream and routing messages per "_error" column.
On the other hand there is no equivalent solution to create on Materialized view in Clickhouse.
Materialized view can suffer from 'Parse Exception'.
You can used it with combination of materialized_views_ignore_errors=true in order to proceed with following messages but no way to check failed messages when doing so.
There are scenarios in which “DeadLetter” Table style solution is blessed.
In this kind of solution Users will be able to control 'materialized_views_ignore_errors' but will be able to add destination table for Error scenarios in which the raw message (prior for parsing) will be written.
Table structure can be generic:
Metadata - JSON that will holds the metadata for the message ( can be offset,partition etc in cases of source table is kafka engine) not sure which data is available in the writing to the materialized view phase.
RawData - String of Raw data
A per table metric for Error count is also a good option for monitoring purpose and should be controlled by "materialized_views_error_metric=true\false" flag on the view.
UnamedRus
commented
1 year ago
Today we have the option to create some kind of dead letter table for 'Kafka engine" tables using kafka_handle_error_mode=stream and routing messages per "_error" column.
On the other hand there is no equivalent solution to create on Materialized view in Clickhouse. Materialized view can suffer from 'Parse Exception'. You can used it with combination of materialized_views_ignore_errors=true in order to proceed with following messages but no way to check failed messages when doing so.
There are scenarios in which “DeadLetter” Table style solution is blessed.
In this kind of solution Users will be able to control 'materialized_views_ignore_errors' but will be able to add destination table for Error scenarios in which the raw message (prior for parsing) will be written.
Table structure can be generic: Metadata - JSON that will holds the metadata for the message ( can be offset,partition etc in cases of source table is kafka engine) not sure which data is available in the writing to the materialized view phase. RawData - String of Raw data
A per table metric for Error count is also a good option for monitoring purpose and should be controlled by "materialized_views_error_metric=true\false" flag on the view.