questdb / kafka-questdb-connector

QuestDB connector for Kafka.
https://github.com/questdb/kafka-questdb-connector/releases/
Apache License 2.0
30 stars 5 forks source link

fix: records with data type mismatching goes to DLQ instead of failing the connector #27

Closed jerrinot closed 1 month ago

jerrinot commented 2 months ago

fixes #26 This is a first impl. It could be optimized further, but I assume bad data are rare and it already does the job as it is.

There is a potential issue with inflightSinkRecords referencing all in-flight records. In theory, this can +- double memory consumption. but we need the original SinkRecord so we can send it to DLQ.

This is implemented for the HTTP transport only, because the TCP transport cannot report issues properly.