search

confluentinc / kafka-connect-bigquery

A Kafka Connect BigQuery sink connector
Apache License 2.0
3 stars 2 forks source link

Debezium micro-timestamps with negative epochs cause the connector to fail due to string formatting problem in the conversion code #401

Open james-johnston-thumbtack opened 7 months ago

james-johnston-thumbtack commented 7 months ago

This timestamp conversion code does not account for timestamps with negative epochs: https://github.com/confluentinc/kafka-connect-bigquery/blob/c9aedfb47b5fb61ec67404adc4f14cf05a51d82b/kcbq-connector/src/main/java/com/wepay/kafka/connect/bigquery/convert/logicaltype/DebeziumLogicalConverters.java#L134-L136

For example, we have seen the connector crash with an input timestamp value of -6795364578871000. The error is:

[row index 0] (location value.end_date, reason: invalid): Could not parse '1754-08-30 22:43:41.-871000' as a timestamp. Required format is YYYY-MM-DD HH:MM[:SS[.SSSSSS]]

As you can see, a negative fractional second is provided in a way that makes no sense and thus can't be parsed.

The expected date/time would be Friday, August 30, 1754 10:43:41.129 PM. So we should see the fractional second portion be .129000, not .-871000.