confluentinc / kafka-connect-bigquery

A Kafka Connect BigQuery sink connector
Apache License 2.0
3 stars 2 forks source link

BQ Connector incorrectly parsing DBZ timestamps with millisecond precision #408

Open aakarshg opened 6 months ago

aakarshg commented 6 months ago

Here's the sample data from the postgres table:

database=> select created_time, last_modified_time from blah where <select_row>;
      created_time       |     last_modified_time
-------------------------+----------------------------
 2023-01-28 12:55:05.094 | 2023-08-06 00:36:48.435488
(1 row)

database=> select EXTRACT(EPOCH from created_time) as created_time, EXTRACT(EPOCH from last_modified_time) as last_modified_time from blah where <select_row>;
  created_time  | last_modified_time
----------------+--------------------
 1674910505.094 |  1691282208.435488
(1 row)

The same row however shows up in BQ as follows:

created_time | last_modified_time |  
-- | -- | --
1 | 2023-01-28 12:55:05.940000 UTC | 2023-08-06 00:36:48.435488 UTC

As we can see the created_time has the decimal right shifted by 1 digit, but last_modified_time remains unchanged.

Additional details:

BQ connector version 2.5.2