search

wepay / kafka-connect-bigquery

DEPRECATED. PLEASE USE https://github.com/confluentinc/kafka-connect-bigquery. A Kafka Connect BigQuery sink connector
Apache License 2.0
155 stars 192 forks source link

Convert io.debezium.time.MicroTimestamp to DATETIME #305

Open LarsKlingen opened 2 years ago

LarsKlingen commented 2 years ago

For legacy reasons I would like to have DATETIME formau for columns that come as MicroTimestamp (We use Debezium for postgressql that converts timestamps to MicroTimestamp). The Sink automatically converts to TIMESTAMP format. I tried creating the table first with type DATETIME for the columns to circumentvent this, but then I got the error:

Field updated_at has changed type from DATETIME to TIMESTAMP; See logs for more detail
    at com.wepay.kafka.connect.bigquery.write.batch.KCBQThreadPoolExecutor.maybeThrowEncounteredErrors(KCBQThreadPoolExecutor.java:108)
    at com.wepay.kafka.connect.bigquery.BigQuerySinkTask.put(BigQuerySinkTask.java:233)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:582)
    ... 10 more

Is there a way to save columns as DATETIME on BQ? I also considered using a view to convert to DATETIME, but then we lose the parittioning property on these columns. Thanks!

saumyasuhagiya commented 2 years ago

same question. @Onecricketeer

LarsKlingen commented 2 years ago

This repo is deprecated, please take a look here: https://github.com/confluentinc/kafka-connect-bigquery Opened the same issue there.