search

wepay / kafka-connect-bigquery

DEPRECATED. PLEASE USE https://github.com/confluentinc/kafka-connect-bigquery. A Kafka Connect BigQuery sink connector
Apache License 2.0
155 stars 192 forks source link

Support for configure BigQuery table partition type #169

Open duanshiqiang opened 5 years ago

duanshiqiang commented 5 years ago

The decision was made 2+ years ago to always use data-partitioned tables when letting kafka-connect-bigquery to manage BigQuery tables. (refer to https://github.com/wepay/kafka-connect-bigquery/issues/39#issuecomment-267175814)

But BigQuery now supports 2 different kinds of date-partitioned tables (the old partitioned by ingestion time type and the new field-based date-partitioned tables).

We are heavily using the field-based date-partitioned tables and currently how we integrate with kafka-connect-bigquery connector is that we manually create the tables required with field partition enabled and only using kafka-connect-bigquery to ingest data into them. But it is troublesome to do because we need to take care of schema update, etc.

It would be better to allow users to configure the table partition type. And I personally think that allowing users to disable partition totally should also be supported.

duanshiqiang commented 5 years ago

https://cloud.google.com/bigquery/docs/partitioned-tables

magiciiboy commented 5 years ago

@duanshiqiang totally agree and wait for this feature

darshanmehta10 commented 4 years ago

@duanshiqiang @magiciiboy Added a PR for this: https://github.com/wepay/kafka-connect-bigquery/pull/229

darrenhaken commented 4 years ago

any update on this?