search

GoogleCloudPlatform / k8s-config-connector

GCP Config Connector, a Kubernetes add-on for managing GCP resources
https://cloud.google.com/config-connector/docs/overview
Apache License 2.0
901 stars 235 forks source link

Add avroOptions and useLogicalTypes to Config Connector #582

Open mathiesonsterling opened 2 years ago

mathiesonsterling commented 2 years ago

Describe the feature or resource https://issuetracker.google.com/issues/210367196

When loading AVRO files into BigQuery, the useAvroLogicalTypes parameter must be set in order to load timestamps properly typed.

External tables for BigQuery can specify this. We have verified it works correctly when doing a load job (not external tables) with an avro file with logical types.

However the Config Connector does not allow this field when creating a config connector!

Importance This is a blocker to our company using AVRO as our loading mechanism (which we currently are).

If we cannot get this feature, we will need to re architect our entire loading for 200+ pipelines in the near future.

Related to https://github.com/hashicorp/terraform-provider-google/issues/10119

toumorokoshi commented 2 years ago

Hello! Thanks for the feature request.

When the terraform PR is merged you should see the new fields pop up in Config Connector a couple of weeks after.