Open kennknowles opened 2 years ago
@kennknowles any updates on this issue? This is causing problems with us in production
For anyone interested, a workaround is explained here: https://dev.to/stack-labs/tricky-dataflow-ep-1-auto-create-bigquery-tables-in-pipelines-n2k
My workflow : KAFKA -> Dataflow streaming -> BigQuery
Given that having low-latency isn't important in my case, I use FILE_LOADS to reduce the costs. I'm using BigQueryIO.Write with a DynamicDestination, which is a table with the current hour as a suffix.
This BigQueryIO.Write is configured like this :
The first table is successfully created and is written to. But then the following tables are never created and I get these exceptions:
The CreateDisposition used is _CREATENEVER, contrary as _CREATE_IFNEEDED as specified.
Imported from Jira BEAM-3772. Original Jira may contain additional context. Reported by: benjben.