Currently, the BQ IO can not handle schema changes if a table schema changes during a streaming job the IO returns an error and the rows are dropped.
We propose a new mechanism that will enable BQ to handle a schema change:
On startup a check will be made to ensure table schemas are up to date, if not, update the current schema
Upon receiving a schema error, check if there are any columns in the schema supplied to the BQ IO step that is not in the existing table schema, the column(s) will be added to the table
@tomlynchRNA
@ahmedabu98
Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
What would you like to happen?
Currently, the BQ IO can not handle schema changes if a table schema changes during a streaming job the IO returns an error and the rows are dropped. We propose a new mechanism that will enable BQ to handle a schema change:
@tomlynchRNA @ahmedabu98
Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
Issue Components