Open loveeklund-osttra opened 1 month ago
if a new column is added to a stream that has upsert, it will cause an error because the merge statement will do something along the lines of target.<new_column> = source.<new_column> and fail on <new_column> not found inside target at [1:416].
target.<new_column> = source.<new_column>
<new_column> not found inside target at [1:416]
I think this is because the update_schema function https://github.com/z3z1ma/target-bigquery/blob/9d1d0b08606a716a5a36f53b3388cbd6055535a8/target_bigquery/core.py#L587C1-L601C14 only does a resolve and then updates the schema if it is different.
I created a PR here that could solve it by getting the actual target tables schema and comparing it in update_schema. See PR here https://github.com/z3z1ma/target-bigquery/pull/102
if a new column is added to a stream that has upsert, it will cause an error because the merge statement will do something along the lines of
target.<new_column> = source.<new_column>and fail on<new_column> not found inside target at [1:416].I think this is because the update_schema function https://github.com/z3z1ma/target-bigquery/blob/9d1d0b08606a716a5a36f53b3388cbd6055535a8/target_bigquery/core.py#L587C1-L601C14 only does a resolve and then updates the schema if it is different.
I created a PR here that could solve it by getting the actual target tables schema and comparing it in update_schema. See PR here https://github.com/z3z1ma/target-bigquery/pull/102