Edge case for object properties with empty schemas

z3z1ma / target-bigquery

target-bigquery is a Singer target for BigQuery. It supports storage write, GCS, streaming, and batch load methods. Built with the Meltano SDK.

MIT License

28 stars 38 forks source link

Closed rubenvereecken closed 6 months ago

rubenvereecken commented 6 months ago

I ran into issues with properties that had the following type

{'items': {'type': 'object'}, 'type': ['null', 'array']}

because BQ was expecting a schema for the objects, but target-bigquery was generating an empty schema RECORD type.

This PR changes that to a JSON type if there's nothing to base a schema on, in line with the general assumption that JSON is a good fallback.

Should break nothing, as hitting this edge case produces errors without this PR.

buu-nguyen commented 6 months ago

LGTM I faced the same issue. After merging this into my fork, it works perfectly.

z3z1ma commented 6 months ago

LGTM, thanks for this!