search

mozilla / jsonschema-transpiler

Compile JSON Schema into Avro and BigQuery schemas
Mozilla Public License 2.0
42 stars 10 forks source link

Bug 1839640 - BigQuery: Convert json atoms to a JSON column type #107

Closed badboy closed 7 months ago

badboy commented 1 year ago

I think this is all we need here? At least it works in my hacky schema locally.

badboy commented 1 year ago

Backwards compatibility will be the real pain here, although we may not actually be generating any JSON fields at the moment. One way to do check is to run mozilla-schema-generator with your changes here and compare the generated schemas to our prod runs, and verify that they match.

The docs could also use updated. It's not clear to me from this change when a JSON column gets reported, since technically everything being passed in is JSON.

yeah, as can be seen from the test failure there's definitely some edge cases and it's unclear whether those can occur in real code from us. in the worst case we might need to restrict this a bit further.

badboy commented 10 months ago

After talking with akomar we came to the conclusion that we need some metadata configuration to pass down which fields get to be JSON columns. The idea is to put that into mozPipelineMetadata, which this PR now implements. I'm still testing this out across the pipeline tools.