mozilla / jsonschema-transpiler

Compile JSON Schema into Avro and BigQuery schemas
Mozilla Public License 2.0
43 stars 9 forks source link

Fix #84 - Add `--force-nullable` option to make all fields nullable #85

Closed acmiyaguchi closed 5 years ago

acmiyaguchi commented 5 years ago

This adds a flag for making all output fields nullable. In addition to the unit tests, I've also manually tested against mozilla-pipeline-schemas.

$ scripts/mps-download-schemas.sh

# generate flat folder with generated schemas
$ ./scripts/mps-generate-schemas.sh control --type bigquery
$ ./scripts/mps-generate-schemas.sh nullable --type bigquery --enforce-nullable
$ git checkout v1.3.0
$ ./scripts/mps-generate-schemas.sh old --type bigquery

# First check -- check that REQUIRED are being transformed into NULLABLE
# see https://gist.github.com/acmiyaguchi/42d30c313baf0a7aaa7aca653f7d54d0
$ diff control nullable

# Second check -- number of ocurrances of REQUIRED
$ grep -rnw control -e 'REQUIRED' | wc
    2935    8805  245939
$ grep -rnw nullable -e 'REQUIRED' | wc
       0       0       0
$ grep -rnw old -e 'REQUIRED' | wc
    2935    8805  234199

# Third check -- control and old are the same
$ diff control old | wc
       0       0       0

# Fourth check -- control and old are the same with respect to nullable
$ diff <(diff control nullable) <(diff old nullable) | grep NULLABLE | wc
       0       0       0
acmiyaguchi commented 5 years ago

One side-effect of this flag is that all of the output fields in the schema are NULLABLE, including fields like submission_timestamp and metadata that are added by the decoder.