This adds a flag for making all output fields nullable. In addition to the unit tests, I've also manually tested against mozilla-pipeline-schemas.
$ scripts/mps-download-schemas.sh
# generate flat folder with generated schemas
$ ./scripts/mps-generate-schemas.sh control --type bigquery
$ ./scripts/mps-generate-schemas.sh nullable --type bigquery --enforce-nullable
$ git checkout v1.3.0
$ ./scripts/mps-generate-schemas.sh old --type bigquery
# First check -- check that REQUIRED are being transformed into NULLABLE
# see https://gist.github.com/acmiyaguchi/42d30c313baf0a7aaa7aca653f7d54d0
$ diff control nullable
# Second check -- number of ocurrances of REQUIRED
$ grep -rnw control -e 'REQUIRED' | wc
2935 8805 245939
$ grep -rnw nullable -e 'REQUIRED' | wc
0 0 0
$ grep -rnw old -e 'REQUIRED' | wc
2935 8805 234199
# Third check -- control and old are the same
$ diff control old | wc
0 0 0
# Fourth check -- control and old are the same with respect to nullable
$ diff <(diff control nullable) <(diff old nullable) | grep NULLABLE | wc
0 0 0
One side-effect of this flag is that all of the output fields in the schema are NULLABLE, including fields like submission_timestamp and metadata that are added by the decoder.
This adds a flag for making all output fields nullable. In addition to the unit tests, I've also manually tested against mozilla-pipeline-schemas.