OMI omits `null` values from json during processing

[chrwm]

When omi processes json metadata it omits values and metadata that are null. This is due when omi reads a json into a python dict, null is read in as None. This results in missing metadata fields when compiling with omi.

e.g. missing description field and unit field

input

"schema": {
                "fields": [
                    {
                        "name": "fid",
                        "description": null,
                        "type": "int",
                        "unit": null,
                        "isAbout": [
                            {
                                "name": null,
                                "path": null
                            }
                        ],
                        "valueReference": [
                            {
                                "value": null,
                                "name": null,
                                "path": null
                            }
                        ]
                    }

output

 "schema": {
                "primaryKey": [],
                "foreignKeys": [],
                "fields": [{
                        "name": "fid",
                        "type": "int",
                        "isAbout": [{}
                        ],
                        "valueReference": [{}
                        ]
                    }

This is important as oem2orm throughs an error for some keys when generating tables from metadata for the OEP and they're missing, e.g. description.

ERROR:Could not generate tables from metadatafile:

KeyError                                  Traceback (most recent call last)
<ipython-input-17-7e96dde4dbd2> in <module>
----> 1 tables_orm = oem2orm.collect_tables_from_oem(db, metadata_folder)

~\Miniconda3\envs\p_py38_oep-upload\lib\site-packages\oem2orm\oep_oedialect_oem2orm.py in collect_tables_from_oem(db, oem_folder_path)
    272     for metadata_file in metadata_files:
    273         try:
--> 274             md_tables = create_tables_from_metadata_file(db, metadata_file)
    275             logging.info(md_tables)
    276         except:

~\Miniconda3\envs\p_py38_oep-upload\lib\site-packages\oem2orm\oep_oedialect_oem2orm.py in create_tables_from_metadata_file(db, metadata_file)
    217                     column_type,
    218                     primary_key=field["name"] in primary_keys,
--> 219                     comment=field["description"],
    220                 )
    221             columns.append(column)

KeyError: 'description'

The current status makes automatic metadata processing and upload impracticable.

EDIT

After implementing omit_none=False results

output

"schema": {
                "primaryKey": [],
                "foreignKeys": [],
                "fields": [{
                        "name": "fid",
                        "description": null,
                        "type": "int",
                        "isAbout": [{
                                "name": null,
                                "path": null
                            }
                        ],
                        "valueReference": [{
                                "value": null,
                                "name": null,
                                "path": null
                            }
                        ],
                        "unit": null
                    }