OpenEnergyPlatform / omi

Repository for the Open Metadata Integration (OMI). For metadata definition see metadata repo:
https://github.com/OpenEnergyPlatform/metadata
GNU Affero General Public License v3.0
7 stars 4 forks source link

OMI conversion not working anymore #46

Closed jh-RLI closed 2 years ago

jh-RLI commented 2 years ago

Since we have developed the metadata standard OEM further and have not adapted omi immediately, the metadata conversion from a low version to the latest version is currently not supported.

With version v1.5.0 the structure of the metadata was changed, essentially under the key "temporal" the subkey timeseries (already introduced in v1.4.1) was changed to the type list of objects :[{..},{..}]. Since omi works with the current version of the oem structure and does not use one structure definition per version, it is necessary to define in the parser classes or directly in the conversion how the current structure can be translated to the desired metadata version. Also the other new keys have to be included in the conversion method.

If you install omi and try to convert a metadata file v1.4.1 to v.1.5.0 (using the metadata from the oemetadata repository) there is some output but the output is very limited and not correct. omi translate -f oep-v1.4 -t oep-v1.5 ..\oemetadata\metadata\v141\example.json

Output:

{
    "title": "Ludee",
    "object": "metadata",
    "comment": "Release metadata version OEP-1.3.0",
    "date": "2019-07-09"
}
],
"resources": [
{
    "profile": "tabular-data-resource",
    "name": "model_draft.oep_metadata_table_example_v141",
    "path": "http://openenergyplatform.org/dataedit/view/model_draft/oep_metadata_table_example_v141",
    "format": "PostgreSQL",
    "encoding": "UTF-8",
    "schema": {
        "primaryKey": [
            "id"
        ],
        "foreignKeys": [
            {
                "fields": [
                    "year"
                ],
                "reference": {
                    "resource": "schema.table",
                    "fields": [
                        "year"
                    ]
                }
            }
        ],
        "fields": [
            {
                "name": "id",
                "description": "Unique identifier",
                "type": "serial"
            },
            {
                "name": "year",
                "description": "Reference year",
                "type": "integer"
            },
            {
                "name": "value",
                "description": "Example value",
                "type": "double precision",
                "unit": "MW"
            },
            {
                "name": "geom",
                "description": "Geometry",
                "type": "geometry(Point, 4326)"
            }
        ]
    },
    "dialect": {
        "decimalSeparator": "."
    }
}
],
"metaMetadata": {
"metadataVersion": "OEP-1.5.0",
"metadataLicense": {
    "name": "CC0-1.0",
    "title": "Creative Commons Zero v1.0 Universal",
    "path": "https://creativecommons.org/publicdomain/zero/1.0/"
}
}
}
jh-RLI commented 2 years ago

@henhuy looked into omi´s conversion functionality and found out is outdated quite a lot. To me it seems like I would need too much time to fix it ATM. It wasn't touched since oem-v1.3.0. The conversion method uses one of the available parser classes (see the upaded one for oem-v1.5.0) . Since we have made some changes to the metadata structure, the conversion tool needs to know the structures of all metadata versions. But at the moment we only work with one structure, which should always be up to date with the latest metadata version. (see comment above)

henhuy commented 2 years ago

okay, thanks for checking. Conversion looks rather complicated and I am not familiar with it. So I will not add this to our meta_creator right now.