Problems with uploading metadata due to "id" allegedly not being present in the metadata

[f-innatech]

Hello,

im trying to upload metadata for my tables on the OEP. However, even after meticously following the instructions from here: https://openenergyplatform.github.io/academy/tutorials/01_api/02_api_upload/#notes-on-data-types

and using three different json examples. I always get the same error: Exception: Failed to upload metadata to table all_ghg_emissions_by_sector. Status code: 400, Response: {"reason": "metadata string does not contain an id"}

I used the example for metadata from the above link, json metadata that I copied from this entry: https://openenergyplatform.org/dataedit/view/scenario/eu_leg_data_2014_eio_ir_article23_t1 which already contains full metadata and my indivudally created metadata based on the scheme provided. THis is the table im trying to upload to: https://openenergyplatform.org/dataedit/view/model_draft/all_ghg_emissions_by_sector

here is the function i used to try and upload the metadata. I tried to do it for one of my own tables on the model_draft schema

def upload_metadata_to_table(api_url, table_name, path_json):

Check if the JSON file exists

if not os.path.exists(path_json):
    raise FileNotFoundError(f"The file {path_json} does not exist.")

# Load the JSON data
with open(path_json, 'r') as json_file:
    try:
        data = json.load(json_file)
        print("Loaded JSON data:")
        print(json.dumps(data, indent=4))  # Pretty-print JSON data for inspection
    except json.JSONDecodeError as e:
        raise ValueError(f"Error decoding JSON from the file {path_json}: {e}")

# Check if 'id' and '@id' fields are present
if 'id' not in data:
    raise ValueError("The JSON data does not contain the required 'id' field.")
if '@id' not in data:
    raise ValueError("The JSON data does not contain the required '@id' field.")

# Get the API token
token = environ.get("OEP_API_TOKEN") or getpass("Enter your OEP API token:")
auth_headers = {"Authorization": f"Token {token}"}

# Define the API URL
table_api_url = f"{api_url}/schema/{schema}/tables/{table_name}/meta/"

# Print the API URL, headers, and payload for debugging
print("API URL:", table_api_url)
print("Headers:", auth_headers)
print("Payload:", json.dumps({"query": data}, indent=4))

# Send the POST request
response = requests.post(table_api_url, json={"query": data}, headers=auth_headers)

# Check the response
if response.status_code == 200:
    print(f"Metadata uploaded to table {table_name} successfully.")
else:
    print(f"Error: {response.status_code} - {response.text}")
    raise Exception(f"Failed to upload metadata to table {table_name}. Status code: {response.status_code}, Response: {response.text}")

json_ghg.json test.json

[jh-RLI]

Hey @f-innatech sorry to hear that. I can reproduce this locally. I will let you know once i found the reason.

[jh-RLI]

The tutorial may not be very clear. If you are only uploading the metadata, you should not add the query wrapper element when preparing the query payload:

json={"query": data} -> json=data.

But I think it's worth preventing this in the API.

Additionally i found some other errors in the metadata and fixed them here:

1. the data format was incorrect in the "timeseries" element

   "start": "01.01.2020",
   "end": "31.12.2050",

to

"start": "2020-01-01",
"end": "2050-12-31",

2. and the description of the resource was empty. As you have already created the table on the oep, the template metadata already contains the resource description. I downloaded it and added it to the metadata.

fixed_json_ghg.json

Internally, the metadata is checked with a tool called [omi] (https://github.com/OpenEnergyPlatform/omi). It is currently being revised and will become much more usable (as a CLI tool), but it still helps to find errors. Unfortunately there is no tutorial explaining how to use omi yet.