search

C2SM / Sirocco

AiiDA based Weather and climate workflow tool
1 stars 0 forks source link

Add `meta` field to `data` entries #3

Open GeigerJ2 opened 5 months ago

GeigerJ2 commented 5 months ago

In the test_config.yaml file, the "stream 1/2" files are defined via:

stream 1: {type: ICON_output_stream, rel_path: output_1}

I suppose they are normal files? Why are we specifying the "type" here? I feel like the "type" field in that case is not really suitable. I propose we instead add a "meta" field to these data entries, where users put further specifications such as "ICON_output_stream". This, of course, will also depend on what we eventually do with this information, if it changes anything in terms of parsing, storage, etc.

Overall, I'd propose the following schema for the "data" entries: 

type: file / dir ( / zip? Or this possibly better at "format")
format: ncdf (which other formats do we expect?)
abs_path / rel_path: ...
meta: ICON_restart / ICON_output_stream (which other special identifiers do we expect?)
leclairm commented 5 months ago

I'm fairly open there, no pb. Another type of meta data we might need is pattern wtith placeholders for date and time elements, so that tasks can access the files in a directory at a given date and time.