i4Ds / Karabo-Pipeline

The Karabo Pipeline can be used as Digital Twin for SKA
https://i4ds.github.io/Karabo-Pipeline/
MIT License
11 stars 5 forks source link

Thread metadata objects through pipelines #607

Open sfiruch opened 1 month ago

sfiruch commented 1 month ago

Scenario

Bob reads .vis files from MWA. These .vis files contain metadata, like provenance ("these visibilities were measured by MWA"). Bob runs the visibilities through imagers.

Actual Outcome

The resulting Karabo image does not contain information where the data came from.

Expected Outcome

The resulting Karabo image contains metadata information ("this image was generated from MWA visibilities").

Proposed Design

  1. All Karabo objects contain an (perhaps initially untyped?) "metadata" property
  2. All functions that create a new object, e.g. creating an image from visibilities, assign the metadata object from its sources.
  3. Code/frameworks that write external files should try to include as much information from the metadata object in the written files.

For future versions, the functions mentioned in #2 should try to update/change/convert metadata for the respective data product.

Lukas113 commented 1 month ago

I think we should put some thoughts into something like that before introducing: