ESA-APEx / apex_algorithms

Hosted APEx algorithms
Apache License 2.0
0 stars 0 forks source link

Schema for benchmark scenario definitions #14

Open soxofaan opened 6 days ago

soxofaan commented 6 days ago

Under #4/#13 I did initial setup of running benchmark scenarios, based on an initial use case definition.

Some things to discuss/(re)consider/finetune:

  1. current structure of the JSON files is a top-level array with scenario objects ([ {"id": "...", ...}, {"id": ...} ]). This makes it impossible to add top-level properties in the future. Things we might want to add at top-level: references to owner/author, default values for parameters, run schedule constraints, credit constraints, .... Instead, change toplevel to an object, e.g.

    {
      "type": "benchmark-scenarios",
      "scenarios": [
        {"id": ...},
        {"id": ...},
      ]
    }
  2. scenarios currently have "type": "openeo", but that is quite generic if you compare it how the "type" field is used in GeoJSON, STAC, OGC, ... I guess "type": "openeo" intends to say that this is a openeo based benchmark. I would make type a bit more verbose, e.g. "openeo-benchmark" or "openeo-process". Or use a different field name, e.g. "processing_mode": "openeo"
  3. related to previous: is it intended to also have non-openeo benchmarks here?
  4. Should benchmarks always run batch jobs, or should there be an option for sync processing too?
  5. current examples don't have a save_result node, e.g. to set output format. We probably should set this explicitly to avoid depending on implicit behavior