[EPIC] APEx openEO UDP benchmark & regression workflow

[jdries]

See: https://jdries-vito.quarto.pub/apex-design/algorithm_pipelines.html#integration-test

In APEx, a very important task is to check if algorithms still work, and to estimate their cost. We expect a large number of algorithms, so need some tooling for that. The main idea is to work UDP based, so that we can work with configuration in json rather than code.

The main configuration repo is this one (https://github.com/ESA-APEx/apex_algorithms), it contains:

• records describing the algorithm, pointing to the UDP's (some of them are also in this repo)
• json containing openEO requests that can be run, with job options
• backend info (endpoints, ...)

The runs generate:

• data files, to be compared with reference data stored in object storage. Comparison report should be html, to be put online?
• stac metadata, to be validated
• derived from link: to be compared with reference
• metrics/costs: to be stored in geoparquet on object storage

Prior work:

https://github.com/eu-cdse/openeo-benchmarks

Planning:

• By APEX MVP (oct 2024) a basic setup should be there
• Try to have a very basic thing in june 2024
• Get feedback, and elaborate over summer

Task elaboration:

• [x] #3
• [ ] #2
• [ ] recording of benchmark results
• [ ] reference data management for regressions
• [ ] comparison against reference data, with visual reporting of differences