APEx benchmarks: add metrics

[JeroenVerstraelen]

Add metrics to the initial benchmark tests.

• For each benchmark, extract metrics using openEO
  • Preferably relative metrics:
    • cost per produced sqkm
    • cost per input pixel
  • Timing related metrics
  • Discuss which other metrics we can include
• Upload results as geoparquet file to S3 object storage
  • Which endpoint/bucket?

Goal: partitioned parquet file (partitioned on: time based, benchmarking scenarios)

[soxofaan]

added some docs at https://github.com/ESA-APEx/apex_algorithms/blob/main/docs/benchmarking.md

[soxofaan]

most important aspect to still cover under this ticket:

• [ ] record collected metrics in a parquet file on S3

[soxofaan]

First results from pushing some metrics as parquet to S3:

[soxofaan]

and now with unrolling the usage stats:

[soxofaan]

merged PR #26 which covers the part about partitioned parquet files on S3

[soxofaan]

Ok I think it's time to close this ticket. There were quite some aspects to it and I'm still unsure about some parts of the current approach, so this feels mostly like a proof of concept solution. Also, not everything of this ticket's original requirements are met, but those were "TBD" anyway.

Some details and discussion about this PoC:

• benchmarks are implemented as pytest test suite
• metrics are collected through a pytest fixture track_metric provided by pytest plugin apex_algorithm_qa_tools.pytest.pytest_track_metrics. Currently collected metrics/properties:
  • scenario_id, job_id
  • job costs
  • job's usage metrics (e.g. usage:cpu:cpu-seconds, usage:memory:mb-seconds)
  • test info: outcome (passed/failed), start time, duration

• these metrics are written in parquet format to S3 (bucket "APEx-benchmarks"), in a folder structure starting at "metrics/v1/metrics.parquet"

  • current partitioning is based on month of the benchmark run, e.g. folder "metrics/v1/metrics.parquet/2024-08/"

  • data is written using "pyarrow.parquet.write_to_dataset" using existing_data_behavior=overwrite_or_ignore mode: there is no appending to an existing file, but each run results in a separate file on S3 (e.g. "metrics/v1/metrics.parquet/2024-08/gh-10612685244-0.parquet").

    • advantage: no concurrency issue or other risks related to file appending (locking/synchronization, risk of corrupting the whole data set with a bad write, ...).
    • disadvantage: will result in overly partitioned data set (lot of small files), so this might require some reconsideration in the longer term (or some workflow where small files are compiled together to larger ones)

    illustration of current "file listing" (containing the results of 5 benchmark runs):

    <FileInfo for 'APEx-benchmarks/metrics/v1/metrics.parquet/2024-08': type=FileType.Directory>,
    <FileInfo for 'APEx-benchmarks/metrics/v1/metrics.parquet/2024-08/2355975626d246df866dae027936bd3d-0.parquet': type=FileType.File, size=6345>,
    <FileInfo for 'APEx-benchmarks/metrics/v1/metrics.parquet/2024-08/7b65314e31614f0fb8390a4b20c70484-0.parquet': type=FileType.File, size=6347>,
    <FileInfo for 'APEx-benchmarks/metrics/v1/metrics.parquet/2024-08/d8b177d03e32493fae8bc0ace6fdf5f3-0.parquet': type=FileType.File, size=6345>,
    <FileInfo for 'APEx-benchmarks/metrics/v1/metrics.parquet/2024-08/gh-10612404377-0.parquet': type=FileType.File, size=7215>,
    <FileInfo for 'APEx-benchmarks/metrics/v1/metrics.parquet/2024-08/gh-10612685244-0.parquet': type=FileType.File, size=7223>]

• As illustration, a view of currently recorded metrics:
• Note that with current setup it is possible to download individual parquet files without credentials, but I did not manage yet to load the larger data set (multiple partitions/files) without credentials with tools like pyarrow.parquet.read_table. I guess this is because of lack of S3 dir/file listing permissions. Need more investigation