Store statistics in array metadata

joshmoore commented 3 weeks ago

Under the key _ome2024_ngff_challenge_stats we now store the following statistics for each array conversion:

input: endpoint of the input location, "default", or "" if not s3
output: endpoint of the output location, "default", or "" if not s3
read: total bytes read
written: total bytes written
start: epoch at which the conversion started
stop: epoch at which the conversion stopped
elapsed: equals "stop - start"

For example:

  "attributes": {
    "_ome2024_ngff_challenge_stats": {
      "input": "https://uk1s3.embassy.ebi.ac.uk",
      "output": "",
      "start": 1724228072.7066,
      "stop": 1724228073.515819,
      "read": 2312282,
      "written": 1661288,
      "elapsed": 0.8092191219329834
    }

joshmoore commented 3 weeks ago

Thanks. Merging to get the metadata work discussed during this morning's challenge chat opened before the afternoon session.

joshmoore commented 3 weeks ago

@perlman suggestions something like --performance-notes with free-text notes but more specifics might include:

threads
cpus
memory
entire command-line
"run on the cloud"

joshmoore commented 3 weeks ago

Opened https://github.com/ome/ome2024-ngff-challenge/pull/35

ome / ome2024-ngff-challenge

Store statistics in array metadata #31