search

ome / ome2024-ngff-challenge

Project planning and material repository for the 2024 challenge to generate 1 PB of OME-Zarr data
https://ome.github.io/ome2024-ngff-challenge/
BSD 3-Clause "New" or "Revised" License
15 stars 14 forks source link

Async stats #61

Open joshmoore opened 1 month ago

joshmoore commented 1 month ago

The parse.py command asynchronously parses all the CSVs to discover Zarrs and then from their attempts to load all JSONs within the Zarrs. Running the code then prints all (or nearly all) files other than chunks to stderr which can be captured and passed to wget.

https://github.com/ome/ome2024-ngff-challenge-metadata is the result of:

./parse 2>&1 | tee log
# cleanup the misc. logging
wget -x -i log
git add *

cc: @will-moore @khaledk2 @perlman

netlify[bot] commented 1 month ago

Deploy Preview for ngff-challenge canceled.

Name Link
Latest commit f003985a81bfcdec333d9003c1970f59d5b43a7b
Latest deploy log https://app.netlify.com/sites/ngff-challenge/deploys/67091062399ebe00082fc518