yarikoptic
commented
9 months ago Here are preliminary results for a simple "download subtree part of zarr" using aws s3 and rclone directly from S3, vs access from dandidav's dandisets/ and then zarrs/ endpoints:
bash script
```shell #!/bin/bash set -eu ds=000108 asset=0dfbb4b4-be63-45a6-8354-347cb98fdb5b if false; then # just ot cut/paste outputs below curl --silent https://api.dandiarchive.org/api/dandisets/000108/versions/draft/assets/$asset/ | { echo -n "version="; jq -r '.digest["dandi:dandi-zarr-checksum"]'; } curl --silent https://api.dandiarchive.org/api/dandisets/000108/versions/draft/assets/$asset/ | { echo -n "zarr="; jq -r '.contentUrl[1]' | sed -e 's,.*zarr/,,g' -e 's,/,,g'; } curl --silent https://api.dandiarchive.org/api/dandisets/000108/versions/draft/assets/$asset/ | { echo -n "path="; jq -r '.path' ; } fi # too big already #zarr=04767811-3cea-43c8-956d-81da5e496f80 #version=WRONG!! #path=sub-MITU01/ses-20211002h21m40s58/micr/sub-MITU01_ses-20211002h21m40s58_sample-37_stain-YO_run-1_chunk-10_SPIM.ome.zarr # a little smaller version=0395d0a3767524377b58da3945b3c063-48379--27115470 zarr=0d5b9be5-e626-4f6a-96da-b6b602954899 path=sub-U01hm15x/ses-20220731h17m24s47/micr/sub-U01hm15x_ses-20220731h17m24s47_sample-mEhmAD031x15R2_ABETA_stain-ABETA_run-1_chunk-1_SPIM.ome.zarr part=${PART:-0/0/0/0/0/} : ${CONCUR:=20} OUT="/tmp/zarr-bm" rm -rf "$OUT"/* || : #set -x # simple download directly from S3 get_aws_s3() { aws configure set default.s3.max_concurrent_requests $CONCUR /usr/bin/time chronic aws s3 --no-sign-request sync s3://dandiarchive/zarr/$zarr/$part "$1" } get_rclone_s3() { aws configure set default.s3.max_concurrent_requests $CONCUR /usr/bin/time rclone sync --transfers $CONCUR DANDI-S3:dandiarchive/zarr/$zarr/$part "$1" } get_rclone_dandisets() { /usr/bin/time rclone sync --transfers $CONCUR DANDI-WEBDAV:dandisets/$ds/draft/$path/$part "$1" } get_rclone_zarr_manifest() { /usr/bin/time rclone sync --transfers $CONCUR DANDI-WEBDAV:zarrs/${zarr:0:3}/${zarr:3:3}/${zarr}/$version/$part "$1" } echo "Downloading part $part within zarr $zarr asking for up to $CONCUR processes" for method in get_aws_s3 get_rclone_s3 get_rclone_dandisets get_rclone_zarr_manifest; do out="$OUT/$method" echo "---------------" echo "$method: $out" $method "$out" checksum=$(TQDM_DISABLE=1 zarrsum local "$out" | tail -n 1) if [ -z "$part" ] && [ $checksum != "$version" ]; then echo "wrong checksum $checksum != $version" fi if [ -n "$part" ]; then echo "checksum $checksum" fi done ```so we see that /dandisets/ endpoint is quite slow, but the manifests based and shorter path of /zarrs/ one is quite performant (well -- still twice slower but that is just that -- twice! ;))
(dev3) yoh@typhon:~/proj/dandi/zarr-benchmarking/tools$ PART=0/0/0/0/ CONCUR=10 ./simple_parallel_get.sh
Downloading part 0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 10 processes
---------------
get_aws_s3: /tmp/zarr-bm/get_aws_s3
22.49user 2.03system 0:29.91elapsed 82%CPU (0avgtext+0avgdata 119156maxresident)k
0inputs+24064outputs (0major+110906minor)pagefaults 0swaps
checksum 742c8d77baf7240437d33117f1d063fb-3008--1684480
---------------
get_rclone_s3: /tmp/zarr-bm/get_rclone_s3
3.64user 0.94system 0:16.16elapsed 28%CPU (0avgtext+0avgdata 85516maxresident)k
0inputs+24064outputs (0major+5286minor)pagefaults 0swaps
checksum 742c8d77baf7240437d33117f1d063fb-3008--1684480
---------------
get_rclone_dandisets: /tmp/zarr-bm/get_rclone_dandisets
4.63user 1.99system 3:16.25elapsed 3%CPU (0avgtext+0avgdata 74280maxresident)k
0inputs+24064outputs (0major+5663minor)pagefaults 0swaps
checksum 742c8d77baf7240437d33117f1d063fb-3008--1684480
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
3.47user 1.55system 0:32.48elapsed 15%CPU (0avgtext+0avgdata 74440maxresident)k
0inputs+24064outputs (0major+5382minor)pagefaults 0swaps
checksum 742c8d77baf7240437d33117f1d063fb-3008--1684480on a serial access we are up to par with aws on /zarrs/ but rclone beats us:
(dev3) yoh@typhon:~/proj/dandi/zarr-benchmarking/tools$ PART=0/0/0/0/ CONCUR=1 ./simple_parallel_get.sh
Downloading part 0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 1 processes
---------------
get_aws_s3: /tmp/zarr-bm/get_aws_s3
20.58user 1.82system 3:06.44elapsed 12%CPU (0avgtext+0avgdata 109096maxresident)k
0inputs+24064outputs (0major+97814minor)pagefaults 0swaps
checksum 742c8d77baf7240437d33117f1d063fb-3008--1684480
---------------
get_rclone_s3: /tmp/zarr-bm/get_rclone_s3
4.69user 1.75system 2:43.25elapsed 3%CPU (0avgtext+0avgdata 81728maxresident)k
0inputs+24064outputs (0major+6314minor)pagefaults 0swaps
checksum 742c8d77baf7240437d33117f1d063fb-3008--1684480
---------------
get_rclone_dandisets: /tmp/zarr-bm/get_rclone_dandisets
4.86user 2.44system 12:44.14elapsed 0%CPU (0avgtext+0avgdata 71256maxresident)k
0inputs+24064outputs (0major+3675minor)pagefaults 0swaps
checksum 742c8d77baf7240437d33117f1d063fb-3008--1684480
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
4.45user 2.20system 3:04.90elapsed 3%CPU (0avgtext+0avgdata 71740maxresident)k
0inputs+24064outputs (0major+4791minor)pagefaults 0swaps
checksum 742c8d77baf7240437d33117f1d063fb-3008--1684480the /dandisets/ path remains many times slower - we should look into optimizing there. But I guess the first thing would be to establish efficient dandi-archive API per
- https://github.com/dandi/dandi-archive/issues/1837 and there on, right @jwodder ?
jwodder
satra
@satra did just a quick&dirty test using zarr viewer comparing access directly to S3 URL and then our testbed dandi.centerforopenneuroscience.org instance (on a zarr in 000108 under
/dandisets/not the manifests based one since no manifest for it was there yet) so there are no quantified characteristics available butWe need to come up with
Compare
to identify how much any component contributes and if we could improve any aspect (e.g. parallel access etc). I hope that it is not all simply due to redirects.
Later we need to create some nice "versioned zarr" with a few versions and to use for benchmarks and also benchmark on the
/zarrendpoints.