dandi / dandisets

735 Dandisets, 812.2 TB total. DataLad super-dataset of all Dandisets from https://github.com/dandisets
10 stars 0 forks source link

Add `zarr-checksum <DIRPATH>` subcommand #288

Closed jwodder closed 1 year ago

jwodder commented 1 year ago

Closes #287.

codecov[bot] commented 1 year ago

Codecov Report

Base: 77.19% // Head: 76.78% // Decreases project coverage by -0.40% :warning:

Coverage data is based on head (b019c4a) compared to base (8aa9912). Patch coverage: 36.84% of modified lines in pull request are covered.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## draft #288 +/- ## ========================================== - Coverage 77.19% 76.78% -0.41% ========================================== Files 14 14 Lines 2271 2287 +16 Branches 369 371 +2 ========================================== + Hits 1753 1756 +3 - Misses 374 384 +10 - Partials 144 147 +3 ``` | [Impacted Files](https://codecov.io/gh/dandi/dandisets/pull/288?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi) | Coverage Δ | | |---|---|---| | [tools/backups2datalad/adandi.py](https://codecov.io/gh/dandi/dandisets/pull/288/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#diff-dG9vbHMvYmFja3VwczJkYXRhbGFkL2FkYW5kaS5weQ==) | `79.72% <ø> (ø)` | | | [tools/backups2datalad/zarr.py](https://codecov.io/gh/dandi/dandisets/pull/288/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#diff-dG9vbHMvYmFja3VwczJkYXRhbGFkL3phcnIucHk=) | `73.91% <0.00%> (ø)` | | | [tools/backups2datalad/util.py](https://codecov.io/gh/dandi/dandisets/pull/288/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#diff-dG9vbHMvYmFja3VwczJkYXRhbGFkL3V0aWwucHk=) | `77.14% <20.00%> (-2.12%)` | :arrow_down: | | [tools/backups2datalad/asyncer.py](https://codecov.io/gh/dandi/dandisets/pull/288/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#diff-dG9vbHMvYmFja3VwczJkYXRhbGFkL2FzeW5jZXIucHk=) | `78.47% <28.57%> (-1.71%)` | :arrow_down: | | [tools/backups2datalad/\_\_main\_\_.py](https://codecov.io/gh/dandi/dandisets/pull/288/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#diff-dG9vbHMvYmFja3VwczJkYXRhbGFkL19fbWFpbl9fLnB5) | `63.72% <60.00%> (-0.10%)` | :arrow_down: | | [tools/backups2datalad/config.py](https://codecov.io/gh/dandi/dandisets/pull/288/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#diff-dG9vbHMvYmFja3VwczJkYXRhbGFkL2NvbmZpZy5weQ==) | `90.69% <100.00%> (+0.10%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

yarikoptic commented 1 year ago

FWIW it seems to compute correct one on the "incorrect"ly computed zarr (although also creates the dandisets subdir etc... will ignore for now):

(dandisets) dandi@drogon:/mnt/backup/dandi/dandisets-devel$ python -m tools.backups2datalad zarr-checksum ../dandizarrs/fc3f8c83-f4a9-48e4-a67c-aa24b530f82a
2022-10-27T11:10:31-0400 [INFO    ] backups2datalad: Creating dataset for superdataset
[INFO] Running procedure cfg_text2git
[INFO] == Command start (output follows) =====
[INFO] == Command exit (modification check follows) =====
run(ok): /mnt/backup/dandi/dandisets-devel/dandisets (dataset) [/home/dandi/miniconda3/envs/dandisets/bi...]
create(ok): /mnt/backup/dandi/dandisets-devel/dandisets (dataset)
action summary:
  create (ok: 1)
  run (ok: 1)
2022-10-27T11:10:33-0400 [INFO    ] backups2datalad: Saving logs to dandisets/.git/dandi/backups2datalad/2022.10.27.15.10.33Z.log
30b2374a24485193f7913085155c0f40-32385--55430784494

and on a sample other zarr:

(dandisets) dandi@drogon:/mnt/backup/dandi/dandisets-devel$ python -m tools.backups2datalad zarr-checksum ../dandizarrs/100bec22-b4fb-46aa-9810-e5e8a7f20a20/
2022-10-27T11:17:09-0400 [INFO    ] backups2datalad: Saving logs to dandisets/.git/dandi/backups2datalad/2022.10.27.15.17.09Z.log
601808f54951445cef2fbd97d81718bd-96616--45716753503

(dandisets) dandi@drogon:/mnt/backup/dandi/dandisets-devel$ cat ../dandizarrs/100bec22-b4fb-46aa-9810-e5e8a7f20a20/.dandi/zarr-checksum
601808f54951445cef2fbd97d81718bd-96616--45716753503

is good. Thanks!