search

dandi / dandisets

760 Dandisets, 817.2 TB total. DataLad super-dataset of all Dandisets from https://github.com/dandisets
10 stars 0 forks source link

Add `backup-zarrs` subcommand #216

Closed jwodder closed 2 years ago

jwodder commented 2 years ago

For #213.

This PR adds a backups2datalad backup-zarrs [-P <partial-dir>] [-w <workers>] <dandiset> subcommand that goes through all of the Zarrs of the given Dandiset one by one, backing them up and adding them to the Dandiset dataset individually. In order to keep track of which Zarrs have been completely backed up, Zarrs that are in-progress are stored in a "partial dir" (default: $backup_root/partial-zarrs) and then moved to dandizarrs once the backup is complete, before cloning to the Dandiset dataset. A shell script prepare-partial-zarrs.sh is also included for identifying any Zarrs in dandizarrs that are not completely backed up and moving them to partial-zarrs, while also adding any complete Zarr backups to the Dandiset dataset; this script should be run before running backup-zarrs.

codecov[bot] commented 2 years ago

Codecov Report

Merging #216 (1eb0ece) into draft (2288bc3) will increase coverage by 0.24%. The diff coverage is 76.23%.

@@            Coverage Diff             @@
##            draft     #216      +/-   ##
==========================================
+ Coverage   77.96%   78.20%   +0.24%     
==========================================
  Files          13       13              
  Lines        1788     1872      +84     
  Branches      303      314      +11     
==========================================
+ Hits         1394     1464      +70     
- Misses        283      287       +4     
- Partials      111      121      +10
Impacted Files Coverage Δ
tools/backups2datalad/asyncer.py 80.96% <50.00%> (ø)
tools/backups2datalad/__main__.py 62.43% <64.28%> (+0.14%) :arrow_up:
tools/backups2datalad/datasetter.py 71.47% <75.47%> (+1.25%) :arrow_up:
tools/backups2datalad/adandi.py 78.15% <83.33%> (+1.83%) :arrow_up:
tools/backups2datalad/adataset.py 80.12% <84.61%> (+2.26%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 7cbddc5...1eb0ece. Read the comment docs.

yarikoptic commented 2 years ago

did cursory review. FTR: made a snapshot of the entire partition on drogon (/mnt/backup/.snapshots/20220629) just in case. Let's proceed