dandi / dandisets

760 Dandisets, 817.2 TB total. DataLad super-dataset of all Dandisets from https://github.com/dandisets
10 stars 0 forks source link

Make backups and population asynchronously parallel #205

Closed jwodder closed 2 years ago

jwodder commented 2 years ago

Closes #173.

This adds a -w/--workers option to the update-from-backup, populate, and populate-zarrs commands; the option can also be set via the config file.

codecov[bot] commented 2 years ago

Codecov Report

Merging #205 (eafbec4) into draft (b1b54bd) will increase coverage by 2.57%. The diff coverage is 79.19%.

@@            Coverage Diff             @@
##            draft     #205      +/-   ##
==========================================
+ Coverage   76.82%   79.40%   +2.57%     
==========================================
  Files           8       13       +5     
  Lines        1450     1777     +327     
  Branches      233      299      +66     
==========================================
+ Hits         1114     1411     +297     
+ Misses        260      257       -3     
- Partials       76      109      +33     
Impacted Files Coverage Δ
tools/backups2datalad/__main__.py 62.28% <52.38%> (+8.11%) :arrow_up:
tools/backups2datalad/datasetter.py 71.16% <69.81%> (-1.28%) :arrow_down:
tools/backups2datalad/adandi.py 76.31% <76.31%> (ø)
tools/backups2datalad/zarr.py 81.27% <80.00%> (+1.34%) :arrow_up:
tools/backups2datalad/asyncer.py 81.22% <84.52%> (-0.50%) :arrow_down:
tools/backups2datalad/logging.py 85.29% <85.29%> (ø)
tools/backups2datalad/adataset.py 85.47% <85.47%> (ø)
tools/backups2datalad/aioutil.py 86.50% <86.50%> (ø)
tools/backups2datalad/annex.py 90.32% <89.47%> (-0.86%) :arrow_down:
tools/backups2datalad/config.py 91.30% <91.30%> (ø)
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update af10b63...eafbec4. Read the comment docs.

yarikoptic commented 2 years ago

thanks. current one crashed due to #207 , so let's just merge and try this one!