search

dandi / dandisets

735 Dandisets, 812.2 TB total. DataLad super-dataset of all Dandisets from https://github.com/dandisets
10 stars 0 forks source link

FOI: 000108 backup crashes due to cloning of an empty zarr repo #328

Closed yarikoptic closed 1 year ago

yarikoptic commented 1 year ago

Run was going very well

```shell [INFO ] scanning for annexed files (this may take some time) [INFO ] Remote origin not usable by git-annex; setting annex-ignore [INFO ] https://github.com/dandizarrs/62f42eaa-495c-47dd-987a-d0c9f8e8919a/config download failed: Not Found [INFO ] access to 1 dataset sibling dandi-dandizarrs-dropbox not auto-enabled, enable with: | datalad siblings -d "/mnt/backup/dandi/dandisets/000108/sub-SChmi53/ses-20220927h13m20s15/micr/sub-SChmi53_ses-20220927h13m20s15_sample-SChmi53_25_stain-LEC_run-1_chunk-2_SPIM.ome.zarr" enable -s dandi-dandizarrs-dropbox install(ok): /mnt/backup/dandi/dandisets/000108/sub-SChmi53/ses-20220927h13m20s15/micr/sub-SChmi53_ses-20220927h13m20s15_sample-SChmi53_25_stain-LEC_run-1_chunk-2_SPIM.ome.zarr (dataset) Renaming remote references: 100% (4/4), done. Adding existing repo at 'sub-SChmi53/ses-20220927h13m20s15/micr/sub-SChmi53_ses-20220927h13m20s15_sample-SChmi53_25_stain-LEC_run-1_chunk-2_SPIM.ome.zarr' to the index add .gitmodules (non-large file; adding content to git repository) ok (recording state in git...) [WARNING] You appear to have cloned an empty repository. Clone attempt: 50%|█████████████████████████████████████████████████████████████████████████████████████████████ | 1.00/2.00 [00:00<00:00, 2.75 Candidate locations/s]2023-02-03T07:00:51-0500 [WARNING] datalad.gitrepo: You appear to have cloned an empty repository. [WARNING] Cloned /mnt/backup/dandi/dandisets/000108/sub-SChmi53/ses-20220927h13m20s15/micr/sub-SChmi53_ses-20220927h13m20s15_sample-SChmi53_25_stain-LEC_run-1_chunk-3_SPIM.ome.zarr but could not find a branch with commits 2023-02-03T07:00:51-0500 [WARNING] datalad.core.distributed.clone: Cloned /mnt/backup/dandi/dandisets/000108/sub-SChmi53/ses-20220927h13m20s15/micr/sub-SChmi53_ses-20220927h13m20s15_sample-SChmi53_25_stain-LEC_run-1_chunk-3_SPIM.ome.zarr but could not find a branch with commits install(ok): /mnt/backup/dandi/dandisets/000108/sub-SChmi53/ses-20220927h13m20s15/micr/sub-SChmi53_ses-20220927h13m20s15_sample-SChmi53_25_stain-LEC_run-1_chunk-3_SPIM.ome.zarr (dataset) fatal: 'sub-SChmi53/ses-20220927h13m20s15/micr/sub-SChmi53_ses-20220927h13m20s15_sample-SChmi53_25_stain-LEC_run-1_chunk-3_SPIM.ome.zarr' does not have a commit checked out 2023-02-03T07:00:52-0500 [ERROR ] backups2datalad: Job failed on input : Traceback (most recent call last): File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 171, in dowork outp = await func(inp) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 145, in update_dandiset changed = await self.sync_dataset(dandiset, ds, dmanager) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 188, in sync_dataset await syncer.sync_assets(error_on_change) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/syncer.py", line 36, in sync_assets self.report = await async_assets( File "/mnt/backup/dandi/dandisets/tools/backups2datalad/asyncer.py", line 554, in async_assets await ds.add_submodule( File "/mnt/backup/dandi/dandisets/tools/backups2datalad/adataset.py", line 508, in add_submodule await self.call_git("submodule", "add", "--", url, path) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/adataset.py", line 118, in call_git await aruncmd( File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 205, in aruncmd return await anyio.run_process(argstrs, **kwargs) File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/anyio/_core/_subprocesses.py", line 90, in run_process raise CalledProcessError(cast(int, process.returncode), command, output, errors) subprocess.CalledProcessError: Command '['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'submodule', 'add', '--', 'https://github.com/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f', 'sub-SChmi53/ses-20220927h13m20s15/micr/sub-SChmi53_ses-20220927h13m20s15_sample-SChmi53_25_stain-LEC_run-1_chunk-3_SPIM.ome.zarr']' returned non-zero exit status 128. Traceback (most recent call last): File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/__main__.py", line 513, in main(_anyio_backend="asyncio") File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/asyncclick/core.py", line 1157, in __call__ return anyio.run(self._main, main, args, kwargs, **opts) File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/anyio/_core/_eventloop.py", line 70, in run return asynclib.run(func, *args, **backend_options) File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 292, in run return native_run(wrapper(), debug=debug) File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper return await func(*args) File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/asyncclick/core.py", line 1160, in _main return await main(*args, **kwargs) File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/asyncclick/core.py", line 1076, in main rv = await self.invoke(ctx) File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/asyncclick/core.py", line 1687, in invoke return await _process_result(await sub_ctx.command.invoke(sub_ctx)) File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/asyncclick/core.py", line 1434, in invoke return await ctx.invoke(self.callback, **ctx.params) File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/asyncclick/core.py", line 780, in invoke rv = await rv File "/mnt/backup/dandi/dandisets/tools/backups2datalad/__main__.py", line 186, in update_from_backup await datasetter.update_from_backup(dandisets, exclude=exclude) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 97, in update_from_backup raise RuntimeError( RuntimeError: Backups for 1 Dandiset failed real 1930m45.191s user 1423m14.969s sys 1070m21.006s ```

https://github.com/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f is indeed empty. But the one we have locally is not:

(dandisets) dandi@drogon:/mnt/backup/dandi/dandisets$ ls -l ../dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f
total 4
drwxr-sr-x 1 dandi dandi  16 Jan 16 20:40 0
drwxr-sr-x 1 dandi dandi  16 Jan 16 20:51 1
drwxr-sr-x 1 dandi dandi  16 Jan 16 20:52 2
drwxr-sr-x 1 dandi dandi  16 Jan 16 20:53 3
drwxr-sr-x 1 dandi dandi  16 Jan 16 20:53 4
drwxr-sr-x 1 dandi dandi  16 Jan 16 20:53 5
drwxr-sr-x 1 dandi dandi  16 Jan 16 20:53 6
lrwxrwxrwx 1 dandi dandi 114 Jan 16 20:53 info -> .git/annex/objects/2W/3f/MD5E-s2661--10c0cf51246ebdb68cc758693acebcbb/MD5E-s2661--10c0cf51246ebdb68cc758693acebcbb

so I guess we didn't push. The question why/how we didn't spot - probably happened in one of the crushed/killed cases... I will dig a little more but overall I think I would just ensure that we have everything pushed among present zarrs, and restart backup.

yarikoptic commented 1 year ago
From the most recent log (run above) cfe2814c-005c-4b5d-85b5-7b1d316e027f 2023.02.02.03.50.10Z.log ```shell (dandisets) dandi@drogon:/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad$ grep cfe2814c-005c-4b5d-85b5-7b1d316e027f 2023.02.02.03.50.10Z.log | less 2023-02-01T23:13:57-0500 [DEBUG ] backups2datalad: Dandiset 000108: Zarr cfe2814c-005c-4b5d-85b5-7b1d316e027f: Creating GitHub sibling 2023-02-01T23:13:57-0500 [DEBUG ] backups2datalad: Running: git -c receive.autogc=0 -c gc.auto=0 remote [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] 2023-02-01T23:14:09-0500 [DEBUG ] backups2datalad: GitHub remote already exists for cfe2814c-005c-4b5d-85b5-7b1d316e027f 2023-02-01T23:14:09-0500 [DEBUG ] backups2datalad: Dandiset 000108: Zarr cfe2814c-005c-4b5d-85b5-7b1d316e027f: Created GitHub sibling 2023-02-01T23:14:13-0500 [DEBUG ] backups2datalad: Opening pipe to `git -c receive.autogc=0 -c gc.auto=0 ls-tree -r --name-only -z HEAD` [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] 2023-02-01T23:14:25-0500 [DEBUG ] backups2datalad: Command `git -c receive.autogc=0 -c gc.auto=0 ls-tree -r --name-only -z HEAD` [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] exited with return code 0 2023-02-01T23:15:46-0500 [INFO ] backups2datalad: Dandiset 000108: Zarr cfe2814c-005c-4b5d-85b5-7b1d316e027f: backup up to date 2023-02-01T23:15:52-0500 [INFO ] backups2datalad: Dandiset 000108: Zarr cfe2814c-005c-4b5d-85b5-7b1d316e027f: no changes; not committing 2023-02-01T23:15:52-0500 [DEBUG ] backups2datalad: Running: git -c receive.autogc=0 -c gc.auto=0 show -s --format=%H [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] 2023-02-01T23:16:08-0500 [DEBUG ] backups2datalad: Running: git -c receive.autogc=0 -c gc.auto=0 show -s --format=%H [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] 2023-02-03T07:00:51-0500 [DEBUG ] backups2datalad: Running: git -c receive.autogc=0 -c gc.auto=0 config --file .datalad/config --get datalad.dataset.id [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] 2023-02-03T07:00:52-0500 [DEBUG ] backups2datalad: Running: git -c receive.autogc=0 -c gc.auto=0 submodule add -- https://github.com/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f sub-SChmi53/ses-20220927h13m20s15/micr/sub-SChmi53_ses-20220927h13m20s15_sample-SChmi53_25_stain-LEC_run-1_chunk-3_SPIM.ome.zarr [cwd=/mnt/backup/dandi/dandisets/000108] subprocess.CalledProcessError: Command '['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'submodule', 'add', '--', 'https://github.com/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f', 'sub-SChmi53/ses-20220927h13m20s15/micr/sub-SChmi53_ses-20220927h13m20s15_sample-SChmi53_25_stain-LEC_run-1_chunk-3_SPIM.ome.zarr']' returned non-zero exit status 128. ```
prior one: 2023.01.30.22.05.08Z.log - no updates, no pushing ```shell (dandisets) dandi@drogon:/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad$ grep cfe2814c-005c-4b5d-85b5-7b1d316e027f 2023.01.30.22.05.08Z.log 2023-01-30T19:07:30-0500 [DEBUG ] backups2datalad: Dandiset 000108: Zarr cfe2814c-005c-4b5d-85b5-7b1d316e027f: Creating GitHub sibling 2023-01-30T19:07:30-0500 [DEBUG ] backups2datalad: Running: git -c receive.autogc=0 -c gc.auto=0 remote [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] 2023-01-30T19:07:30-0500 [DEBUG ] backups2datalad: GitHub remote already exists for cfe2814c-005c-4b5d-85b5-7b1d316e027f 2023-01-30T19:07:30-0500 [DEBUG ] backups2datalad: Dandiset 000108: Zarr cfe2814c-005c-4b5d-85b5-7b1d316e027f: Created GitHub sibling 2023-01-30T19:07:33-0500 [DEBUG ] backups2datalad: Opening pipe to `git -c receive.autogc=0 -c gc.auto=0 ls-tree -r --name-only -z HEAD` [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] 2023-01-30T19:07:38-0500 [DEBUG ] backups2datalad: Command `git -c receive.autogc=0 -c gc.auto=0 ls-tree -r --name-only -z HEAD` [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] exited with return code 0 2023-01-30T19:08:11-0500 [INFO ] backups2datalad: Dandiset 000108: Zarr cfe2814c-005c-4b5d-85b5-7b1d316e027f: backup up to date 2023-01-30T19:08:14-0500 [INFO ] backups2datalad: Dandiset 000108: Zarr cfe2814c-005c-4b5d-85b5-7b1d316e027f: no changes; not committing 2023-01-30T19:08:14-0500 [INFO ] backups2datalad: /mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f: Counting up files ... 2023-01-30T19:08:15-0500 [DEBUG ] backups2datalad: Opening pipe to `git ls-tree -lrz HEAD` [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] 2023-01-30T19:08:29-0500 [DEBUG ] backups2datalad: Command `git ls-tree -lrz HEAD` [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] exited with return code 0 2023-01-30T19:08:29-0500 [DEBUG ] backups2datalad: Opening pipe to `git -c receive.autogc=0 -c gc.auto=0 annex find '--include=*' --json` [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] 2023-01-30T19:09:14-0500 [DEBUG ] backups2datalad: Command `git -c receive.autogc=0 -c gc.auto=0 annex find '--include=*' --json` [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] exited with return code 0 2023-01-30T19:09:15-0500 [INFO ] backups2datalad: /mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f: Done counting up files 2023-01-30T19:09:15-0500 [DEBUG ] backups2datalad: Running: git -c receive.autogc=0 -c gc.auto=0 show -s --format=%H [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] 2023-01-30T19:09:19-0500 [DEBUG ] backups2datalad: Editing repository dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f 2023-01-30T19:09:23-0500 [DEBUG ] backups2datalad: Running: git -c receive.autogc=0 -c gc.auto=0 show -s --format=%H [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] ```
the oldest - 2023.01.06.15.11.45Z.log - is the one where it was created/populated not clear if failed anyhow or was killed ... ```shell 2023-01-10T02:03:38-0500 [INFO ] backups2datalad: Dandiset 000108: Zarr cfe2814c-005c-4b5d-85b5-7b1d316e027f: 0/0/0/7/9/87: Syncing 2023-01-10T02:03:38-0500 [INFO ] backups2datalad: Dandiset 000108: Zarr cfe2814c-005c-4b5d-85b5-7b1d316e027f: 0/0/0/7/9/87: Not in dataset; will add 2023-01-10T02:03:39-0500 [DEBUG ] backups2datalad: Running: git -c receive.autogc=0 -c gc.auto=0 rm -f --ignore-unmatch -- 0/0/0/7/9/87 [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] 2023-01-10T02:03:44-0500 [DEBUG ] backups2datalad: Waiting for `git -c receive.autogc=0 -c gc.auto=0 annex fromkey --force --batch --json --json-error-messages` [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] to terminate 2023-01-10T02:03:49-0500 [DEBUG ] backups2datalad: Command `git -c receive.autogc=0 -c gc.auto=0 annex fromkey --force --batch --json --json-error-messages` [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] did not terminate in time; sending SIGTERM 2023-01-10T02:03:49-0500 [DEBUG ] backups2datalad: Command `git -c receive.autogc=0 -c gc.auto=0 annex fromkey --force --batch --json --json-error-messages` [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] successfully terminated 2023-01-10T02:03:49-0500 [DEBUG ] backups2datalad: Waiting for `git -c receive.autogc=0 -c gc.auto=0 annex examinekey --batch --migrate-to-backend=MD5E` [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] to terminate 2023-01-10T02:03:50-0500 [DEBUG ] backups2datalad: Command `git -c receive.autogc=0 -c gc.auto=0 annex examinekey --batch --migrate-to-backend=MD5E` [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] exited with return code 0 2023-01-10T02:03:50-0500 [DEBUG ] backups2datalad: Waiting for `git -c receive.autogc=0 -c gc.auto=0 annex whereis --batch-keys --json --json-error-messages` [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] to terminate 2023-01-10T02:03:50-0500 [DEBUG ] backups2datalad: Command `git -c receive.autogc=0 -c gc.auto=0 annex whereis --batch-keys --json --json-error-messages` [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] exited with return code 1 2023-01-10T02:03:50-0500 [DEBUG ] backups2datalad: Waiting for `git -c receive.autogc=0 -c gc.auto=0 annex registerurl -c annex.alwayscompact=false --batch --json --json-error-messages` [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] to terminate 2023-01-10T02:03:55-0500 [DEBUG ] backups2datalad: Command `git -c receive.autogc=0 -c gc.auto=0 annex registerurl -c annex.alwayscompact=false --batch --json --json-error-messages` [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] did not terminate in time; sending SIGTERM 2023-01-10T02:03:56-0500 [DEBUG ] backups2datalad: Command `git -c receive.autogc=0 -c gc.auto=0 annex registerurl -c annex.alwayscompact=false --batch --json --json-error-messages` [cwd=/mnt/backup/dandi/dandizarrs/cfe2814c-005c-4b5d-85b5-7b1d316e027f] successfully terminated ``` overall -- the run died due to ``` 2023-01-10T02:04:16-0500 [ERROR ] backups2datalad: Job failed on input : Traceback (most recent call last): File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 168, in dowork outp = await func(inp) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 145, in update_dandiset changed = await self.sync_dataset(dandiset, ds, dmanager) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 188, in sync_dataset await syncer.sync_assets(error_on_change) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/syncer.py", line 36, in sync_assets self.report = await async_assets( File "/mnt/backup/dandi/dandisets/tools/backups2datalad/asyncer.py", line 500, in async_assets nursery.start_soon(dm.read_addurl) File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__ raise exceptions[0] File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 537, in sync_zarr await zsync.run() File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 264, in run raise RuntimeError( RuntimeError: Zarr 9fc86864-2ccf-428e-8605-7130425f223b: local checksum '68e0a6f07d4ed53cf473124c50c0e93c-26689--26322880109' differs from remote checksum '1a413201ba27b8bc991702b1f80ac8b4-26940--26481139599' after backup, and no change on server was detected ```
yarikoptic commented 1 year ago

FWIW doing 

/bin/ls -td *-*-* | while read d; do
    ( cd $d; gsha=$(git show-ref github/draft | awk '{print $1;}'); lsha=$(git show-ref heads/draft | awk '{print $1;}'); [ -n "$lsha" ] && [ "$lsha" == "$gsha" ] || { echo $d; datalad push --to=github; } );
done

now to go through all dandizarrs from newer to older and push whenever head is not the same on github

yarikoptic commented 1 year ago

"interesting" is that it found quite a few to push. Lots (if not all) are from Dec 14th 2022, currently on 

(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandizarrs$ nl /tmp/dandizars.list | grep 2456b396-edb5-4344-adb1-5b05d1f32808
   442  drwxr-sr-x 1 dandi dandi 134 Dec 14 03:29 2456b396-edb5-4344-adb1-5b05d1f32808

out of

(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandizarrs$ nl /tmp/dandizars.list | tail -n 1
  3939  drwxr-sr-x 1 dandi dandi 126 May 26  2022 1825d3a0-81f4-4fe3-923b-a6149d059601
yarikoptic commented 1 year ago

so I did all the pushes etc, hopefully made it into a consistent state. Let's close for now