Got a backup on 000026 crashed with (pasting full output just in case)
Subject: Cron <dandi@drogon> chronic flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron.lock bash -c '/mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron'
2022-12-19T05:02:26-0500 [WARNING ] backups2datalad: Retrying GET request to /dandisets/000049/versions/ in 1.012717 seconds as it raised ReadTimeout:
2022-12-19T05:02:26-0500 [WARNING ] backups2datalad: Retrying GET request to /dandisets/000051/versions/ in 0.972173 seconds as it raised ReadTimeout:
2022-12-19T05:02:26-0500 [WARNING ] backups2datalad: Retrying GET request to /dandisets/000050/versions/draft/ in 1.023426 seconds as it raised ReadTimeout:
2022-12-19T05:02:26-0500 [WARNING ] backups2datalad: Retrying GET request to /dandisets/000052/versions/ in 1.046905 seconds as it raised ReadTimeout:
2022-12-19T05:02:32-0500 [WARNING ] backups2datalad: Retrying GET request to /dandisets/000051/versions/ in 1.908134 seconds as it raised ReadTimeout:
2022-12-19T05:02:32-0500 [WARNING ] backups2datalad: Retrying GET request to /dandisets/000049/versions/ in 2.068188 seconds as it raised ReadTimeout:
2022-12-19T05:02:32-0500 [WARNING ] backups2datalad: Retrying GET request to /dandisets/000050/versions/draft/ in 1.910042 seconds as it raised ReadTimeout:
2022-12-19T05:02:32-0500 [WARNING ] backups2datalad: Retrying GET request to /dandisets/000052/versions/ in 2.062547 seconds as it raised ReadTimeout:
add dandiset.yaml (non-large file; adding content to git repository) ok
(recording state in git...)
2022-12-19T05:03:23-0500 [WARNING ] backups2datalad: Retrying GET request to /dandisets/000166/versions/ in 0.966733 seconds as it raised ReadTimeout:
2022-12-19T05:03:23-0500 [WARNING ] backups2datalad: Retrying GET request to /dandisets/000168/versions/ in 0.987266 seconds as it raised ReadTimeout:
2022-12-19T05:03:23-0500 [WARNING ] backups2datalad: Retrying GET request to /dandisets/000169/versions/ in 0.989756 seconds as it raised ReadTimeout:
2022-12-19T05:03:23-0500 [WARNING ] backups2datalad: Retrying GET request to /dandisets/000167/versions/ in 1.025325 seconds as it raised ReadTimeout:
2022-12-19T05:05:23-0500 [ERROR ] backups2datalad: Job failed on input <Dandiset 000026/draft>:
Traceback (most recent call last):
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 168, in dowork
outp = await func(inp)
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 145, in update_dandiset
changed = await self.sync_dataset(dandiset, ds, dmanager)
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 188, in sync_dataset
await syncer.sync_assets(error_on_change)
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/syncer.py", line 36, in sync_assets
self.report = await async_assets(
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/asyncer.py", line 500, in async_assets
nursery.start_soon(dm.read_addurl)
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
raise exceptions[0]
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/asyncer.py", line 263, in process_asset
await self.ds.remove(asset.path)
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/adataset.py", line 245, in remove
r = await aruncmd("fuser", "-v", lockfile, stdout=subprocess.PIPE)
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 202, in aruncmd
return await anyio.run_process(argstrs, **kwargs)
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/anyio/_core/_subprocesses.py", line 90, in run_process
raise CalledProcessError(cast(int, process.returncode), command, output, errors)
subprocess.CalledProcessError: Command '['fuser', '-v', '/mnt/backup/dandi/dandisets/000026/.git/index.lock']' returned non-zero exit status 1.
Traceback (most recent call last):
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/__main__.py", line 502, in <module>
main(_anyio_backend="asyncio")
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/asyncclick/core.py", line 1157, in __call__
return anyio.run(self._main, main, args, kwargs, **opts)
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/anyio/_core/_eventloop.py", line 70, in run
return asynclib.run(func, *args, **backend_options)
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 292, in run
return native_run(wrapper(), debug=debug)
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
return await func(*args)
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/asyncclick/core.py", line 1160, in _main
return await main(*args, **kwargs)
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/asyncclick/core.py", line 1076, in main
rv = await self.invoke(ctx)
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/asyncclick/core.py", line 1687, in invoke
return await _process_result(await sub_ctx.command.invoke(sub_ctx))
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/asyncclick/core.py", line 1434, in invoke
return await ctx.invoke(self.callback, **ctx.params)
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/asyncclick/core.py", line 780, in invoke
rv = await rv
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/__main__.py", line 186, in update_from_backup
await datasetter.update_from_backup(dandisets, exclude=exclude)
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 97, in update_from_backup
raise RuntimeError(
RuntimeError: Backups for 1 Dandiset failed
so we are again hitting that rm issue we haven't seen for awhile but now differently . I guess we should just ignore fuser failing and do output stdout/stderr it provides. Worth also showing the exit code which I guess could vary.
Got a backup on 000026 crashed with (pasting full output just in case)
so we are again hitting that
rmissue we haven't seen for awhile but now differently . I guess we should just ignore fuser failing and do output stdout/stderr it provides. Worth also showing the exit code which I guess could vary.meanwhile I will hard reset and restart backup