search

dandi / dandisets

692 Dandisets, 804.4 TB total. DataLad super-dataset of all Dandisets from https://github.com/dandisets
10 stars 0 forks source link

--mode verify on 000026: Zarr ... entry '0/1/29/72' added, but Dandiset draft timestamp was not updated on server #340

Closed yarikoptic closed 4 months ago

yarikoptic commented 1 year ago

Might be the result of what I have forecasted in #246 but also reminds some issue on dandi-archive level but I have failed to find an open issue so might be not that.

Twice in a row we have cron job fails on that 000026 which now is acquiring some zarrs

anyio._backends._asyncio.ExceptionGroup: 6 exceptions were raised in the task group:
----------------------------
Traceback (most recent call last):
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 359, in needs_sync
    local_paths.remove(path)
KeyError: '0/1/29/72'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 560, in sync_zarr
    await zsync.run()
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 140, in run
    if not await self.needs_sync(client, last_sync, local_paths):
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 361, in needs_sync
    self.check_change(f"entry {path!r} added")
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 489, in check_change
    raise UnexpectedChangeError(
tools.backups2datalad.util.UnexpectedChangeError: Dandiset 000026: Zarr 92ab6451-8123-4a04-94b0-571ed9650d8d: entry '0/1/29/72' added, but Dandiset draft timestamp was not updated on server
more of traceback ``` 2023-04-03T01:50:14-0400 [WARNING ] dandi: A newer version (0.51.0) of dandi/dandi-cli is available. You are using 0.48.1 Traceback (most recent call last): File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/__main__.py", line 516, in main(_anyio_backend="asyncio") File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/asyncclick/core.py", line 1157, in __call__ return anyio.run(self._main, main, args, kwargs, **opts) File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run return asynclib.run(func, *args, **backend_options) File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run return native_run(wrapper(), debug=debug) File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/base_events.py", line 641, in run_until_complete return future.result() File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper return await func(*args) File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/asyncclick/core.py", line 1160, in _main return await main(*args, **kwargs) File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/asyncclick/core.py", line 1076, in main rv = await self.invoke(ctx) File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/asyncclick/core.py", line 1687, in invoke return await _process_result(await sub_ctx.command.invoke(sub_ctx)) File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/asyncclick/core.py", line 1434, in invoke return await ctx.invoke(self.callback, **ctx.params) File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/asyncclick/core.py", line 780, in invoke rv = await rv File "/mnt/backup/dandi/dandisets/tools/backups2datalad/__main__.py", line 111, in wrapped await f(datasetter, *args, **kwargs) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/__main__.py", line 213, in update_from_backup await datasetter.update_from_backup(dandisets, exclude=exclude) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 79, in update_from_backup report = await pool_amap( File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 178, in pool_amap async with anyio.create_task_group() as tg: File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__ raise exceptions[0] File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 171, in dowork outp = await func(inp) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 149, in update_dandiset changed = await self.sync_dataset( File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 186, in sync_dataset await syncer.sync_assets(error_on_change) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/syncer.py", line 36, in sync_assets self.report = await async_assets( File "/mnt/backup/dandi/dandisets/tools/backups2datalad/asyncer.py", line 471, in async_assets async with await open_git_annex( File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 660, in __aexit__ raise ExceptionGroup(exceptions) anyio._backends._asyncio.ExceptionGroup: 6 exceptions were raised in the task group: ---------------------------- Traceback (most recent call last): File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 359, in needs_sync local_paths.remove(path) KeyError: '0/1/29/72' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 560, in sync_zarr await zsync.run() File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 140, in run if not await self.needs_sync(client, last_sync, local_paths): File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 361, in needs_sync self.check_change(f"entry {path!r} added") File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 489, in check_change raise UnexpectedChangeError( tools.backups2datalad.util.UnexpectedChangeError: Dandiset 000026: Zarr 92ab6451-8123-4a04-94b0-571ed9650d8d: entry '0/1/29/72' added, but Dandiset draft timestamp was not updated on server ---------------------------- Traceback (most recent call last): File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_core/_subprocesses.py", line 74, in run_process async with create_task_group() as tg: File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__ raise exceptions[0] File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_core/_subprocesses.py", line 58, in drain_stream async for chunk in stream: File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/abc/_streams.py", line 126, in __anext__ return await self.receive() File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1010, in receive data = await self._stream.read(max_bytes) File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/streams.py", line 669, in read await self._wait_for_data('read') File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/streams.py", line 502, in _wait_for_data await self._waiter asyncio.exceptions.CancelledError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 589, in sync_zarr stats = await ds.get_stats(config=manager.config) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/adataset.py", line 401, in get_stats stored_stats = await self.get_stored_stats() File "/mnt/backup/dandi/dandisets/tools/backups2datalad/adataset.py", line 450, in get_stored_stats if stored_commit == await self.get_commit_hash(): File "/mnt/backup/dandi/dandisets/tools/backups2datalad/adataset.py", line 460, in get_commit_hash return await self.read_git("show", "-s", "--format=%H") File "/mnt/backup/dandi/dandisets/tools/backups2datalad/adataset.py", line 135, in read_git return await areadcmd( File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 211, in areadcmd r = await aruncmd(*args, **kwargs) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 205, in aruncmd return await anyio.run_process(argstrs, **kwargs) File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_core/_subprocesses.py", line 85, in run_process process.kill() File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1056, in kill self._process.kill() File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/subprocess.py", line 143, in kill self._transport.kill() File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/base_subprocess.py", line 153, in kill self._check_proc() File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/base_subprocess.py", line 142, in _check_proc raise ProcessLookupError() ProcessLookupError ---------------------------- Traceback (most recent call last): File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_core/_subprocesses.py", line 74, in run_process async with create_task_group() as tg: File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__ raise exceptions[0] File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_core/_subprocesses.py", line 58, in drain_stream async for chunk in stream: File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/abc/_streams.py", line 126, in __anext__ return await self.receive() File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1010, in receive data = await self._stream.read(max_bytes) File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/streams.py", line 669, in read await self._wait_for_data('read') File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/streams.py", line 502, in _wait_for_data await self._waiter asyncio.exceptions.CancelledError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 589, in sync_zarr stats = await ds.get_stats(config=manager.config) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/adataset.py", line 401, in get_stats stored_stats = await self.get_stored_stats() File "/mnt/backup/dandi/dandisets/tools/backups2datalad/adataset.py", line 450, in get_stored_stats if stored_commit == await self.get_commit_hash(): File "/mnt/backup/dandi/dandisets/tools/backups2datalad/adataset.py", line 460, in get_commit_hash return await self.read_git("show", "-s", "--format=%H") File "/mnt/backup/dandi/dandisets/tools/backups2datalad/adataset.py", line 135, in read_git return await areadcmd( File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 211, in areadcmd r = await aruncmd(*args, **kwargs) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 205, in aruncmd return await anyio.run_process(argstrs, **kwargs) File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_core/_subprocesses.py", line 85, in run_process process.kill() File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1056, in kill self._process.kill() File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/subprocess.py", line 143, in kill self._transport.kill() File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/base_subprocess.py", line 153, in kill self._check_proc() File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/base_subprocess.py", line 142, in _check_proc raise ProcessLookupError() ProcessLookupError ---------------------------- Traceback (most recent call last): File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_core/_subprocesses.py", line 74, in run_process async with create_task_group() as tg: File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__ raise exceptions[0] File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_core/_subprocesses.py", line 58, in drain_stream async for chunk in stream: File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/abc/_streams.py", line 126, in __anext__ return await self.receive() File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1010, in receive data = await self._stream.read(max_bytes) File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/streams.py", line 669, in read await self._wait_for_data('read') File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/streams.py", line 502, in _wait_for_data await self._waiter asyncio.exceptions.CancelledError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 589, in sync_zarr stats = await ds.get_stats(config=manager.config) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/adataset.py", line 401, in get_stats stored_stats = await self.get_stored_stats() File "/mnt/backup/dandi/dandisets/tools/backups2datalad/adataset.py", line 450, in get_stored_stats if stored_commit == await self.get_commit_hash(): File "/mnt/backup/dandi/dandisets/tools/backups2datalad/adataset.py", line 460, in get_commit_hash return await self.read_git("show", "-s", "--format=%H") File "/mnt/backup/dandi/dandisets/tools/backups2datalad/adataset.py", line 135, in read_git return await areadcmd( File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 211, in areadcmd r = await aruncmd(*args, **kwargs) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 205, in aruncmd return await anyio.run_process(argstrs, **kwargs) File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_core/_subprocesses.py", line 85, in run_process process.kill() File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1056, in kill self._process.kill() File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/subprocess.py", line 143, in kill self._transport.kill() File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/base_subprocess.py", line 153, in kill self._check_proc() File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/base_subprocess.py", line 142, in _check_proc raise ProcessLookupError() ProcessLookupError ---------------------------- Traceback (most recent call last): File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_core/_subprocesses.py", line 74, in run_process async with create_task_group() as tg: File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__ raise exceptions[0] File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_core/_subprocesses.py", line 58, in drain_stream async for chunk in stream: File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/abc/_streams.py", line 126, in __anext__ return await self.receive() File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1010, in receive data = await self._stream.read(max_bytes) File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/streams.py", line 669, in read await self._wait_for_data('read') File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/streams.py", line 502, in _wait_for_data await self._waiter asyncio.exceptions.CancelledError ... ```

needs to be troubleshooted and addressed .

jwodder commented 1 year ago

@yarikoptic I don't know how this happened, but somehow the backup of Zarr 92ab6451-8123-4a04-94b0-571ed9650d8d that was committed at Fri Mar 31 08:16:28 2023 -0400 ended up with a last_modified timestamp (which should be the latest LastModified field of all Zarr entries) equal to 2023-03-31T12:08:10+00:00, yet it also lacked the entry 0/1/29/72, which has a LastModified timestamp of 2023-03-31 10:10:49+00:00. Because --zarr-mode defaults to timestamp (which causes any Zarr entries with a LastModified before the Zarr backup's last_modified to not be synced), subsequent backups of the Zarr did not add the entry, and then when the backup was run with --mode verify, the absence of an entry on the server from the backup lead to an error.

Running a (non-verify) backup with --zarr-mode force should fix the problem.

yarikoptic commented 1 year ago

interesting, ok, running on drogon flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron.lock bash -c '/mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --zarr-mode force 000026'

yarikoptic commented 1 year ago

I think --zar-mode force run indeed fixed it, so let's consider addressed. Thank you @jwodder for the analysis and clear instructions on how to mitigate.

yarikoptic commented 9 months ago

it seems that issue persists/came back. Got two runs failing

Date: Sun, 01 Oct 2023 02:22:36 -0400                                                                                                                                                                                                                              
From: Cron Daemon <root@drogon.datalad.org>                                                                                                                                                                                                                        
To: dandi@drogon.datalad.org                                                                                                                                                                                                                                       
Subject: Cron <dandi@drogon> chronic flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron.lock bash -c '/mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --mode verify'                                                                              

2023-10-01T01:50:22-0400 [WARNING ] dandi: A newer version (0.56.2) of dandi/dandi-cli is available. You are using 0.55.1                                                                                                                                          
2023-10-01T01:57:20-0400 [WARNING ] backups2datalad: Retrying GET request to /dandisets/000153/versions/ in 1.007122 seconds as it raised RemoteProtocolError: Server disconnected without sending a response.                                                     
2023-10-01T01:58:02-0400 [ERROR   ] backups2datalad: Job failed on input <Dandiset 000026/draft>:                                                                                                                                                                  
Traceback (most recent call last):                                                                                                                                                                                                                                 
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 173, in dowork                                                                                                                                                                         
    outp = await func(inp)                                                                                                                                                                                                                                         
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 151, in update_dandiset                                                                                                                                                             
    changed = await self.sync_dataset(                                                                                                                                                                                                                             
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 188, in sync_dataset                                                                                                                                                                
    await syncer.sync_assets(error_on_change)                                                                                                                                                                                                                      
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/syncer.py", line 36, in sync_assets                                                                                                                                                                      
    self.report = await async_assets(                                                                                                                                                                                                                              
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/asyncer.py", line 522, in async_assets                                                                                                                                                                   
    async with AsyncAnnex(ds.pathobj) as annex, httpx.AsyncClient(                                                                                                                                                                                                 
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__                                                                                                                                  
    raise exceptions[0]                                                                                                                                                                                                                                            
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 560, in sync_zarr                                                                                                                                                                         
    await zsync.run()                                                                                                                                                                                                                                              
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 140, in run                                                                                                                                                                               
    if not await self.needs_sync(client, last_sync, local_paths):                                                                                                                                                                                                  
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 361, in needs_sync                                                                                                                                                                        
    self.check_change(f"entry {path!r} added")                                                                                                                                                                                                                     
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 489, in check_change                                                                                                                                                                      
    raise UnexpectedChangeError(                                                                                                                                                                                                                                   
tools.backups2datalad.util.UnexpectedChangeError: Dandiset 000026: Zarr ddaf60c9-9ff3-406e-bb50-943edcca33c1: entry '0/0/14/37' added, but Dandiset draft timestamp was not updated on server

and then

Traceback (most recent call last):                                                                                                                                                                                                                                 
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 560, in sync_zarr                                                                                                                                                                         
    await zsync.run()                                                                                                                                                                                                                                              
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 140, in run                                                                                                                                                                               
    if not await self.needs_sync(client, last_sync, local_paths):                                                                                                                                                                                                  
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 361, in needs_sync                                                                                                                                                                        
    self.check_change(f"entry {path!r} added")                                                                                                                                                                                                                     
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 489, in check_change                                                                                                                                                                      
    raise UnexpectedChangeError(                                                                                                                                                                                                                                   
tools.backups2datalad.util.UnexpectedChangeError: Dandiset 000026: Zarr 3b8c730d-147f-49e3-8a7a-de3b215b8d0f: entry '0/0/33/59' added, but Dandiset draft timestamp was not updated on server

I will try to redo that invocation flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron.lock bash -c '/mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --zarr-mode force 000026' to see if helps

yarikoptic commented 9 months ago

I think that run helped but now run ran into it again! There seems to be some logic issue somewhere... will need to analyze later. Meanwhile will try to force again

yarikoptic commented 9 months ago

apparently it is not sufficient to bring the dataset into correct state:

here is the error I got originally in email from cron job: 

tools.backups2datalad.util.UnexpectedChangeError: Dandiset 000026: Zarr fe01bdb8-9e6c-4fca-8607-7cbd417a4a35: entry '0/0/29/20' added, but Dandiset draft timestamp was not updated on server

rerunning the --zarr-mode force resulted in no new commits,

the zarr is said to be complete

❯ curl -X 'GET' 'https://api.dandiarchive.org/api/zarr/fe01bdb8-9e6c-4fca-8607-7cbd417a4a35/' -H 'accept: application/json'
{"name":"sub-I59/ses-SPIM/micr/sub-I59_ses-SPIM_sample-BrocaAreaS21_stain-NeuN_SPIM.ome.zarr","dandiset":"000026","zarr_id":"fe01bdb8-9e6c-4fca-8607-7cbd417a4a35","status":"Complete","checksum":"00a168a8ecf111c307f9930d2a44f0a5-16267--25462074744","file_count":16267,"size":25462074744}

locally we do not have that path in the zarr

(dandisets) dandi@drogon:/mnt/backup/dandi/dandisets/000026$ ls -l ../../dandizarrs/fe01bdb8-9e6c-4fca-8607-7cbd417a4a35/0/0/29/20
ls: cannot access '../../dandizarrs/fe01bdb8-9e6c-4fca-8607-7cbd417a4a35/0/0/29/20': No such file or directory

but it does exist on the archive

(dandisets) dandi@drogon:/mnt/backup/dandi/dandisets/000026$ curl -X 'GET' \
>   'https://api.dandiarchive.org/api/zarr/fe01bdb8-9e6c-4fca-8607-7cbd417a4a35/files/?prefix=0%2F0%2F29%2F20&limit=10&download=false' \
>   -H 'accept: application/json'
{"next":null,"results":[{"Key":"0/0/29/20","LastModified":"2023-10-01 07:38:47+00:00","ETag":"fe25e5f3c7e2a7c8aa1324ad464693f0","Size":2217884}]}

so the "LastModified":"2023-10-01 07:38:47+00:00" and the last commit in that zarr we have

(dandisets) dandi@drogon:/mnt/backup/dandi/dandisets/000026$ git -C  ../../dandizarrs/fe01bdb8-9e6c-4fca-8607-7cbd417a4a35/ show | head
commit 60f90894cad66d3c65b212c750dd52c9bf673dd2
Author: DANDI User <info@dandiarchive.org>
Date:   Sun Oct 1 12:39:02 2023 +0000

    [backups2datalad] 4532 files added, checksum updated

so after that date (but again -- missing that path). So it seems that we ended up in an inconsistent state...

edit 1: Running now with force for overall check: flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron.lock bash -c '/mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --zarr-mode force --mode force 000026'

yarikoptic commented 8 months ago

we got similar one but on .nwb in some new dandiset, there is an email (E1qvYjF-00D6eF-Kr@drogon.datalad.org) with log containing the flood of

Traceback (most recent call last):                                                                                                                                                                                   
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/asyncer.py", line 251, in process_asset                                                                                                                    
    raise UnexpectedChangeError(                                                                                                                                                                                     
tools.backups2datalad.util.UnexpectedChangeError: Dandiset 000640: Metadata for asset sub-176/sub-176_ses-20200923.nwb was changed/added but draft timestamp was not updated on server                               
----------------------------                                                                                                                                                                                         
Traceback (most recent call last):                                                                                                                                                                                   
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/asyncer.py", line 251, in process_asset                                                                                                                    
    raise UnexpectedChangeError(                                                                                                                                                                                     
tools.backups2datalad.util.UnexpectedChangeError: Dandiset 000640: Metadata for asset sub-176/sub-176_ses-20200929.nwb was changed/added but draft timestamp was not updated on server

Before then dandiset was empty

dandi@drogon:/mnt/backup/dandi/dandisets/000640$ git lg
* 99e74cf - (HEAD -> draft, github/draft) [backups2datalad] Only some metadata updates (5 weeks ago) [DANDI User]
* c791088 - Instruct annex to add text files to Git (5 weeks ago) [DANDI User]
* d92ea04 - [DATALAD] new dataset (5 weeks ago) [DANDI User]
and diff (now reset) to dandiset.yaml was ```shell dandi@drogon:/mnt/backup/dandi/dandisets/000640$ git diff --cached diff --git a/dandiset.yaml b/dandiset.yaml index 0579ef8..4d69bc7 100644 --- a/dandiset.yaml +++ b/dandiset.yaml @@ -6,9 +6,21 @@ access: - schemaKey: AccessRequirements status: dandi:OpenAccess assetsSummary: - numberOfBytes: 0 - numberOfFiles: 0 + approach: [] + dataStandard: + - identifier: RRID:SCR_015242 + name: Neurodata Without Borders (NWB) + schemaKey: StandardsType + measurementTechnique: [] + numberOfBytes: 14849530840 + numberOfFiles: 214 + numberOfSubjects: 14 schemaKey: AssetsSummary + species: + - identifier: http://purl.obolibrary.org/obo/NCBITaxon_10116 + name: Rattus norvegicus - Norway rat + schemaKey: SpeciesType + variableMeasured: [] citation: Koloski, Miranda (2023) 32-CH Local Field Potential Data During Probabilistic Reversal Learning Task (Version draft) [Data set]. DANDI archive. https://doi.org/10.80507/dandi.123456/0.123456.1234 contributor: @@ -20,7 +32,7 @@ contributor: - dcite:ContactPerson schemaKey: Person dateCreated: '2023-09-22T00:16:23.478717+00:00' -datePublished: '2023-09-22T00:16:41.387621+00:00' +datePublished: '2023-10-25T07:39:28.735955+00:00' description: Male Long Evans Rats 7-12 months of age. 24 total animals. 256 sessions. 13 animals received a frontal traumatic brain injury (controlled cortical impact). 11 animals received Sham (no craniotomy no TBI) conditions. One week later, rats @@ -43,13 +55,13 @@ manifestLocation: - https://api.dandiarchive.org/api/dandisets/000640/versions/draft/assets/ name: 32-CH Local Field Potential Data During Probabilistic Reversal Learning Task publishedBy: - endDate: '2023-09-22T00:16:41.387621+00:00' - id: urn:uuid:7e1289a2-dd48-4c5f-bd6c-5dfc120f38c2 + endDate: '2023-10-25T07:39:28.735955+00:00' + id: urn:uuid:8b37e074-dfe9-46e1-9ca7-86cd566fb2cd name: DANDI publish schemaKey: PublishActivity - startDate: '2023-09-22T00:16:41.387621+00:00' + startDate: '2023-10-25T07:39:28.735955+00:00' wasAssociatedWith: - - id: urn:uuid:50e8ca2c-95bb-44b4-83cf-7c5ccefdd47d + - id: urn:uuid:b5d9f6fc-1244-4c96-bcdc-f4645a3505c3 identifier: RRID:SCR_017571 name: DANDI API schemaKey: Software ```

edit: running flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron.lock bash -c '/mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron 000640' resulted in clean execution and a commit which added all those assets.

here is the diff between diffs of above and committed changes to dandiset.yaml -- nothing about date/time but has difference in stats, so dandiset was still "being uploaded to" ```diff --- a/dandiset.yaml +++ b/dandiset.yaml @@ -6,9 +6,21 @@ access: @@ -14,9 +20,9 @@ + name: Neurodata Without Borders (NWB) + schemaKey: StandardsType + measurementTechnique: [] -+ numberOfBytes: 14849530840 -+ numberOfFiles: 214 -+ numberOfSubjects: 14 ++ numberOfBytes: 18708666000 ++ numberOfFiles: 247 ++ numberOfSubjects: 19 schemaKey: AssetsSummary + species: + - identifier: http://purl.obolibrary.org/obo/NCBITaxon_10116 @@ -53,4 +59,3 @@ identifier: RRID:SCR_017571 name: DANDI API schemaKey: Software - ```
yarikoptic commented 8 months ago
fresh one on 000026 ```shell 2023-10-27T01:50:10-0400 [WARNING ] dandi: A newer version (0.56.2) of dandi/dandi-cli is available. You are using 0.55.1 2023-10-27T02:06:48-0400 [ERROR ] backups2datalad: Job failed on input : Traceback (most recent call last): File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 173, in dowork outp = await func(inp) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 151, in update_dandiset changed = await self.sync_dataset( File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 188, in sync_dataset await syncer.sync_assets(error_on_change) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/syncer.py", line 36, in sync_assets self.report = await async_assets( File "/mnt/backup/dandi/dandisets/tools/backups2datalad/asyncer.py", line 522, in async_assets async with AsyncAnnex(ds.pathobj) as annex, httpx.AsyncClient( File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__ raise exceptions[0] File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 560, in sync_zarr await zsync.run() File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 140, in run if not await self.needs_sync(client, last_sync, local_paths): File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 361, in needs_sync self.check_change(f"entry {path!r} added") File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 489, in check_change raise UnexpectedChangeError( tools.backups2datalad.util.UnexpectedChangeError: Dandiset 000026: Zarr 5c37c233-222f-4e60-96e7-a7536e08ef61: entry '0/1/31/70' added, but Dandiset draft timestamp was not updated on server 2023-10-27T02:10:36-0400 [WARNING ] backups2datalad: Retrying GET request to /dandisets/000053/versions/draft/ in 1.009969 seconds as it raised ConnectError: All connection attempts failed 2023-10-27T02:10:38-0400 [WARNING ] backups2datalad: Retrying GET request to /dandisets/000049/versions/ in 1.004621 seconds as it raised ConnectError: All connection attempts failed 2023-10-27T02:10:43-0400 [WARNING ] backups2datalad: Retrying GET request to /dandisets/000051/versions/ in 1.007710 seconds as it raised ConnectError: All connection attempts failed 2023-10-27T02:57:02-0400 [WARNING ] backups2datalad: Retrying GET request to /dandisets/000292/versions/ in 0.959762 seconds as it raised ConnectError: All connection attempts failed 2023-10-27T03:48:06-0400 [WARNING ] backups2datalad: Failed [rc=1]: git -c receive.autogc=0 -c gc.auto=0 annex add dandiset.yaml [cwd=/mnt/backup/dandi/dandisets/000623] Stdout: Stderr: fatal: git show-ref: bad ref refs/heads/git-annex (0000000000000000000000000000000000000000) fatal: git show-ref: bad ref refs/heads/git-annex (0000000000000000000000000000000000000000) error: object file .git/objects/b6/d1d20e316838110ae3385d8aeff437db79fe9f is empty fatal: b6d1d20e316838110ae3385d8aeff437db79fe9f is not a valid object git-annex: user error (git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","commit-tree","b6d1d20e316838110ae3385d8aeff437db79fe9f","--no-gpg-sign"] exited 128) 2023-10-27T03:48:06-0400 [ERROR ] backups2datalad: Job failed on input : Traceback (most recent call last): File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 173, in dowork outp = await func(inp) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 151, in update_dandiset changed = await self.sync_dataset( File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 187, in sync_dataset await update_dandiset_metadata(dandiset, ds, log=manager.log) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/util.py", line 185, in update_dandiset_metadata await ds.add(dandiset_metadata_file) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/adataset.py", line 233, in add await self.call_annex("add", path) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/adataset.py", line 146, in call_annex await aruncmd("git", *GIT_OPTIONS, "annex", *args, cwd=self.path, **kwargs) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 217, in aruncmd raise e File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 202, in aruncmd r = await anyio.run_process(argstrs, **kwargs) File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_core/_subprocesses.py", line 90, in run_process raise CalledProcessError(cast(int, process.returncode), command, output, errors) subprocess.CalledProcessError: Command '['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'add', 'dandiset.yaml']' returned non-zero exit status 1. 2023-10-27T03:58:45-0400 [ERROR ] backups2datalad: An error occurred: Traceback (most recent call last): File "/mnt/backup/dandi/dandisets/tools/backups2datalad/__main__.py", line 111, in wrapped await f(datasetter, *args, **kwargs) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/__main__.py", line 213, in update_from_backup await datasetter.update_from_backup(dandisets, exclude=exclude) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 97, in update_from_backup raise RuntimeError( RuntimeError: Backups for 2 Dandisets failed Logs saved to /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2023.10.27.05.50.09Z.log action summary: publish (notneeded: 2) ```

edit: then it "cleared up" with some next run... may be related to the fact that zarr lifecycle is yet "in flux" and might need to get adjusted, see https://github.com/dandi/dandi-archive/pull/1698 . Although above case with .nwb suggests that it might be a generic issue with asset lifecycle although could be some other altogether since we do not typically observe them for nwb

yarikoptic commented 5 months ago

more of fresh ones:

000344 README.md ```shell 2024-02-09T02:14:49-0500 [ERROR ] backups2datalad: Job failed on input : + Exception Group Traceback (most recent call last): | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/aioutil.py", line 176, in dowork | outp = await func(inp) | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 159, in update_dandiset | changed = await self.sync_dataset( | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 203, in sync_dataset | await syncer.sync_assets() | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/syncer.py", line 77, in sync_assets | report = await async_assets( | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 498, in async_assets | async with ( | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 664, in __aexit__ | raise BaseExceptionGroup( | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob | raise UnexpectedChangeError( | backups2datalad.util.UnexpectedChangeError: Dandiset 000344: Metadata for asset README.md was changed/added but draft timestamp was not updated on server +------------------------------------ ```
000729 sub-256/dwi/sub-256_dwi.json ```shell 2024-02-09T02:17:53-0500 [ERROR ] backups2datalad: Job failed on input : + Exception Group Traceback (most recent call last): | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/aioutil.py", line 176, in dowork | outp = await func(inp) | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 159, in update_dandiset | changed = await self.sync_dataset( | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 203, in sync_dataset | await syncer.sync_assets() | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/syncer.py", line 77, in sync_assets | report = await async_assets( | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 498, in async_assets | async with ( | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 664, in __aexit__ | raise BaseExceptionGroup( | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (6 sub-exceptions) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob | raise UnexpectedChangeError( | backups2datalad.util.UnexpectedChangeError: Dandiset 000729: Metadata for asset derivatives/DWIprep_V.1.0/dataset_description.json was changed/added but draft timestamp was not updated on server +---------------- 2 ---------------- | Traceback (most recent call last): | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob | raise UnexpectedChangeError( | backups2datalad.util.UnexpectedChangeError: Dandiset 000729: Metadata for asset dataset_description.json was changed/added but draft timestamp was not updated on server +---------------- 3 ---------------- | Traceback (most recent call last): | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob | raise UnexpectedChangeError( | backups2datalad.util.UnexpectedChangeError: Dandiset 000729: Metadata for asset participants.json was changed/added but draft timestamp was not updated on server +---------------- 4 ---------------- | Traceback (most recent call last): | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob | raise UnexpectedChangeError( | backups2datalad.util.UnexpectedChangeError: Dandiset 000729: Metadata for asset participants.tsv was changed/added but draft timestamp was not updated on server +---------------- 5 ---------------- | Traceback (most recent call last): | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob | raise UnexpectedChangeError( | backups2datalad.util.UnexpectedChangeError: Dandiset 000729: Metadata for asset sub-243/dwi/sub-243_dwi.json was changed/added but draft timestamp was not updated on server +---------------- 6 ---------------- | Traceback (most recent call last): | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob | raise UnexpectedChangeError( | backups2datalad.util.UnexpectedChangeError: Dandiset 000729: Metadata for asset sub-256/dwi/sub-256_dwi.json was changed/added but draft timestamp was not updated on server +------------------------------------ ```
000731 -- multiple files ```shell 2024-02-09T02:17:56-0500 [ERROR ] backups2datalad: Job failed on input : + Exception Group Traceback (most recent call last): | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/aioutil.py", line 176, in dowork | outp = await func(inp) | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 159, in update_dandiset | changed = await self.sync_dataset( | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 203, in sync_dataset | await syncer.sync_assets() | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/syncer.py", line 77, in sync_assets | report = await async_assets( | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 498, in async_assets | async with ( | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 664, in __aexit__ | raise BaseExceptionGroup( | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (4 sub-exceptions) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob | raise UnexpectedChangeError( | backups2datalad.util.UnexpectedChangeError: Dandiset 000731: Metadata for asset dataset_description.json was changed/added but draft timestamp was not updated on server +---------------- 2 ---------------- | Traceback (most recent call last): | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob | raise UnexpectedChangeError( | backups2datalad.util.UnexpectedChangeError: Dandiset 000731: Metadata for asset derivatives/dataset_description.json was changed/added but draft timestamp was not updated on server +---------------- 3 ---------------- | Traceback (most recent call last): | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob | raise UnexpectedChangeError( | backups2datalad.util.UnexpectedChangeError: Dandiset 000731: Metadata for asset participants.json was changed/added but draft timestamp was not updated on server +---------------- 4 ---------------- | Traceback (most recent call last): | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob | raise UnexpectedChangeError( | backups2datalad.util.UnexpectedChangeError: Dandiset 000731: Metadata for asset participants.tsv was changed/added but draft timestamp was not updated on server ```

@jwodder -- please troubleshoot this further -- we might need more output to be provided (e.g. what in metadata has changed to be listed in the exception or some ERROR log) to be able to actually troubleshoot whenever it happens next.

the issue is currently easily reproducible so good time to get that done: ```shell dandi@drogon:~$ flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron-nonzarr.lock bash dandi@drogon:~$ crontab -l | grep backup.*veri 50 1 * * * chronic flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron-nonzarr.lock bash -c '/mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --mode verify' dandi@drogon:~$ /mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --mode verify 000731 2024-02-09T09:42:45-0500 [ERROR ] backups2datalad: Job failed on input : + Exception Group Traceback (most recent call last): | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/aioutil.py", line 176, in dowork | outp = await func(inp) | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 159, in update_dandiset | changed = await self.sync_dataset( | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 203, in sync_dataset | await syncer.sync_assets() | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/syncer.py", line 77, in sync_assets | report = await async_assets( | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 498, in async_assets | async with ( | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 664, in __aexit__ | raise BaseExceptionGroup( | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (4 sub-exceptions) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob | raise UnexpectedChangeError( | backups2datalad.util.UnexpectedChangeError: Dandiset 000731: Metadata for asset dataset_description.json was changed/added but draft timestamp was not updated on server +---------------- 2 ---------------- | Traceback (most recent call last): | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob | raise UnexpectedChangeError( | backups2datalad.util.UnexpectedChangeError: Dandiset 000731: Metadata for asset derivatives/dataset_description.json was changed/added but draft timestamp was not updated on server +---------------- 3 ---------------- | Traceback (most recent call last): | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob | raise UnexpectedChangeError( | backups2datalad.util.UnexpectedChangeError: Dandiset 000731: Metadata for asset participants.json was changed/added but draft timestamp was not updated on server +---------------- 4 ---------------- | Traceback (most recent call last): | File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob | raise UnexpectedChangeError( | backups2datalad.util.UnexpectedChangeError: Dandiset 000731: Metadata for asset participants.tsv was changed/added but draft timestamp was not updated on server +------------------------------------ 2024-02-09T09:42:45-0500 [ERROR ] backups2datalad: An error occurred: Traceback (most recent call last): File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/__main__.py", line 119, in wrapped await f(datasetter, *args, **kwargs) File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/__main__.py", line 228, in update_from_backup await datasetter.update_from_backup(dandisets, exclude=exclude) File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 94, in update_from_backup raise RuntimeError( RuntimeError: Backups for 1 Dandiset failed Logs saved to /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.02.09.14.42.44Z.log ```
yarikoptic commented 4 months ago

those recent were all private and we required --force 'full rerun, see