zarr sync could crash due to "files deleted, but Dandiset draft timestamp was not updated on server"

Date: Sat, 24 Feb 2024 01:57:43 -0500                                                                                                                                                                                
From: Cron Daemon <root@drogon.datalad.org>                                                                                                                                                                          
To: dandi@drogon.datalad.org                                                                                                                                                                                         
Subject: Cron <dandi@drogon> chronic flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron-nonzarr.lock bash -c                                                                                                      
        '/mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --mode verify'                                                                                                                                

2024-02-24T01:50:56-0500 [ERROR   ] backups2datalad: Job failed on input <Dandiset 000029/draft>:                                                                                                                    
  + Exception Group Traceback (most recent call last):                                                                                                                                                               
  |   File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/aioutil.py", line 177, in dowork                                                                                    
  |     outp = await func(inp)                                                                                                                                                                                       
  |   File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 159, in update_dandiset                                                                        
  |     changed = await self.sync_dataset(                                                                                                                                                                           
  |   File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 203, in sync_dataset                                                                           
  |     await syncer.sync_assets()                                                                                                                                                                                   
  |   File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/syncer.py", line 77, in sync_assets                                                                                 
  |     report = await async_assets(                                                                                                                                                                                 
  |   File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 499, in async_assets                                                                              
  |     async with (                                                                                                                                                                                                 
  |   File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 664, in __aexit__                                                                                
  |     raise BaseExceptionGroup(                                                                                                                                                                                    
  | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)                                                                                                                                 
  +-+---------------- 1 ----------------                                                                                                                                                                             
    | Traceback (most recent call last):                                                                                                                                                                             
    |   File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/zarr.py", line 551, in sync_zarr                                                                                  
    |     await zsync.run()                                                                                                                                                                                          
    |   File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/zarr.py", line 136, in run                                                                                        
    |     if not await self.needs_sync(client, last_sync, to_delete):                                                                                                                                                
    |   File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/zarr.py", line 371, in needs_sync                                                                                 
    |     self.check_change(f"{quantify(len(local_paths), 'file')} deleted")                                                                                                                                         
    |   File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/zarr.py", line 480, in check_change                                                                               
    |     raise UnexpectedChangeError(                                                                                                                                                                               
    | backups2datalad.util.UnexpectedChangeError: Dandiset 000029: Zarr fd6ab3ea-cff6-4006-a9bf-acfa5d983985: 197154 files deleted, but Dandiset draft timestamp was not updated on server                           
    +------------------------------------                                                                                                                                                                            
whereis: 2 failed                                                                                                                                                                                                    
2024-02-24T01:57:43-0500 [ERROR   ] backups2datalad: An error occurred:                                                                                                                                              
Traceback (most recent call last):                                                                                                                                                                                   
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/__main__.py", line 119, in wrapped                                                                                      
    await f(datasetter, *args, **kwargs)                                                                                                                                                                             
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/__main__.py", line 228, in update_from_backup                                                                           
    await datasetter.update_from_backup(dandisets, exclude=exclude)                                                                                                                                                  
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 94, in update_from_backup                                                                          
    raise RuntimeError(                                                                                                                                                                                              
RuntimeError: Backups for 1 Dandiset failed                                                                                                                                                                          
Logs saved to /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.02.24.06.50.11Z.log                                                                                                                        
action summary:                                                                                                                                                                                                      
  publish (notneeded: 2)                                                                                                                                                                                             
publish(ok): . (dataset) [refs/heads/draft->github:refs/heads/draft 79ecfea..27bfc60]                                                                                                                                
publish(ok): . (dataset) [refs/heads/git-annex->github:refs/heads/git-annex 0b5b7aa..197da80]                                                                                                                        
action summary:                                                                                                                                                                                                      
  publish (ok: 2)
I am thinking that
there might be a bug in dandi-archive that upon issuing DELETE from zarr no timestamp update which should probably happen right away whenever DELETE requested, and then whenever completed. Bug report (or PR) should be filed against dandi-archive, ideally with a little reproducer
edit: hard reset'ed 000029, zarr fd6ab3ea-cff6-4006-a9bf-acfa5d983985 repo was clean, now running dandi@drogon:~$ time flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron-nonzarr.lock bash -c '/mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --mode force 000029'
dandi / backups2datalad

zarr sync could crash due to "files deleted, but Dandiset draft timestamp was not updated on server" #33
