dandi / dandisets

755 Dandisets, 815.8 TB total. DataLad super-dataset of all Dandisets from https://github.com/dandisets
10 stars 0 forks source link

backup failed due to AssertionError: Commits both before and after #334

Closed yarikoptic closed 1 year ago

yarikoptic commented 1 year ago
Date: Thu, 23 Feb 2023 12:03:03 -0500                                                                                                                                                                                                                              
From: Cron Daemon <root@drogon.datalad.org>                                                                                                                                                                                                                        
To: dandi@drogon.datalad.org                                                                                                                                                                                                                                       
Subject: Cron <dandi@drogon> chronic flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron.lock bash -c '/mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron'                                                                                            

2023-02-23T12:02:14-0500 [WARNING ] dandi: A newer version (0.49.1) of dandi/dandi-cli is available. You are using 0.48.1                                                                                                                                          
2023-02-23T12:02:21-0500 [ERROR   ] backups2datalad: Job failed on input <Dandiset 000039/draft>:                                                                                                                                                                  
Traceback (most recent call last):                                                                                                                                                                                                                                 
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 171, in dowork                                                                                                                                                                         
    outp = await func(inp)                                                                                                                                                                                                                                         
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 158, in update_dandiset                                                                                                                                                             
    await self.tag_releases(                                                                                                                                                                                                                                       
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 280, in tag_releases                                                                                                                                                                
    await self.mkrelease(dandiset.for_version(v), ds, push=push, log=log)                                                                                                                                                                                          
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 361, in mkrelease                                                                                                                                                                   
    assert len(matching) < 2, (                                                                                                                                                                                                                                    
AssertionError: Commits both before and after 2023-02-23 12:16:32.760672+00:00 have matching asset metadata     

commit b1a4621c5da1da8afbc5e0fce768d34b4f57f0fc (github/draft, github/HEAD) Author: DANDI User info@dandiarchive.org Date: Thu Jan 20 17:07:22 2022 +0000

[backups2datalad] Only some metadata updates

.dandi/assets-state.json | 3 +++ 1 file changed, 3 insertions(+)


so indeed what changed was `assets-state.json` and `dandiset.yaml` -- I guess - legit situation?
- what are we do about that to unblock backup of that dandiset?  
jwodder commented 1 year ago

@yarikoptic This only happened because the .dandi/assets-state.json file was added in commit b1a4621 without any other changes. I think the simplest solution is to not error (but still emit a warning?) when multiple candidate commits for a tag have matching asset metadata and instead just use the first commit.

yarikoptic commented 1 year ago

sounds good to me.