search

dandi / dandisets

730 Dandisets, 807.1 TB total. DataLad super-dataset of all Dandisets from https://github.com/dandisets
10 stars 0 forks source link

locking complaints + some files not added/committed for 000026 #346

Closed yarikoptic closed 1 year ago

yarikoptic commented 1 year ago

345 made our process not crash BUT brought another interesting/odd behavior ...

complains about lock file but not crash, and then does not add/commit some files ```shell (base) dandi@drogon:/mnt/backup/dandi/dandisets$ flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron.lock bash -c '/mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron' add dandiset.yaml (non-large file; adding content to git repository) ok (recording state in git...) fatal: Unable to create '/mnt/backup/dandi/dandisets/000026/.git/index.lock': File exists. Another git process seems to be running in this repository, e.g. an editor opened by 'git commit'. Please make sure all processes are terminated then try again. If it still fails, a git process may have crashed in this repository earlier: remove the file manually to continue. fatal: Unable to create '/mnt/backup/dandi/dandisets/000026/.git/index.lock': File exists. Another git process seems to be running in this repository, e.g. an editor opened by 'git commit'. Please make sure all processes are terminated then try again. If it still fails, a git process may have crashed in this repository earlier: remove the file manually to continue. fatal: Unable to create '/mnt/backup/dandi/dandisets/000026/.git/index.lock': File exists. Another git process seems to be running in this repository, e.g. an editor opened by 'git commit'. Please make sure all processes are terminated then try again. If it still fails, a git process may have crashed in this repository earlier: remove the file manually to continue. fatal: Unable to create '/mnt/backup/dandi/dandisets/000026/.git/index.lock': File exists. ... create mode 100644 sub-KC001/ses-MRI/anat/sub-KC001_ses-MRI_echo-2_flip-3_VFA.json create mode 100644 sub-KC001/ses-MRI/anat/sub-KC001_ses-MRI_echo-3_flip-1_VFA.json create mode 100644 sub-KC001/ses-MRI/anat/sub-KC001_ses-MRI_echo-3_flip-2_VFA.json create mode 100644 sub-KC001/ses-MRI/anat/sub-KC001_ses-MRI_echo-3_flip-3_VFA.json create mode 100644 sub-KC001/ses-MRI/anat/sub-KC001_ses-MRI_echo-4_flip-2_VFA.json create mode 100644 sub-KC001/ses-MRI/anat/sub-KC001_ses-MRI_echo-4_flip-3_VFA.json create mode 100644 sub-KC001/ses-MRI/anat/sub-KC001_ses-MRI_echo-4_flip-4_VFA.json 2023-04-19T13:01:30-0400 [ERROR ] backups2datalad: Job failed on input : Traceback (most recent call last): File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 172, in dowork outp = await func(inp) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 141, in update_dandiset changed = await self.sync_dataset(dandiset, ds, dmanager) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 186, in sync_dataset await syncer.sync_assets(error_on_change) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/syncer.py", line 36, in sync_assets self.report = await async_assets( File "/mnt/backup/dandi/dandisets/tools/backups2datalad/asyncer.py", line 599, in async_assets await ds.commit( File "/mnt/backup/dandi/dandisets/tools/backups2datalad/adataset.py", line 191, in commit raise RuntimeError( RuntimeError: /mnt/backup/dandi/dandisets/000026 is still dirty after committing. Please check if all changes were staged. 2023-04-19T13:01:30-0400 [ERROR ] backups2datalad: An error occurred: Traceback (most recent call last): File "/mnt/backup/dandi/dandisets/tools/backups2datalad/__main__.py", line 111, in wrapped await f(datasetter, *args, **kwargs) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/__main__.py", line 213, in update_from_backup await datasetter.update_from_backup(dandisets, exclude=exclude) File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 95, in update_from_backup raise RuntimeError( RuntimeError: Backups for 1 Dandiset failed Logs saved to /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2023.04.19.16.49.25Z.log (base) dandi@drogon:/mnt/backup/dandi/dandisets$ git -C 000026 status On branch draft Your branch is ahead of 'github/draft' by 1 commit. (use "git push" to publish your local commits) Untracked files: (use "git add ..." to include in what will be committed) derivatives/PSOCT_postprocessing_code/PSOCT_code/lipofuscin_segmentation.m~ derivatives/PSOCT_postprocessing_code/cortical_thickness/LineSegments.mat derivatives/PSOCT_postprocessing_code/cortical_thickness/MLI_thickness_MEX.mexa64 derivatives/PSOCT_postprocessing_code/cortical_thickness/MLI_thickness_MEX.mexmaci64 derivatives/PSOCT_postprocessing_code/cortical_thickness/MLI_thickness_MEX.mexw64 derivatives/PSOCT_postprocessing_code/cortical_thickness/cortical_thickness_3D.m~ sub-I52/ses-MRI/anat/sub-I52_ses-MRI_echo-1_flip-1_VFA.json sub-I52/ses-MRI/anat/sub-I52_ses-MRI_echo-2_flip-1_VFA.json sub-I52/ses-MRI/anat/sub-I52_ses-MRI_echo-2_flip-2_VFA.json sub-I52/ses-MRI/anat/sub-I52_ses-MRI_echo-2_flip-3_VFA.json sub-I52/ses-MRI/anat/sub-I52_ses-MRI_echo-3_flip-3_VFA.json sub-I52/ses-MRI/anat/sub-I52_ses-MRI_echo-4_flip-1_VFA.json sub-I52/ses-MRI/anat/sub-I52_ses-MRI_echo-4_flip-2_VFA.json sub-I52/ses-MRI/anat/sub-I52_ses-MRI_echo-4_flip-3_VFA.json sub-I53/ses-MRI/anat/sub-I53_ses-MRI_echo-1_flip-2_VFA.json sub-I53/ses-MRI/anat/sub-I53_ses-MRI_echo-2_flip-1_VFA.json sub-I53/ses-MRI/anat/sub-I53_ses-MRI_echo-3_flip-1_VFA.json sub-I53/ses-MRI/anat/sub-I53_ses-MRI_echo-3_flip-2_VFA.json sub-I53/ses-MRI/anat/sub-I53_ses-MRI_echo-4_flip-2_VFA.json sub-I55/ses-MRI/anat/sub-I55_ses-MRI_echo-1_flip-1_VFA.json sub-I55/ses-MRI/anat/sub-I55_ses-MRI_echo-1_flip-2_VFA.json sub-I55/ses-MRI/anat/sub-I55_ses-MRI_echo-2_flip-2_VFA.json sub-I55/ses-MRI/anat/sub-I55_ses-MRI_echo-3_flip-1_VFA.json sub-I55/ses-MRI/anat/sub-I55_ses-MRI_echo-3_flip-2_VFA.json sub-I55/ses-MRI/anat/sub-I55_ses-MRI_echo-4_flip-1_VFA.json sub-I55/ses-MRI/anat/sub-I55_ses-MRI_echo-4_flip-2_VFA.json sub-I55/ses-MRI/anat/sub-I55_ses-MRI_echo-4_flip-3_VFA.json sub-I58/ses-MRI/anat/sub-I58_ses-MRI_echo-1_flip-1_VFA.nii.gz sub-I58/ses-MRI/anat/sub-I58_ses-MRI_echo-1_flip-2_VFA.nii.gz sub-I58/ses-MRI/anat/sub-I58_ses-MRI_echo-1_flip-3_VFA.nii.gz sub-I58/ses-MRI/anat/sub-I58_ses-MRI_echo-1_flip-4_VFA.nii.gz sub-I58/ses-MRI/anat/sub-I58_ses-MRI_echo-2_flip-1_VFA.nii.gz sub-I58/ses-MRI/anat/sub-I58_ses-MRI_echo-2_flip-2_VFA.nii.gz sub-I58/ses-MRI/anat/sub-I58_ses-MRI_echo-2_flip-3_VFA.nii.gz sub-I58/ses-MRI/anat/sub-I58_ses-MRI_echo-2_flip-4_VFA.nii.gz sub-I58/ses-MRI/anat/sub-I58_ses-MRI_echo-3_flip-1_VFA.nii.gz sub-I58/ses-MRI/anat/sub-I58_ses-MRI_echo-3_flip-2_VFA.nii.gz sub-I58/ses-MRI/anat/sub-I58_ses-MRI_echo-3_flip-3_VFA.nii.gz sub-I58/ses-MRI/anat/sub-I58_ses-MRI_echo-3_flip-4_VFA.nii.gz sub-I58/ses-MRI/anat/sub-I58_ses-MRI_echo-4_flip-1_VFA.nii.gz sub-I58/ses-MRI/anat/sub-I58_ses-MRI_echo-4_flip-2_VFA.nii.gz sub-I58/ses-MRI/anat/sub-I58_ses-MRI_echo-4_flip-3_VFA.nii.gz sub-I58/ses-MRI/anat/sub-I58_ses-MRI_echo-4_flip-4_VFA.nii.gz sub-I61/ses-MRI/anat/sub-I61_ses-MRI_echo-1_flip-1_VFA.nii.gz sub-I61/ses-MRI/anat/sub-I61_ses-MRI_echo-1_flip-2_VFA.nii.gz sub-I61/ses-MRI/anat/sub-I61_ses-MRI_echo-1_flip-3_VFA.nii.gz sub-I61/ses-MRI/anat/sub-I61_ses-MRI_echo-1_flip-4_VFA.nii.gz sub-I61/ses-MRI/anat/sub-I61_ses-MRI_echo-2_flip-1_VFA.nii.gz sub-I61/ses-MRI/anat/sub-I61_ses-MRI_echo-2_flip-2_VFA.nii.gz sub-I61/ses-MRI/anat/sub-I61_ses-MRI_echo-2_flip-3_VFA.nii.gz sub-I61/ses-MRI/anat/sub-I61_ses-MRI_echo-2_flip-4_VFA.nii.gz sub-I61/ses-MRI/anat/sub-I61_ses-MRI_echo-3_flip-1_VFA.nii.gz sub-I61/ses-MRI/anat/sub-I61_ses-MRI_echo-3_flip-2_VFA.nii.gz sub-I61/ses-MRI/anat/sub-I61_ses-MRI_echo-3_flip-3_VFA.nii.gz sub-I61/ses-MRI/anat/sub-I61_ses-MRI_echo-3_flip-4_VFA.nii.gz sub-I61/ses-MRI/anat/sub-I61_ses-MRI_echo-4_flip-1_VFA.nii.gz sub-I61/ses-MRI/anat/sub-I61_ses-MRI_echo-4_flip-2_VFA.nii.gz sub-I61/ses-MRI/anat/sub-I61_ses-MRI_echo-4_flip-3_VFA.nii.gz sub-I61/ses-MRI/anat/sub-I61_ses-MRI_echo-4_flip-4_VFA.nii.gz sub-KC001/ses-MRI/anat/sub-KC001_ses-MRI_echo-1_flip-2_VFA.json sub-KC001/ses-MRI/anat/sub-KC001_ses-MRI_echo-1_flip-4_VFA.json sub-KC001/ses-MRI/anat/sub-KC001_ses-MRI_echo-2_flip-2_VFA.json sub-KC001/ses-MRI/anat/sub-KC001_ses-MRI_echo-2_flip-4_VFA.json sub-KC001/ses-MRI/anat/sub-KC001_ses-MRI_echo-3_flip-4_VFA.json sub-KC001/ses-MRI/anat/sub-KC001_ses-MRI_echo-4_flip-1_VFA.json nothing added to commit but untracked files present (use "git add" to track) ```

my wild suspicion is that due to removed precommit call... so I will return it back quickly and try again

yarikoptic commented 1 year ago

did that in 1e24823928154e62077c5abb41dbb00ad9fcb094, have done a run and I thought it all worked out nice but then there were no commit in super dataset and neither of them was pushed. I vaguely remember complaining that there was output in stderr,... odd, but I will ignore for now.