jwodder
commented
2 months ago @yarikoptic This is because https://github.com/dandi/dandi-archive/issues/1996 is still unresolved.
Closed yarikoptic closed 2 weeks ago
jwodder
commented
2 months ago @yarikoptic This is because https://github.com/dandi/dandi-archive/issues/1996 is still unresolved.
yarikoptic
commented
2 months ago ah. So we have somewhere on datalad side some insufficient communication back to git-annex so instead of CHECKURL failed with no reason given we could get something like CHECKURL failed because of needing authentication or alike. Let's wait for a day or two on a possible resolution on dandi-archive side.
Just to make sure -- we have no easy way ATM to skip embargoed dandisets generally right? and in general not all need to be skipped really since it is only in the case of text files which we are trying to add directly to git (instead of git-annex) we do try to get their content and then fail this way. Right?
jwodder
commented
2 months ago @yarikoptic
we have no easy way ATM to skip embargoed dandisets generally right?
Correct.
and in general not all need to be skipped really since it is only in the case of text files which we are trying to add directly to
git(instead ofgit-annex) we do try to get their content and then fail this way. Right?
Yes, though this problem only seems to occur for 001089, so we could just exclude that Dandiset for now.
yarikoptic
commented
2 weeks ago 001089 did fine with --verify now, so closing
(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandisets$ duct flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron-nonzarr.lock bash -c '/mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --mode verify 001089'
duct is executing flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron-nonzarr.lock bash -c /mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --mode verify 001089...
Log files will be written to .duct/logs/2024.10.17T14.30.14-858882_
Logs saved to /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.10.17.18.30.24Z.log
publish(ok): . (dataset) [refs/heads/draft->github:refs/heads/draft f035485..39d6098]
action summary:
publish (notneeded: 1, ok: 1)
Already up to date.
publish(ok): . (dataset) [refs/heads/draft->github:refs/heads/draft d343483..00035ec]
action summary:
publish (notneeded: 1, ok: 1)
Exit Code: 0
Command: flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron-nonzarr.lock bash -c /mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --mode verify 001089
Log files location: .duct/logs/2024.10.17T14.30.14-858882_
Wall Clock Time: 43.454 sec sec
Memory Peak Usage (RSS): 46072 KiB
Memory Average Usage (RSS): 116366.952 KiB
Virtual Memory Peak Usage (VSZ): 61372 KiB
Virtual Memory Average Usage (VSZ): 205054411.810 KiB
Memory Peak Percentage: 0.0%
Memory Average Percentage: 0.067%
CPU Peak Usage: 133.3%
Average CPU Usage: 58.219%
Samples Collected: 42
Reports Written: 1
> '[' -s .git/tmp/stderr-2 ']'
> echo 'There was stderr from backup/datalad invocations:'
> cat .git/tmp/stderr-2
There was stderr from backup/datalad invocations:
> backups2datalad -l WARNING --backup-root /mnt/backup/dandi --config tools/backups2datalad.cfg.yaml update-from-backup --workers 5 -e '000(026|108|243)$' --mode verify 001089
> grep -v 'nothing to save, working tree clean'
> git pull --commit --no-edit
> datalad push -J 5
Just got from cron job