Closed yarikoptic closed 9 months ago
@yarikoptic The "no need to populate" message is emitted if the value of the dandi.populated
Git option for the dandiset equals the commit hash of its current HEAD. dandi.populated
is updated after every successful run of git-annex copy
, which happens when populate
is run while dandi.populated
is out of date.
Do the dandisets in question have uncommitted changes? That would be one way for there to be files in the dandiset that haven't been pushed to Dropbox. Other than that, I don't know how this would have happened (aside from blaming git-annex). One possible way to address this would be to give populate
a --force
option for skipping the dandi.populated
check.
Do the dandisets in question have uncommitted changes?
One possible way to address this would be to give
populate
a--force
option for skipping thedandi.populated
check.
yes. Let's do it, besides may be making it into an option not a flag. update-from-backup
has --mode verify
, may be let's add a similar --mode force-fast
here?
But I personally failed to make it "fast" or copy at all (might just need to rest...): https://git-annex.branchable.com/bugs/copy_--fast_--from_--to_checks_destination_files/
may be let's add a similar
--mode force-fast
here?
Are you saying you want this new mode to pass --fast
to git-annex copy
even though you can't get it to work right?
ATM let's just wait for @joeyh follow up on that bug report since indeed I cannot make it work (do you see where I might have made a mistake?) -- I am even wondering if the reason why it doesn't work for me relates why we got some files not backed up...?
ok, troubleshooted and filed a dedicated https://git-annex.branchable.com/bugs/copy_--from_--to_does_not_copy_if_present_locally/
I think for a workaround:
annex copy --fast --to dandi-dandisets-dropbox; annex copy --fast --from web --to dandi-dandisets-dropbox
. I am running that + git annex drop --all
on some sample dandisets now--mode force-fast
which would trigger that composite call@yarikoptic Should the two copy
commands be run every time populate
and populate-zarrs
are run, or only when --mode force-fast
is given?
Should the two
copy
commands be run every timepopulate
andpopulate-zarrs
are run
no -- they should be run according to the logic you provided above. i.e. not every time but depending on dandi.populated
.
or only when
--mode force-fast
is given?
I think this mode should just pretty much disable the dandi.populated
check and proceed to triggering that copy
tandem.
@yarikoptic
git-annex copy
command is replaced by two copy
's, both invoked with --fast
, and if the only thing --mode force-fast
does is skip the dandi.populated
check, why name it "force-fast" instead of just "force"?--mode force[-fast]
to be --force[-fast]
instead. If you're worried about making a breaking change to the CLI later if & when more modes are added, those can be done via feature switches.
- If the current
git-annex copy
command is replaced by twocopy
's, both invoked with--fast
, and if the only thing--mode force-fast
does is skip thedandi.populated
check, why name it "force-fast" instead of just "force"?
we might later get "force"
which would do the same without --fast
option added -- it would be slower since would be more thoroughly checking presence on the destination. But I hope we would avoid needing it.
- I am very much not a fan of CLI options that only accept one specific required string as an argument; it would make much more sense for
--mode force[-fast]
to be--force[-fast]
instead. If you're worried about making a breaking change to the CLI later if & when more modes are added, those can be done via feature switches.
I guess could be indeed. I would not fight heavily here over it, especially since ATM this is all internal functionality and thus we can easily break interfaces, especially the ones for "emergency use" later on. So feel welcome proceed the way you feel better of. But in my opinion:
force
without --fast
), are ok.--flags
in favor options when extending--help
longer and possibly spreading around relevant to the same mode options, causing combination of flags to be conflicting, and thus overall less preferable ok, troubleshooted and filed a dedicated https://git-annex.branchable.com/bugs/copy_--from_--to_does_not_copy_if_present_locally/
fixed in 10.20230926-142-g6e3bcbf4dd .
FTR, did
for d in 00*; do echo $d; git -C $d annex move -J4 --fast --to dandi-dandisets-dropbox; done
to move what is local to dropbox first, and eventually transfers started to stall. I think the first one was:
000363
(recording state in git...)
move sub-440956/sub-440956_ses-20190207T120657_behavior+ecephys+image+ogen.nwb
move sub-440956/sub-440956_ses-20190208T133600_behavior+ecephys+ogen.nwb ok (to dandi-dandisets-dropbox...)
move sub-440957/sub-440957_ses-20190212T153751_behavior+ecephys+image+ogen.nwb (to dandi-dandisets-dropbox...)
move sub-440957/sub-440957_ses-20190211T143614_behavior+ecephys+image+ogen.nwb (to dandi-dandisets-dropbox...)
move sub-440957/sub-440957_ses-20190214T144611_behavior+ecephys+image+ogen.nwb (to dandi-dandisets-dropbox...)
move sub-440956/sub-440956_ses-20190209T150135_behavior+ecephys+image+ogen.nwb
2023/11/18 08:43:07 ERROR : : error listing: directory not found 2023/11/18 08:43:07 Failed to size with 2 errors: last error was: directory not found
Total objects: 1 Total size: 953.674 MBytes (1000000000 Bytes)
0% 953.67 MiB 618 MiB/s 11m12s
2023/11/18 08:73:08 ERROR7: : error l14T144611irectory not found 2023/11/18 08:(to dandi-dandisets-dropbox...) rs: last error was: directory not found
1% 1.86 GiB 472 MiB/s 9m47s
Total objects:61 Total si6e: 953.674 08T133600000000000 Bytes) ogen.nwb ok
Total objects: 1 Total size: 953.674 M2T153751000000000 Bytes)
0% 1.86 GiB 417 MiB/s 16m8s
move sub-440957/sub-440957_ses-20190211T143614_behavior+ecephys+image+ogen.nwb (to dandi-dandisets-dropbox...)
0% 953.67 MiB 618 MiB/s 11m12s
move sub-440957/sub-440957_ses-20190214T144611_behavior+ecephys+image+ogen.nwb (to dandi-dandisets-dropbox...)
move sub-440956/sub-440956_ses-20190209T150135_behavior+ecephys+image+ogen.nwb (to dandi-dandisets-dropbox...)
Transfer stalled
Total objects: 1 Total size: 953.674 MBytes (1000000000 Bytes)
2023/11/18 08:43:14 ERROR : : error listing: directory not found 2023/11/18 08:43:14 Failed to size with 2 errors: last error was: directory not found
move sub-440957/sub-440957_ses-20190211T143614_behavior+ecephys+image+ogen.nwb (to dandi-dandisets-dropbox...)
Transfer stalled
Transfer stalled
failed
move sub-440956/sub-440956_ses-20190207T120657_behavior+ecephys+image+ogen.nwb (to dandi-dandisets-dropbox...)
Transfer stalled
I interrupted for now... need to troubleshoot (may be finally unlimited dropbox became limited?)
FTR: that box is experiencing hard time across all layers... I am running btrfs filesystem resize 1:-30T /mnt/backup/
to start getting rid of old drives in LVM and composing at BTRFS level instead of LVM, but there are some reports of corruption errors etc... inquiring on BTRFS IRC. It also does healthcheck etc -- I will stop/disable that for now.
So may be those above also contribute to a little unstable operation above.
Also tagged/released new git-annex-remote-rclone and testing it (will install new debian pkg when in neurodebian) -- works nicely and does not make noise as much. On one sample - it did complain about stalled transfer but then ended up with ok:
dandi@drogon:/mnt/backup/dandi/dandisets/000477$ PATH=/home/dandi/proj/git-annex-remote-rclone:$PATH git annex move -J4 --fast --to dandi-dandisets-dropbox
move sub-SKKS092/sub-SKKS092_ses-20210304T122622_behavior+ophys.nwb (to dandi-dandisets-dropbox...) ok
move sub-SKKS092/sub-SKKS092_ses-20210217T090900_behavior+ophys.nwb (to dandi-dandisets-dropbox...) ok
move sub-SKKS091/sub-SKKS091_ses-20210518T174412_behavior+ophys.nwb (to dandi-dandisets-dropbox...) ok
move sub-SKKS097/sub-SKKS097_ses-20210217T120353_behavior+ophys.nwb (to dandi-dandisets-dropbox...)
Transfer stalled
ok
(recording state in git...)
although it is easy to see that some files were not pushed to the backup
I didn't check inside how we decide on either worth doing that
copy
...here is a sweep through all dandisets showing that we have quite a good number of assets which were not transferred + some odd unrelated error I am yet to check
```shell dandi@drogon:/mnt/backup/dandi/dandisets$ for ds in 000*; do n=$(git -C $ds annex find --in web --not --in dandi-dandisets-dropbox | grep -v '^|' | wc -l); [ "$n" == "0" ] || echo "$ds $n"; done 000026 6 000037 41 000059 49 000235 2 000238 4 000246 144 000363 159 000409 39 000410 1 000447 1 000458 3 000465 13 000467 508 000477 4 000488 3 000535 3 000540 7 000541 8 000544 2 000547 6 000549 2 000550 3 000552 9 000554 23 000559 11 000561 7 000568 28 000570 1 000571 3 000572 26 000574 17 000576 4 000579 13 000582 1 000618 32 error: object file .git/objects/b6/d1d20e316838110ae3385d8aeff437db79fe9f is empty fatal: b6d1d20e316838110ae3385d8aeff437db79fe9f is not a valid object git-annex: user error (git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","commit-tree","b6d1d20e316838110ae3385d8aeff437db79fe9f","--no-gpg-sign"] exited 128) 000624 14 000626 1 000628 31 000630 210 000631 2 000632 7 000635 4 000636 705 000637 12 000640 167 000674 3 000678 6 000687 5 000691 1 000692 9 000696 1 ```@jwodder -- what do you think, on how we could have end up in such a situation that not everything is copied to dropbox, so we could mitigate with better understanding of the situation?