Closed renelabounek closed 2 years ago
Look what has appeared at the spine-generic/data-multi-subject master title page. I do not think this is what we want to happen next time. Is it a result of the command?
git pull && git annex sync --no-content && git annex get .
@renelabounek this is one of the several unfortunate side effects of https://github.com/neuropoly/data-management/issues/67. As I understand it, git-annex was meant for a single person managing a sprawling collection of data, not many people managing a single collection of data. It tries to track the location of every copy of a file, instead of just looking in the origin repository. Unfortunately, besides being unnecessary for our use case, it does it in a separate branch called git-annex
. And your fork has an old version of git-annex
so that's why it can't find these files.
Keeping the two branches in lockstep is so difficult with git-annex
that we just decided to avoid it. Our workflow doesn't support taking PRs from forks, only from members of our team.
The quick fix is: if all you need is read-only access, delete your entire copy and reclone from this repo, not from your fork. Does that work for you?
If you're trying to help us out with a PR, I think this should do it:
git remote add upstream https://github.com/spine-generic/data-multi-subject
git annex sync -a
but I'm not 100% sure that'll do it for you.
Thanks a lot for the excellent bug report though!
I can delete the whole fork and make clone from the upstream https://github.com/spine-generic/data-multi-subject
; but how can I edit the database then? I should update the file https://github.com/spine-generic/data-multi-subject/blob/master/participants.tsv
about height and weight values. Original plan was to fork the database, make edits in my fork and PR changes into the upstream master vesrion. This plan, you writes, is impossible, right? So before I delete my fork, how can we proceed the participants.tsv file edits?
If all you want to edit is participants.tsv
then in a pinch you don't actually need to have git annex get .
succeed. That file isn't "annexed". We only annexed the niftis. So, just make your edits and submit that and we'll be able to integrate them.
To read the full correct dataset, can you try this?
git remote add upstream https://github.com/spine-generic/data-multi-subject
git branch --track -f git-annex upstream/git-annex
git annex get .
The niftis are read-only though.
We don't have a way for you to edit niftis because git-annex forces everyone to share a single large pool of storage -- and we're using Amazon for that (for the low price point) and need to hold on tight to the keys to it. We haven't figured out a workaround to allow external contributions to the annexed files and, well, personally I want to drop git-annex instead of thinking harder about it because this use-case isn't solvable by git-annex https://github.com/neuropoly/data-management/issues/68.
First suggested commands have downloaded the data. But last line of the STDOUT wrotes some sync fail (i.e. look for sync: 1 failed
):
git remote add upstream https://github.com/spine-generic/data-multi-subject
git annex sync -a
But following git branch --track -f git-annex upstream/git-annex
appears to break the git annex sync -a
command due to user permissions. Please see and track history of commands and STDOUTs below. I guess I should set the git branch --track
to default setting at least. Sorry, I am without idea how to set it back... :-(
PS: I would need both to have the data for analysis purposes and be able to edit files outside git-annex like the participants.tsv file. Many thanks for your help.
[user@server data-multi-subject]$ git remote add upstream https://github.com/spine-generic/data-multi-subject
[user@server data-multi-subject]$ git annex sync -a
pull upstream
remote: Enumerating objects: 14337, done.
remote: Counting objects: 100% (14337/14337), done.
remote: Compressing objects: 100% (3945/3945), done.
remote: Total 14337 (delta 7637), reused 14302 (delta 7637), pack-reused 0
Receiving objects: 100% (14337/14337), 1.23 MiB | 3.50 MiB/s, done.
Resolving deltas: 100% (7637/7637), done.
From https://github.com/spine-generic/data-multi-subject
* branch git-annex -> FETCH_HEAD
* branch synced/git-annex -> FETCH_HEAD
* [new branch] git-annex -> upstream/git-annex
* [new branch] synced/git-annex -> upstream/synced/git-annex
ok
pull origin
From https://github.com/renelabounek/data-multi-subject
* branch git-annex -> FETCH_HEAD
* branch synced/git-annex -> FETCH_HEAD
ok
(merging upstream/git-annex into git-annex...)
(recording state in git...)
get sub-ucdavis01/anat/sub-ucdavis01_T1w.nii.gz (from amazon...)
ok
get sub-ucdavis01/anat/sub-ucdavis01_T2star.nii.gz (from amazon...)
ok
get sub-ucdavis01/anat/sub-ucdavis01_T2w.nii.gz (from amazon...)
ok
get sub-ucdavis01/anat/sub-ucdavis01_acq-MToff_MTS.nii.gz (from amazon...)
ok
get sub-ucdavis01/anat/sub-ucdavis01_acq-MTon_MTS.nii.gz (from amazon...)
ok
get sub-ucdavis01/anat/sub-ucdavis01_acq-T1w_MTS.nii.gz (from amazon...)
ok
get sub-ucdavis01/dwi/sub-ucdavis01_run-01_dwi.nii.gz (from amazon...)
ok
get sub-ucdavis01/dwi/sub-ucdavis01_run-02_dwi.nii.gz (from amazon...)
ok
get sub-ucdavis01/dwi/sub-ucdavis01_run-03_dwi.nii.gz (from amazon...)
ok
get sub-ucdavis02/anat/sub-ucdavis02_T1w.nii.gz (from amazon...)
ok
get sub-ucdavis02/anat/sub-ucdavis02_T2star.nii.gz (from amazon...)
ok
get sub-ucdavis02/anat/sub-ucdavis02_T2w.nii.gz (from amazon...)
ok
get sub-ucdavis02/anat/sub-ucdavis02_acq-MToff_MTS.nii.gz (from amazon...)
ok
get sub-ucdavis02/anat/sub-ucdavis02_acq-MTon_MTS.nii.gz (from amazon...)
ok
get sub-ucdavis02/anat/sub-ucdavis02_acq-T1w_MTS.nii.gz (from amazon...)
ok
get sub-ucdavis02/dwi/sub-ucdavis02_run-01_dwi.nii.gz (from amazon...)
ok
get sub-ucdavis02/dwi/sub-ucdavis02_run-02_dwi.nii.gz (from amazon...)
ok
get sub-ucdavis02/dwi/sub-ucdavis02_run-03_dwi.nii.gz (from amazon...)
ok
get sub-ucdavis03/anat/sub-ucdavis03_T1w.nii.gz (from amazon...)
ok
get sub-ucdavis03/anat/sub-ucdavis03_T2star.nii.gz (from amazon...)
ok
get sub-ucdavis03/anat/sub-ucdavis03_T2w.nii.gz (from amazon...)
ok
get sub-ucdavis03/anat/sub-ucdavis03_acq-MToff_MTS.nii.gz (from amazon...)
ok
get sub-ucdavis03/anat/sub-ucdavis03_acq-MTon_MTS.nii.gz (from amazon...)
ok
get sub-ucdavis03/dwi/sub-ucdavis03_dwi.nii.gz (from amazon...)
ok
get sub-ucdavis04/anat/sub-ucdavis04_T1w.nii.gz (from amazon...)
ok
get sub-ucdavis04/anat/sub-ucdavis04_T2star.nii.gz (from amazon...)
ok
get sub-ucdavis04/anat/sub-ucdavis04_T2w.nii.gz (from amazon...)
ok
get sub-ucdavis04/anat/sub-ucdavis04_acq-MToff_MTS.nii.gz (from amazon...)
ok
get sub-ucdavis04/anat/sub-ucdavis04_acq-MTon_MTS.nii.gz (from amazon...)
ok
get sub-ucdavis04/anat/sub-ucdavis04_acq-T1w_MTS.nii.gz (from amazon...)
ok
get sub-ucdavis04/dwi/sub-ucdavis04_run-01_dwi.nii.gz (from amazon...)
ok
get sub-ucdavis04/dwi/sub-ucdavis04_run-02_dwi.nii.gz (from amazon...)
ok
get sub-ucdavis04/dwi/sub-ucdavis04_run-03_dwi.nii.gz (from amazon...)
ok
get sub-ucdavis05/anat/sub-ucdavis05_T1w.nii.gz (from amazon...)
ok
get sub-ucdavis05/anat/sub-ucdavis05_T2star.nii.gz (from amazon...)
ok
get sub-ucdavis05/anat/sub-ucdavis05_T2w.nii.gz (from amazon...)
ok
get sub-ucdavis05/anat/sub-ucdavis05_acq-MToff_MTS.nii.gz (from amazon...)
ok
get sub-ucdavis05/anat/sub-ucdavis05_acq-MTon_MTS.nii.gz (from amazon...)
ok
get sub-ucdavis05/dwi/sub-ucdavis05_dwi.nii.gz (from amazon...)
ok
get sub-ucdavis06/anat/sub-ucdavis06_T1w.nii.gz (from amazon...)
ok
get sub-ucdavis06/anat/sub-ucdavis06_T2star.nii.gz (from amazon...)
ok
get sub-ucdavis06/anat/sub-ucdavis06_T2w.nii.gz (from amazon...)
ok
get sub-ucdavis06/anat/sub-ucdavis06_acq-MToff_MTS.nii.gz (from amazon...)
ok
get sub-ucdavis06/anat/sub-ucdavis06_acq-MTon_MTS.nii.gz (from amazon...)
ok
get sub-ucdavis06/dwi/sub-ucdavis06_dwi.nii.gz (from amazon...)
ok
get sub-ucdavis07/anat/sub-ucdavis07_T1w.nii.gz (from amazon...)
ok
get sub-ucdavis07/anat/sub-ucdavis07_T2star.nii.gz (from amazon...)
ok
get sub-ucdavis07/anat/sub-ucdavis07_T2w.nii.gz (from amazon...)
ok
get sub-ucdavis07/anat/sub-ucdavis07_acq-MToff_MTS.nii.gz (from amazon...)
ok
get sub-ucdavis07/anat/sub-ucdavis07_acq-MTon_MTS.nii.gz (from amazon...)
ok
get sub-ucdavis07/dwi/sub-ucdavis07_dwi.nii.gz (from amazon...)
ok
pull upstream
From https://github.com/spine-generic/data-multi-subject
* branch git-annex -> FETCH_HEAD
* branch synced/git-annex -> FETCH_HEAD
ok
pull origin
From https://github.com/renelabounek/data-multi-subject
* branch git-annex -> FETCH_HEAD
* branch synced/git-annex -> FETCH_HEAD
ok
(recording state in git...)
push upstream
remote: Permission to spine-generic/data-multi-subject.git denied to renelabounek.
fatal: unable to access 'https://github.com/spine-generic/data-multi-subject/': The requested URL returned error: 403
Pushing to upstream failed.
failed
push origin
Enumerating objects: 13713, done.
Counting objects: 100% (13713/13713), done.
Delta compression using up to 32 threads
Compressing objects: 100% (5677/5677), done.
Writing objects: 100% (7032/7032), 539.40 KiB | 3.67 MiB/s, done.
Total 7032 (delta 4992), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (4992/4992), completed with 2658 local objects.
To https://github.com/renelabounek/data-multi-subject.git
5c768aa9..7b2f84d2 git-annex -> synced/git-annex
ok
(recording state in git...)
sync: 1 failed
[user@server data-multi-subject]$ git branch --track -f git-annex upstream/git-annex
Branch git-annex set up to track remote branch git-annex from upstream.
[user@server data-multi-subject]$ git annex get .
[user@server data-multi-subject]$ git annex sync -a
(merging origin/git-annex into git-annex...)
pull upstream
From https://github.com/spine-generic/data-multi-subject
* branch git-annex -> FETCH_HEAD
* branch synced/git-annex -> FETCH_HEAD
ok
pull origin
From https://github.com/renelabounek/data-multi-subject
* branch git-annex -> FETCH_HEAD
* branch synced/git-annex -> FETCH_HEAD
ok
push upstream
remote: Permission to spine-generic/data-multi-subject.git denied to renelabounek.
fatal: unable to access 'https://github.com/spine-generic/data-multi-subject/': The requested URL returned error: 403
Pushing to upstream failed.
failed
sync: 1 failed
[user@server data-multi-subject]$
First suggested commands have downloaded the data. But last line of the STDOUT wrotes some sync fail (i.e. look for
sync: 1 failed
):git remote add upstream https://github.com/spine-generic/data-multi-subject git annex sync -a
But following
git branch --track -f git-annex upstream/git-annex
appears to break thegit annex sync -a
command due to user permissions. Please see and track history of commands and STDOUTs below. I guess I should set thegit branch --track
to default setting at least. Sorry, I am without idea how to set it back... :-(
I think it worked as well as it's going to! https://github.com/renelabounek/data-multi-subject/tree/git-annex now says
and you were able to download all the subjects.
Just do one more thing:
git config remote.origin.annex-readonly true
and that should prevent the permission errors in the future.
We'll have to think about the clearest way to integrate this advice into our contributor documentation.
Note to future selves: git branch --track -f git-annex upstream/git-annex
was unnecessary; git annex sync
evidently syncs all git-annex
branches it can reach:
pull upstream
From https://github.com/spine-generic/data-multi-subject
* branch git-annex -> FETCH_HEAD
* branch synced/git-annex -> FETCH_HEAD
ok
pull origin
From https://github.com/renelabounek/data-multi-subject
* branch git-annex -> FETCH_HEAD
* branch synced/git-annex -> FETCH_HEAD
ok
It did not prevent the permission errors. + I am not aware, I would do any 4 commints myself to get ahead the upstream/git-annex
. I assume it happened during the serie of executed git commands. And, it is something unimportant for both sites, right?
[user@server data-multi-subject]$ git config remote.origin.annex-readonly true
[user@server data-multi-subject]$ git annex sync -a
pull upstream
From https://github.com/spine-generic/data-multi-subject
* branch git-annex -> FETCH_HEAD
* branch synced/git-annex -> FETCH_HEAD
ok
pull origin
From https://github.com/renelabounek/data-multi-subject
* branch git-annex -> FETCH_HEAD
* branch synced/git-annex -> FETCH_HEAD
ok
push upstream
remote: Permission to spine-generic/data-multi-subject.git denied to renelabounek.
fatal: unable to access 'https://github.com/spine-generic/data-multi-subject/': The requested URL returned error: 403
Pushing to upstream failed.
failed
sync: 1 failed
[user@server data-multi-subject]$
Right sorry, my mistake, you have the names opposite to what I usually use:
git config remote.upstream.annex-readonly true
git config --unset remote.origin.annex-readonly
4 commints myself to get ahead the
upstream/git-annex
. I assume it happened during the serie of executed git commands. And, it is something unimportant for both sites, right?
I can fill you in on this, since you asked :)
This is just something git annex sync
does. You can look at the commits, though they won't be that meaningful to either of us: e.g. https://github.com/renelabounek/data-multi-subject/commit/7b2f84d2a3694bcb28d677f656bf401c053d6c5e
1638212764.542080109s 1 7133c793-335e-4cd9-be0a-9827eec7a9a2
this represents the fact that you have a particular version of a particular annexed file in your local copy (which has been given the name "7133c793-335e-4cd9-be0a-9827eec7a9a2" in the background by git-annex). And that's part of why we definitely don't want to sync your git-annex
branch back into ours: it would allow the possibility of people trying to fetch copies of the MRI images off your laptop, which will never work and will just confuse people.
and https://github.com/renelabounek/data-multi-subject/commit/0a9c9e3a199a28e7cd0ebe8ca4513fbc6114d6eb is just a merge commit to sync our changes with yours.
Yes, it passed without error now. Thanks @kousu for your help with debuging.
I use fetched and merged fork of the data-multi-subject. There I use the master version:
Git head is:
Git-annex head is:
I have tried
git pull && git annex sync --no-content && git annex get .
, but 51 files still fail to download: