Closed alexfoias closed 3 years ago
We didn't get the date_of_scan info. I think we can go without it.
@jcohenadad I have updated the participants.tsv
oopsi, any idea what's going on? https://github.com/spine-generic/data-multi-subject/pull/77/checks?check_run_id=2326970382#step:8:2770
@kousu The build fails. The error looks something like
get sub-ucdavis01/anat/sub-ucdavis01_T1w.nii.gz (not available)
No other repository is known to contain the file.
(Note that these git remotes have annex-ignore set: origin)
failed
Did I do something wrong?
Probably that file didn't get upload to Amazon correctly. I'm investigating.
Possibly because git-annex
keeps its metadata in a separate branch (https://github.com/neuropoly/data-management/issues/67). This was submitted from a fork which means the git-annex
branch is isolated over in https://github.com/alexfoias/data-multi-subject/ and this repo isn't seeing it. Maybe. I still need to wait for things to sync fully before I can check for sure.
I'll see if I can figure out how to tweak the CI script to be able to deal with forks. But because of this separate branch problem git-annex
and the Github GUI are never going to be fully compatible.
It's a simpler, but related, problem: @alexfoias your local git-annex
branch hasn't been synchronized to github:
$ git remote add af https://github.com/alexfoias/data-multi-subject/
$ git fetch af
remote: Enumerating objects: 9058, done.
remote: Counting objects: 100% (9058/9058), done.
remote: Compressing objects: 100% (1984/1984), done.
remote: Total 16921 (delta 5816), reused 8389 (delta 5709), pack-reused 7863
Receiving objects: 100% (16921/16921), 1.55 MiB | 3.50 MiB/s, done.
Resolving deltas: 100% (6918/6918), completed with 2004 local objects.
From https://github.com/alexfoias/data-multi-subject
* [new branch] af/add_ucdavis -> af/af/add_ucdavis
* [new branch] af/update_participants_tsv -> af/af/update_participants_tsv
* [new branch] git-annex -> af/git-annex
* [new branch] master -> af/master
* [new branch] renelabounek-master -> af/renelabounek-master
* [new branch] synced/af/csfseg_json -> af/synced/af/csfseg_json
* [new branch] synced/git-annex -> af/synced/git-annex
* [new branch] synced/jca/17-defacing -> af/synced/jca/17-defacing
* [new branch] synced/jca/21-travis -> af/synced/jca/21-travis
* [new branch] synced/jca/7-check-data-consistency -> af/synced/jca/7-check-data-consistency
* [new branch] synced/jca/derivatives-r20200830-mts -> af/synced/jca/derivatives-r20200830-mts
* [new branch] synced/jca/participants -> af/synced/jca/participants
* [new branch] synced/jca/readme -> af/synced/jca/readme
* [new branch] synced/jca/reupload-disc-labels -> af/synced/jca/reupload-disc-labels
* [new branch] synced/jca/update-derivatives -> af/synced/jca/update-derivatives
* [new branch] synced/master -> af/synced/master
[kousu@requiem data-multi-subject]$ git annex whereis sub-ucdavis01/anat/sub-ucdavis01_T1w.nii.gz
(merging af/git-annex into git-annex...)
(recording state in git...)
Remote af not usable by git-annex; setting annex-ignore
https://github.com/alexfoias/data-multi-subject//config download failed: Not Found
git-annex: sub-ucdavis01/anat/sub-ucdavis01_T1w.nii.gz not found
git-annex: whereis: 1 failed
$ git log --pretty=short af/git-annex^^..af/git-annex
commit 3cc730e6fac61f41c969b60af0d62d794250fe3b (af/synced/git-annex, af/git-annex)
Author: Alexandru Foias <afoias@polymtl.ca>
update
commit 39d9c91f1bacc5fe9452028a2b8c4bc344438de7
Author: Alexandru Foias <afoias@polymtl.ca>
update
$
(notice the timestamps)
You need to do
git annex sync
More directly you can do
git push origin git-annex
which is what I am thinking more and more I want to recommend.
@kousu I did this:
[13/04/21 7:33:29] (base) alex@NeuroPoly-MacBook-Pro data-multi-subject % git push origin git-annex
Enumerating objects: 290, done.
Counting objects: 100% (290/290), done.
Delta compression using up to 12 threads
Compressing objects: 100% (237/237), done.
Writing objects: 100% (261/261), 112.78 KiB | 6.63 MiB/s, done.
Total 261 (delta 53), reused 1 (delta 0)
remote: Resolving deltas: 100% (53/53), completed with 4 local objects.
To https://github.com/alexfoias/data-multi-subject.git
3cc730e6..c7283b1c git-annex -> git-annex
[13/04/21 7:33:33] (base) alex@NeuroPoly-MacBook-Pro data-multi-subject % git annex sync
commit
On branch af/add_ucdavis
Your branch is up to date with 'origin/af/add_ucdavis'.
nothing to commit, working tree clean
ok
pull origin
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 5 (delta 1), reused 3 (delta 1), pack-reused 0
Unpacking objects: 100% (5/5), done.
From https://github.com/alexfoias/data-multi-subject
8143f98b..7f58747b af/add_ucdavis -> origin/af/add_ucdavis
Updating 8143f98b..7f58747b
Fast-forward
README.md | 19 +++++++++++++++----
1 file changed, 15 insertions(+), 4 deletions(-)
ok
push origin
Total 0 (delta 0), reused 0 (delta 0)
To https://github.com/alexfoias/data-multi-subject.git
3cc730e6..c7283b1c git-annex -> synced/git-annex
* [new branch] af/add_ucdavis -> synced/af/add_ucdavis
ok
[13/04/21 7:34:05] (base) alex@NeuroPoly-MacBook-Pro data-multi-subject % git push origin git-annex
Everything up-to-date
@alexfoias Problem 2 seems to be that the files have not been uploaded to Amazon
$ git branch
* af/add_ucdavis
master
$ git annex whereis sub-ucdavis06/
whereis sub-ucdavis06/anat/sub-ucdavis06_T1w.nii.gz (1 copy)
56bbd6c5-a147-4940-bf73-212f50841743 -- alex@NeuroPoly-MacBook-Pro.local:~/data/data-multi-subject
ok
whereis sub-ucdavis06/anat/sub-ucdavis06_T2star.nii.gz (1 copy)
56bbd6c5-a147-4940-bf73-212f50841743 -- alex@NeuroPoly-MacBook-Pro.local:~/data/data-multi-subject
ok
whereis sub-ucdavis06/anat/sub-ucdavis06_T2w.nii.gz (1 copy)
56bbd6c5-a147-4940-bf73-212f50841743 -- alex@NeuroPoly-MacBook-Pro.local:~/data/data-multi-subject
ok
whereis sub-ucdavis06/anat/sub-ucdavis06_acq-MToff_MTS.nii.gz (1 copy)
56bbd6c5-a147-4940-bf73-212f50841743 -- alex@NeuroPoly-MacBook-Pro.local:~/data/data-multi-subject
ok
whereis sub-ucdavis06/anat/sub-ucdavis06_acq-MTon_MTS.nii.gz (1 copy)
56bbd6c5-a147-4940-bf73-212f50841743 -- alex@NeuroPoly-MacBook-Pro.local:~/data/data-multi-subject
ok
whereis sub-ucdavis06/dwi/sub-ucdavis06_dwi.nii.gz (1 copy)
56bbd6c5-a147-4940-bf73-212f50841743 -- alex@NeuroPoly-MacBook-Pro.local:~/data/data-multi-subject
ok
Can you try this?
git annex dead here
git annex sync --content amazon
git annex sync
@kousu Still doesn't work https://gist.github.com/alexfoias/bdf8f28c3e2adbe8048cc3ce4aaf6736
@alexfoias oops, you need to load your Amazon password:
copy sub-ucdavis01/anat/sub-ucdavis01_T1w.nii.gz (checking amazon...) (to amazon...)
Set both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to use S3
No S3 credentials configured
failed
You need to run
export AWS_ACCESS_KEY_ID="..." AWS_SECRET_ACCESS_KEY="..."
You can put those in your ~/.profile (or, ~/.zprofile? You're on the new macOS right? is that the right path now?).
I wonder if we can script this to make this easier. Personally, I have my AWS keys in my password manager so what I actually do is
AWS_ACCESS_KEY_ID="$(pass work/neuropoly/aws | grep username: | awk '{print $2}')" AWS_SECRET_ACCESS_KEY="$(pass work/neuropoly/aws | head -n 1)" git annex sync --content amazon
which prompts me to unlock my password manager.
I probably should have told you git annex copy --to amazon
. I'm sorry you had to wait for the downloads before you could upload, too.
(the docs are here)
Turns out @alexfoias had never had AWS access in the first place. I've granted him write-access to it.
@kousu How do I have to add the AWS credentials ? Do I just have to set the env var export AWS_ACCESS_KEY_ID="..." AWS_SECRET_ACCESS_KEY="..."
?
@alexfoias yes, exactly. Just paste them into your terminal and press enter!
You can also make them more permanent with these suggestions: https://github.com/spine-generic/data-multi-subject/pull/77#issuecomment-819029909
After exporting the AWS credentials:
[16/04/21 11:53:31] (base) alex@NeuroPoly-MacBook-Pro data-multi-subject % git annex dead here
dead here (recording state in git...)
ok
(recording state in git...)
[16/04/21 11:53:33] (base) alex@NeuroPoly-MacBook-Pro data-multi-subject % git annex sync --content amazon
commit
On branch af/add_ucdavis
Your branch is up to date with 'origin/af/add_ucdavis'.
nothing to commit, working tree clean
ok
copy sub-ucdavis01/anat/sub-ucdavis01_T1w.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis01/anat/sub-ucdavis01_T2star.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis01/anat/sub-ucdavis01_T2w.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis01/anat/sub-ucdavis01_acq-MToff_MTS.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis01/anat/sub-ucdavis01_acq-MTon_MTS.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis01/anat/sub-ucdavis01_acq-T1w_MTS.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis01/dwi/sub-ucdavis01_run-01_dwi.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis01/dwi/sub-ucdavis01_run-02_dwi.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis01/dwi/sub-ucdavis01_run-03_dwi.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis02/anat/sub-ucdavis02_T1w.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis02/anat/sub-ucdavis02_T2star.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis02/anat/sub-ucdavis02_T2w.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis02/anat/sub-ucdavis02_acq-MToff_MTS.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis02/anat/sub-ucdavis02_acq-MTon_MTS.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis02/anat/sub-ucdavis02_acq-T1w_MTS.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis02/dwi/sub-ucdavis02_run-01_dwi.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis02/dwi/sub-ucdavis02_run-02_dwi.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis02/dwi/sub-ucdavis02_run-03_dwi.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis03/anat/sub-ucdavis03_T1w.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis03/anat/sub-ucdavis03_T2star.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis03/anat/sub-ucdavis03_T2w.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis03/anat/sub-ucdavis03_acq-MToff_MTS.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis03/anat/sub-ucdavis03_acq-MTon_MTS.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis03/dwi/sub-ucdavis03_dwi.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis04/anat/sub-ucdavis04_T1w.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis04/anat/sub-ucdavis04_T2star.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis04/anat/sub-ucdavis04_T2w.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis04/anat/sub-ucdavis04_acq-MToff_MTS.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis04/anat/sub-ucdavis04_acq-MTon_MTS.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis04/anat/sub-ucdavis04_acq-T1w_MTS.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis04/dwi/sub-ucdavis04_run-01_dwi.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis04/dwi/sub-ucdavis04_run-02_dwi.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis04/dwi/sub-ucdavis04_run-03_dwi.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis05/anat/sub-ucdavis05_T1w.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis05/anat/sub-ucdavis05_T2star.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis05/anat/sub-ucdavis05_T2w.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis05/anat/sub-ucdavis05_acq-MToff_MTS.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis05/anat/sub-ucdavis05_acq-MTon_MTS.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis05/dwi/sub-ucdavis05_dwi.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis06/anat/sub-ucdavis06_T1w.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis06/anat/sub-ucdavis06_T2star.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis06/anat/sub-ucdavis06_T2w.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis06/anat/sub-ucdavis06_acq-MToff_MTS.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis06/anat/sub-ucdavis06_acq-MTon_MTS.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis06/dwi/sub-ucdavis06_dwi.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis07/anat/sub-ucdavis07_T1w.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis07/anat/sub-ucdavis07_T2star.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis07/anat/sub-ucdavis07_T2w.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis07/anat/sub-ucdavis07_acq-MToff_MTS.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis07/anat/sub-ucdavis07_acq-MTon_MTS.nii.gz (checking amazon...) (to amazon...)
ok
copy sub-ucdavis07/dwi/sub-ucdavis07_dwi.nii.gz (checking amazon...) (to amazon...)
ok
(recording state in git...)
[16/04/21 11:54:35] (base) alex@NeuroPoly-MacBook-Pro data-multi-subject % git annex sync
commit
On branch af/add_ucdavis
Your branch is up to date with 'origin/af/add_ucdavis'.
nothing to commit, working tree clean
ok
pull origin
ok
push origin
Enumerating objects: 310, done.
Counting objects: 100% (310/310), done.
Delta compression using up to 12 threads
Compressing objects: 100% (138/138), done.
Writing objects: 100% (161/161), 12.54 KiB | 1.39 MiB/s, done.
Total 161 (delta 57), reused 0 (delta 0)
remote: Resolving deltas: 100% (57/57), completed with 55 local objects.
To https://github.com/alexfoias/data-multi-subject.git
2501e150..9524c416 git-annex -> synced/git-annex
ok
@kousu Houston we have a problem :(
@alexfoias git push origin git-annex
?
Yeah looks like that's the problem:
$ git fetch af
remote: Enumerating objects: 8, done.
remote: Counting objects: 100% (8/8), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 8 (delta 2), reused 3 (delta 2), pack-reused 0
Unpacking objects: 100% (8/8), 9.47 KiB | 111.00 KiB/s, done.
From https://github.com/alexfoias/data-multi-subject
7f58747b5..dc66d8e76 af/add_ucdavis -> af/af/add_ucdavis
It should say
7f58747b5..dc66d8e76 af/add_ucdavis -> af/af/add_ucdavis
9524c4169..something af/git-annex -> af/git-annex
[16/04/21 12:17:31] (base) alex@NeuroPoly-MacBook-Pro data-multi-subject % git push origin git-annex
Everything up-to-date
Oh no, I'm not sure about that.
It looks good on my local copy:
$ git annex whereis sub-ucdavis07
whereis sub-ucdavis07/anat/sub-ucdavis07_T1w.nii.gz (1 copy)
5a5447a8-a9b8-49bc-8276-01a62632b502 -- [amazon]
amazon: https://data-multi-subject---spine-generic---neuropoly.s3.ca-central-1.amazonaws.com/SHA256E-s4484346--9b027c074735c3d2d69f046ebaec5831c4e6fc8959a3a64070428ebd75a62fd9.nii.gz
ok
whereis sub-ucdavis07/anat/sub-ucdavis07_T2star.nii.gz (1 copy)
5a5447a8-a9b8-49bc-8276-01a62632b502 -- [amazon]
amazon: https://data-multi-subject---spine-generic---neuropoly.s3.ca-central-1.amazonaws.com/SHA256E-s1765026--7fb7243348a5bf2c8994cd383848aa3c279ee072701ba56bf43f50b8fb06efbf.nii.gz
ok
whereis sub-ucdavis07/anat/sub-ucdavis07_T2w.nii.gz (1 copy)
5a5447a8-a9b8-49bc-8276-01a62632b502 -- [amazon]
amazon: https://data-multi-subject---spine-generic---neuropoly.s3.ca-central-1.amazonaws.com/SHA256E-s6620793--78252c6bb18e92298b2b7ec79d4ec5d441750790d7849c0dc174a78a95e669a3.nii.gz
ok
whereis sub-ucdavis07/anat/sub-ucdavis07_acq-MToff_MTS.nii.gz (1 copy)
5a5447a8-a9b8-49bc-8276-01a62632b502 -- [amazon]
amazon: https://data-multi-subject---spine-generic---neuropoly.s3.ca-central-1.amazonaws.com/SHA256E-s1355554--69fe2b3b512b64b0bfcedfddfe5f8485725db699142010898c20aa2b3c63c0af.nii.gz
ok
whereis sub-ucdavis07/anat/sub-ucdavis07_acq-MTon_MTS.nii.gz (1 copy)
5a5447a8-a9b8-49bc-8276-01a62632b502 -- [amazon]
amazon: https://data-multi-subject---spine-generic---neuropoly.s3.ca-central-1.amazonaws.com/SHA256E-s1294757--043e6b105eec24f64f1dd72b221d7b58569f02bcc9bed42e99795d99a5dd26d2.nii.gz
ok
whereis sub-ucdavis07/dwi/sub-ucdavis07_dwi.nii.gz (1 copy)
5a5447a8-a9b8-49bc-8276-01a62632b502 -- [amazon]
amazon: https://data-multi-subject---spine-generic---neuropoly.s3.ca-central-1.amazonaws.com/SHA256E-s2348392--12a62969d9a30c726f5471a104c7e42817e3e1509bc5171bbb420e3f4b61fa4f.nii.gz
ok
so I don't yet know what's wrong with Github. But my money is on Github not downloading the git-annex
branch out of your fork.
Yes, I think that's it: https://github.com/spine-generic/data-multi-subject/pull/77/checks?check_run_id=2363833281#step:8:123
Run git fetch --depth=1 origin git-annex:git-annex # CI does a shallow checkout, so it is missing this important branch
From https://github.com/spine-generic/data-multi-subject
* [new branch] git-annex -> git-annex
* [new branch] git-annex -> origin/git-annex
init (scanning for unlocked files...)
Remote origin not usable by git-annex; setting annex-ignore
so it's fetching the git-annex
from this repo not your repo. We'll have to patch the CI script to fix that. Awkward.
I'm going to mess with your branch a bit to figure this out.
This is a classic case of https://github.com/neuropoly/data-management/issues/67
I got it working. I'm going to merge now.
:disappointed_relieved:
thank you @alexfoias and @kousu. These painful experiences add to cons of git-annex
It's not fully git-annex's fault. It's just not fully compatible with the Github workflow. Equally much you could say "add to the cons of Github".
https://github.com/spine-generic/data-multi-subject/actions/runs/756637528 :sweat:
I predicted this just now:
okay I'll see what I can do about it.
Right, of course, it's literally the same bug:
(recording state in git...)
git-annex: get: 51 failed
Error: Process completed with exit code 1.
Clicking the 'merge' button doesn't also merge the git-annex
branch.
What to do about that?
Never use forks ever? Or never merge git-annex changes on the command line; but we have branch protection turned on so we cannot merge from the command line, it has to be done through the Github GUI? Or make sure to make associated PRs for the git-annex
branch?
I'm trying my third strategy this time: https://github.com/spine-generic/data-multi-subject/pull/84. Can one of you approve it @jcohenadad @alexfoias ?
Never use forks ever?
yes, that's also a possibility. We could change our internal workflow, get the data somewhere, review them, and then someone uploads them directly on the master branch.
@kousu Thanks for fixing the issue. Is there an update in the documentation for how we should handle data in the future ?
Should we consider dropping forks/branches, and just having someone from the team review the addition and push it directly on master?
We had a report by email ("git-annex at UMN servers") about these 51 files not being downloadable. I don't see how, CI has been green for the 8 months since they went in, but git annex is an arcane art and maybe the unusual fixup I did broke something somehow when combined with certain git annex versions.
Curently missing sex, age, date of scan.
Here is a QC of defacing ucdavis_spineGeneric_QC.zip