git-annex: pushed data on amazon, not seen from other local stations

[jcohenadad]

as part of https://github.com/spine-generic/data-multi-subject/pull/40 i just pushed some data to amazon:

julien-macbook:~/code/spine-generic/data-multi-subject $ git annex sync
commit 
On branch jca/update-derivatives
nothing to commit, working tree clean
ok
pull origin 
ok
push origin 
Enumerating objects: 66, done.
Counting objects: 100% (66/66), done.
Delta compression using up to 16 threads
Compressing objects: 100% (42/42), done.
Writing objects: 100% (49/49), 4.40 KiB | 1.46 MiB/s, done.
Total 49 (delta 13), reused 0 (delta 0)
remote: Resolving deltas: 100% (13/13), completed with 5 local objects.
To https://github.com/spine-generic/data-multi-subject.git
   42c7b920..957506b3  git-annex -> synced/git-annex
 * [new branch]        jca/update-derivatives -> synced/jca/update-derivatives
ok
julien-macbook:~/code/spine-generic/data-multi-subject $ git-annex whereis derivatives/labels/sub-vuiisAchieva06/anat/sub-vuiisAchieva06_T2star_rms_gmseg-manual.nii.gz
whereis derivatives/labels/sub-vuiisAchieva06/anat/sub-vuiisAchieva06_T2star_rms_gmseg-manual.nii.gz (1 copy) 
    5cdba4fc-8d50-4e89-bb0c-a3a4f9449666 -- julien@julien-macbook.local:~/code/spine-generic/data-multi-subject [here]
ok
julien-macbook:~/code/spine-generic/data-multi-subject $ git-annex sync --content amazon
commit 
On branch jca/update-derivatives
nothing to commit, working tree clean
ok
copy derivatives/labels/sub-fslAchieva01/anat/sub-fslAchieva01_T2star_rms_gmseg-manual.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy derivatives/labels/sub-nottwil02/anat/sub-nottwil02_T2star_rms_gmseg-manual.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy derivatives/labels/sub-nottwil06/anat/sub-nottwil06_T2star_rms_gmseg-manual.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy derivatives/labels/sub-tokyoIngenia03/anat/sub-tokyoIngenia03_T2star_rms_gmseg-manual.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy derivatives/labels/sub-vuiisAchieva06/anat/sub-vuiisAchieva06_T2star_rms_gmseg-manual.nii.gz (checking amazon...) (to amazon...) 
ok                                
(recording state in git...)
julien-macbook:~/code/spine-generic/data-multi-subject $ git-annex whereis derivatives/labels/sub-vuiisAchieva06/anat/sub-vuiisAchieva06_T2star_rms_gmseg-manual.nii.gz
whereis derivatives/labels/sub-vuiisAchieva06/anat/sub-vuiisAchieva06_T2star_rms_gmseg-manual.nii.gz (2 copies) 
    5a5447a8-a9b8-49bc-8276-01a62632b502 -- [amazon]
    5cdba4fc-8d50-4e89-bb0c-a3a4f9449666 -- julien@julien-macbook.local:~/code/spine-generic/data-multi-subject [here]

  amazon: https://data-multi-subject---spine-generic---neuropoly.s3.ca-central-1.amazonaws.com/SHA256E-s28294--5e6dc442d20bb79ab0fd5dd62c5c5b968bf5926fc57b93fe02a1dad4366c3f0f.nii.gz
ok

However, Travis does not see the data, and when trying on compute canada, it's even worse (ie: local copy on julien-laptop is even not shown):

git-annex sync --no-push
[jcohen@gra-login2 data-multi-subject]$ whereis derivatives/labels/sub-vuiisAchieva06/anat/sub-vuiisAchieva06_T2star_rms_gmseg-manual.nii.gz
sub-vuiisAchieva06_T2star_rms_gmseg-manual.nii:

what is going on???

maybe the julien-laptop mirror is not visible on compute canada because i am on the master branch?

nope:

[jcohen@gra-login2 data-multi-subject]$ git checkout -b jca/update-derivatives origin/jca/update-derivatives
Updating files: 100% (11/11), done.
Branch 'jca/update-derivatives' set up to track remote branch 'jca/update-derivatives' from 'origin'.
Switched to a new branch 'jca/update-derivatives'
[jcohen@gra-login2 data-multi-subject]$ whereis derivatives/labels/sub-vuiisAchieva06/anat/sub-vuiisAchieva06_T2star_rms_gmseg-manual.nii.gz
sub-vuiisAchieva06_T2star_rms_gmseg-manual.nii:

[jcohenadad]

what is mind-boggling is that the SHA256 is pushed to the branch: https://github.com/spine-generic/data-multi-subject/blob/34c6c70bc2e4c2a874c5f1da340188b0e8cd00c7/derivatives/labels/sub-tokyoIngenia03/anat/sub-tokyoIngenia03_T2star_rms_gmseg-manual.nii.gz#L1

and the associated URL on amazon does work (ie: data can be downloaded): https://data-multi-subject---spine-generic---neuropoly.s3.ca-central-1.amazonaws.com/SHA256E-s28294--5e6dc442d20bb79ab0fd5dd62c5c5b968bf5926fc57b93fe02a1dad4366c3f0f.nii.gz

[kousu]

I think you forgot to git annex sync origin. git annex sync --content amazon syncs the data, but you missed syncing the metadata after that.

[jcohenadad]

trying to fix it (thanks @kousu):

julien-macbook:~/code/spine-generic/data-multi-subject $ git-annex sync
commit 
On branch jca/update-derivatives
Your branch is up to date with 'origin/jca/update-derivatives'.

nothing to commit, working tree clean
ok
pull origin 
ok
push origin 
Enumerating objects: 33, done.
Counting objects: 100% (33/33), done.
Delta compression using up to 16 threads
Compressing objects: 100% (14/14), done.
Writing objects: 100% (17/17), 1.34 KiB | 1.34 MiB/s, done.
Total 17 (delta 6), reused 0 (delta 0)
remote: Resolving deltas: 100% (6/6), completed with 6 local objects.
To https://github.com/spine-generic/data-multi-subject.git
   957506b3..d320819c  git-annex -> synced/git-annex
   3a107bc7..34c6c70b  jca/update-derivatives -> synced/jca/update-derivatives
ok

[kousu]

I thought that should fix it, but it didn't seem to work:

[kousu@requiem data-multi-subject]$ git annex sync
commit 
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
ok
pull origin 
remote: Enumerating objects: 33, done.
remote: Counting objects: 100% (33/33), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 17 (delta 6), reused 17 (delta 6), pack-reused 0
Unpacking objects: 100% (17/17), 1.32 KiB | 34.00 KiB/s, done.
From https://github.com/spine-generic/data-multi-subject
   957506b3..d320819c  git-annex                     -> origin/git-annex
   957506b3..d320819c  synced/git-annex              -> origin/synced/git-annex
   3a107bc7..34c6c70b  synced/jca/update-derivatives -> origin/synced/jca/update-derivatives
ok
(merging origin/git-annex into git-annex...)
(recording state in git...)
push origin 
Username for 'https://github.com': ^[[^C
[kousu@requiem data-multi-subject]$ git-annex whereis derivatives/labels/sub-vuiisAchieva06/anat/sub-vuiisAchieva06_T2star_rms_gmseg-manual.nii.gz
git-annex: derivatives/labels/sub-vuiisAchieva06/anat/sub-vuiisAchieva06_T2star_rms_gmseg-manual.nii.gz not found
git-annex: whereis: 1 failed

[kousu]

oh I was just on the wrong branch. It works:

[kousu@requiem data-multi-subject]$ git checkout jca/update-derivatives
Updating files: 100% (11/11), done.
Branch 'jca/update-derivatives' set up to track remote branch 'jca/update-derivatives' from 'origin'.
Switched to a new branch 'jca/update-derivatives'
[kousu@requiem data-multi-subject]$ ls -l derivatives/labels/sub-vuiisAchieva06/anat/sub-vuiisAchieva06_T2star_rms_gmseg-manual.nii.gz
-rw-r--r-- 1 kousu kousu 103 Sep  2 21:28 derivatives/labels/sub-vuiisAchieva06/anat/sub-vuiisAchieva06_T2star_rms_gmseg-manual.nii.gz
[kousu@requiem data-multi-subject]$ git-annex whereis derivatives/labels/sub-vuiisAchieva06/anat/sub-vuiisAchieva06_T2star_rms_gmseg-manual.nii.gz
whereis derivatives/labels/sub-vuiisAchieva06/anat/sub-vuiisAchieva06_T2star_rms_gmseg-manual.nii.gz (2 copies) 
    5a5447a8-a9b8-49bc-8276-01a62632b502 -- [amazon]
    5cdba4fc-8d50-4e89-bb0c-a3a4f9449666 -- julien@julien-macbook.local:~/code/spine-generic/data-multi-subject

  amazon: https://data-multi-subject---spine-generic---neuropoly.s3.ca-central-1.amazonaws.com/SHA256E-s28294--5e6dc442d20bb79ab0fd5dd62c5c5b968bf5926fc57b93fe02a1dad4366c3f0f.nii.gz
ok

[kousu]

So fixed?