kousu
commented
4 years ago While logged onto my admin account on the dataset server (since it's a convenient, and ethics-secure place, and with a conveniently fast network to be doing the migrations), follow the instructions at https://www.neuro.polymtl.ca/internal_resources/list_of_computers#duke to get access to this data:
nguenther@data:~$ sudo mount -t cifs //duke.neuro.polymtl.ca/temp /mnt/duke/temp -o username=u[REDACTED],noexec
[sudo] password for nguenther:
🔐 Password for u[REDACTED]@//132.207.65.200/temp: (no echo)
nguenther@data:~$ ls /mnt/duke/temp/
alfoi andreanne charley EvaAlonsoOrtiz harris jcohen lrouhier mariehbourget matlab olivier qc_feedback sebeda uk_biobank_BIDS zougloub
nguenther@data:~$ du -hs /mnt/duke/temp/uk_biobank_BIDS/
9.6G /mnt/duke/temp/uk_biobank_BIDS/Import it to local storage, because we suspect using git/git-annex/datalad over smb is glitchy (#12, #15?, #16):
nguenther@data:~$ time rsync -a /mnt/duke/temp/uk_biobank_BIDS/ datasets/uk-biobank
real 2m31.438s
user 0m8.327s
sys 0m23.644s
nguenther@data:~$ du -hs datasets/uk-biobank/
9.6G datasets/uk-biobank/Now annex it. I don't have proper docs on this yet, I just keep referring back to the last time I annexed a dataset. I have these to work from:
- https://github.com/spine-generic/data-multi-subject_DO-NOT-USE/issues/20#issuecomment-672180622
- https://github.com/spine-generic/data-single-subject/issues/5
- https://github.com/neuropoly/datalad/issues/22#issuecomment-740573053
First, save a ton of local space (#23); this risks losing data, but only if you don't sync to the server eagerly, which is not, I think, going to be a problem in our usage:
git config --global annex.thin trueThen run this:
git init
cat > .gitattributes <<EOF
*.nii filter=annex annex.largefiles=anything
*.nii.gz filter=annex annex.largefiles=anything
EOF
git add README.txt && git commit -m "Initial commit"
git annex init
git add .gitattributes && git commit -m "Configure git-annex"
time git add .
git commit -m "Migrate dataset from smb://duke.neuro.polymtl.ca/temp/uk_biobank_BIDS to git-annex."
git annex dead here # this is *not* a source for others to download from
git remote add origin git@data.neuro.polymtl.ca:datasets/uk-biobank
git annex sync --content origin
git annex sync --content # do it a second time because the first one glitchedupload log
``` nguenther@data:~/datasets/uk-biobank$ git init Initialized empty Git repository in /home/nguenther/datasets/uk-biobank/.git/ nguenther@data:~/datasets/uk-biobank$ cat > .gitattributes <It's weird to have to run sync twice but that's the beans because gitolite needs push to create the repo, butgit-annextries to start by syncing from an existing repo to make sure it's up to date. I guess I should have started withgit push origin`.
Checking the work:
nguenther@data:~/datasets/uk-biobank$ git annex whereis sub-1047359/
whereis sub-1047359/anat/sub-1047359_T1w.nii.gz (1 copy)
ea71146c-9360-4041-8378-87ab0dfff167 -- git@data.neuro.polymtl.ca:~/repositories/datasets/uk-biobank.git [origin]
ok
whereis sub-1047359/anat/sub-1047359_T2w.nii.gz (1 copy)
ea71146c-9360-4041-8378-87ab0dfff167 -- git@data.neuro.polymtl.ca:~/repositories/datasets/uk-biobank.git [origin]
ok
alexfoias has made a minimal viable dataset out of our copy of the UK Biobank data for doing some experiments on. It is currently sitting on our internal server at
smb://duke/tmp/uk_biobank_BIDS. Julien wants it to be nameduk-biobank. I can do that along the way.