search

psychoinformatics-de / studyforrest-data

DataLad superdataset of all studyforrest.org project dataset components
https://studyforrest.org
8 stars 2 forks source link

Ensure a several hosting locations for data #14

Closed adswa closed 3 years ago

adswa commented 3 years ago

It seems that only the datapub server has data for the structural data. I realized that today because the HDF cloud is down for maintenance: 

datalad clone https://github.com/psychoinformatics-de/studyforrest-data-structural.git
[INFO   ] Scanning for unlocked files (this may take some time)                 
[INFO   ] Remote origin not usable by git-annex; setting annex-ignore           
[WARNING] Failed to fetch type=git special remote mddatasrc: CommandError: 'git -c diff.ignoreSubmodules=none fetch --verbose --progress mddatasrc' failed with exitcode 128 under /tmp/studyforrest-data-structural [err: 'warning: redirecting to https://datapub.fz-juelich.de/studyforrest/studyforrest/structural/.git/
fatal: https://datapub.fz-juelich.de/studyforrest/studyforrest/structural/.git/info/refs not valid: is this a git repository?'] [cmd.py:run:408] 
install(ok): /tmp/studyforrest-data-structural (dataset)
(handbook2) adina@muninn in /tmp
❱ cd studyforrest-data-structural 
(handbook2) adina@muninn in /tmp/studyforrest-data-structural on git:master
❱ ls
CHANGES  README.rst  sub-03  sub-06  sub-09  sub-12  sub-15  sub-18  sub-21
ISSUES   sub-01      sub-04  sub-07  sub-10  sub-13  sub-16  sub-19
LICENSE  sub-02      sub-05  sub-08  sub-11  sub-14  sub-17  sub-20
(handbook2) adina@muninn in /tmp/studyforrest-data-structural on git:master
❱ datalad get sub-01/anat/sub-01_SWI.json
[ERROR  ] Unable to parse git config from http://psydata.ovgu.de/studyforrest/structural/.git/config
| Remote mddatasrc not usable by git-annex; setting annex-ignore [get(/tmp/studyforrest-data-structural/sub-01/anat/sub-01_SWI.json)] 
get(error): sub-01/anat/sub-01_SWI.json (file) [Unable to parse git config from http://psydata.ovgu.de/studyforrest/structural/.git/config
Remote mddatasrc not usable by git-annex; setting annex-ignore]

Maybe there are more datasets with only one working registered source.

adswa commented 3 years ago

This can - at least partially - be addressed by #8

adswa commented 3 years ago

Technical detail: GIN repositories will need to be registered as a Git-type special remote for git-annex: https://git-annex.branchable.com/special_remotes/git/

mih commented 3 years ago

Hosted now internally on the cluster, on the storage server.

mih commented 3 years ago

All core derivatives are on GIN too now. The public datasets in the internal store are also copied to a public store on datapub.

As a first approximation of the issue spirit -- this is done.