Open tsalo opened 10 months ago
I wonder if the problem is that I'm trying to push to an organization repository instead of my personal account.
Except when I try it with my personal account, I get create_sibling_gin(error): [Organization does not exist]
dataset_id="ds002156"
base_dir="/cbica/home/salot/open-multi-echo-data/datasets"
superdataset_dir=${base_dir}/${dataset_id}_test
raw_dataset_dir=${superdataset_dir}/inputs/data
code_dir="/cbica/home/salot/open-multi-echo-data/code/code"
# Create the YODA superdataset
datalad create -c yoda \
-D "Create superdataset for OpenNeuro dataset ${dataset_id}" \
"${superdataset_dir}"
cd ../../datasets/ds002156_test/
# Clone the dataset
datalad clone -d ${superdataset_dir} \
-D "Clone of OpenNeuro dataset. May be modified for fMRIPrep/AFNI and pushed to G-Node GIN." \
https://github.com/ME-ICA/${dataset_id}.git ${raw_dataset_dir}
# Download the files
cd ${raw_dataset_dir}
datalad get ${raw_dataset_dir}
# Create the GIN repo (this fails!)
datalad create-sibling-gin \
--access-protocol ssh \
--dataset ${raw_dataset_dir} \
--credential GIN \
tsalo/${dataset_id}_raw
EDIT: From @\adswa (don't want to subscribe her to this issue) this error occurs because it interprets tsalo/XX
as a repo in the organization tsalo
. I need to drop the tsalo/
in the repo name for it to go to my personal account.
No great wisdom, but I've had consistent issues interacting with openneuro from the NIH HPC. In my case it's urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)>
because requests.get()
isn't playing well with our system. Your error is different, but I wonder if it's a similar firewall or security issue. Instead of using all of datalad
, could you try just the command that is fetching the data to see if that's causing the problem.
Fetching the data works fine, as does creating the repo on G-Node GIN. The step that fails is actually pushing the data to the GIN repository.
Some more updates:
~/.ssh/config
and without.
Host g-node.gin.org
Hostname ssh.g-node.gin.org
IdentityFile ~/.ssh/id_ed25519
Port 443
Running with that info in the config file leads it to fail more quickly, but the error messages are the same.
DATALAD_CREDENTIAL_GIN_TOKEN
or not.I joined the Datalad office hour yesterday (where everyone was extremely helpful), and the problem appears to be that UPenn's CUBIC cluster blocks outgoing traffic through Port 22, which is the only one that G-Node GIN accepts for SSH. GitHub works because it uses Port 443 (which is typically used for HTTPS). Also, creating the sibling repo uses HTTPS, which is why that worked fine.
I have four options:
Of the four, I'd recommend 1 & 2.
Filing a ticket with CUBIC will either semi-efficiently let you keep working on this or get a response explaining why this is an actual security issue (which you could share with G-Node GIN).
This is going to be an issue with other clusters and if there's a solution from G-Node GIN, then this is broadly solved.
Any clue if this might be the same issue I'm having with openneuro and the NIH cluster? (i.e. the ssh certificate is being blocked/garbled on one port, but might work if openneuro transfers data over another port?)
Any clue if this might be the same issue I'm having with openneuro and the NIH cluster? (i.e. the ssh certificate is being blocked/garbled on one port, but might work if openneuro transfers data over another port?)
It definitely could be. If you're using datalad to push to OpenNeuro I'd recommend joining one of the weekly datalad office hours like I did. They had me try out a series of commands to see what the situation was on CUBIC and were able to diagnose the problem.
After speaking with Chris M., it might be inappropriate to push derivatives to GIN anyway. OpenNeuro seems to support derivatives-only datasets now, so I might want to switch to that instead.
EDIT: The problem with that is that the openneuro credential tool crashes the UPenn cluster's login node. See https://github.com/OpenNeuroOrg/openneuro/issues/3015.
I created an SSH key and added it to my G-Node GIN user settings, but for some reason I can't push to it from the UPenn HPC.
Steps to reproduce:
This gets me the following error: