how to transfer raw data between universities

[lpantano]

Harvard is subscribed to globus.org. If the people have their institution with that service we can transfer this way:

talk to @sjhosui to get an idea of how it works.

you log in with your harvard credentials.

[sjhosui]

Once a user has set up an endpoint for you to download the data from using Globus, you will receive an email with a URL to access the share. It will be of the form https://www.globus.org/app/transfer?&origin_id=______&origin_path=/&add_identity=______.

Clicking on the link will take you to the Globus Web App. You will need to select "Harvard University" as your institute.

When you click "Continue", you may be redirected to an error page. Ignore this and refresh your browser. It will take you to the HarvardKey login page. Use your HarvardKey credentials to login and you will be taken to the "Transfer files" page that lists the files available for download from their endpoint on the left, and your endpoint on the right.

To set up your endpoint, click on "Endpoints" in the menu above the file browser. Enter a name for your endpoint, and click "Generate a set up key". Download and install Globus Connect Personal

Note that this is to transfer files to your personal machine. To transfer files directly to O2, instructions are available at https://wiki.rc.hms.harvard.edu/display/O2/Globus+Connection+Personal+from+command+line.

Use the set up key to complete the installation. Return to "Transfer files" and select your new endpoint in the right hand panel.

Transfer files as you would for a typical FTP client. Select the directory or file in the left panel, then click the right arrow to move it to your endpoint in the right panel.

[sjhosui]

Just to re-iterate in case you missed it, the above instructions allow you to set up your own machine as an endpoint. To transfer data directly to O2, follow the instructions at https://wiki.rc.hms.harvard.edu/display/O2/Globus+Connection+Personal+from+command+line.

[jnhutchinson]

Has anyone gotten the directions for direct data transfer to O2 to actually work? I can't even login into the transfer node they way they outline. Logging into an interactive node also doesn't work, globus isn't installing.

[jnhutchinson]

I'm following up with HMS RC

[jnhutchinson]

In the instructions, you can a) skip the transfer node login (we don't have permissions) and b) don't use the system packages when setting up the virtual environment (Thanks to Ilya for figuring this out!) as this will try to link to a system file that we also can't access.

So instead of "virtualenv venv --system-site-packages" just use "virtualenv venv"

I emailed HMS RC with this info