Provide large data transfer service to CANFAR Science Platform storage

opencadc / science-platform

Science Platform Infrastructure

GNU Affero General Public License v3.0

14 stars 27 forks source link

Provide large data transfer service to CANFAR Science Platform storage #291

Open sfabbro opened 2 years ago

sfabbro commented 2 years ago

Many times users want to transfer large amount of data in an reliable and unattended manner from and to the science storage platform (/arc). The idea is to provide a user centric web-based solution to allow this transfer beyond the current workaround for vcp/sshfs.

Some ideas:

provide an endpoint to Globus - to allow transfer to Computer Canada HPC systems and other US-based data centres (SDSS,....)
provide an endpoint for CERN-FTS, a reliable transfer service Both of them support OIDC and multiple protocols.

fraserw commented 2 years ago

In my eyes, ease of use is less important than reliability. It seems like half the time vcp fails (so I use your vcp.sh looper to get it to a ocuple percent failure) or vcp from arc can't even find the files that do actually exist (and can fail during transfer anyways).

It seems to me fixing file transfers with current tools might be a higher priority item.

sfabbro commented 2 years ago

Agree - reliability is priority. This is what those two services aim for. My guess is that adding a globus endpoint is a lot less work than fixing the vcp/vos for reliable tranfers. Even with a fully reliable vos, launching a terminal to do ssh + screen + vcp to do transfers is not welcoming to new users. For example, one can not use vcp for transfers between SDSS and CANFAR storage, but could use globus.