Implement Croupier-proxied scp/rsync data transfer for HPC with Internet outgoing access disabled

ari-apc-lab / croupier

Cloudify plugin for HPCs and batch applications

https://hub.docker.com/repository/docker/marangiop/cloudify-croupier-ari-apc-lab

Apache License 2.0

6 stars 4 forks source link

Implement Croupier-proxied scp/rsync data transfer for HPC with Internet outgoing access disabled #16

Closed jesus-gorronogoitia closed 2 years ago

jesus-gorronogoitia commented 2 years ago

For those HPC cluster with disabled/blocked Internet access from HPC frontend (e.g. BSC MareNostrum4, HLRS Hawk), data transfer cannot be initiated from the HPC frontend but from Croupier host, either based on rsync, sftp or scp. Implement such Croupier-proxy based data transfer for scp or rsync.

Implement a test scenario transferring MN4 job result to another HPC cluster (e.g. CESGA)

Check this technology: https://www.linode.com/community/questions/11486/top-tip-how-to-transfer-filesdirectories-between-servers

jesus-gorronogoitia commented 2 years ago

PoC for CESGA -> Sodalite, CESGA -> MN4, MN4->CESGA conducted. In OpenSuse laptop (hosting Croupier) package sshfs should be installed. Key-base access to servers (source, target) is required. Slow transfer rate

jesus-gorronogoitia commented 2 years ago

script for proxied data_transfer created, based on above link. Tested with MN4 transfer to CESGA

jesus-gorronogoitia commented 2 years ago

Implemented for both HTTP and Rsync data transfer. Tested in scenarios:

Cloud ->(http) -> MN4 -> (rsync) -> Cesga
Cloud ->(http) -> Cesga -> (rsync) -> MN4

jesus-gorronogoitia commented 2 years ago

Implementation committed in commit https://github.com/ari-apc-lab/croupier/commit/eae64b6c70c9d10cd1d5c3855d4849a1cfe69b63

Merged into data_management branch