tlvu
commented
1 year ago I vaguely remember we discussed this over email and as usual I can not find that email thread anymore. Email is so bad as long term storage of information.
Not sure what conclusion or work-around we did come up with. If someone find the thread or remember the discussion, please update this issue.
Thinking about this again, I think the simplest way is for each user to start a personal Globus node. Then transfers will be handled by Globus, no more problems with VPN or transfer bandwidth on ECCC side or firewall or "landing spot" on Ouranos side.
Hi there,
At the federal government, we do a lot of work on Linux compute servers that are behind fairly strict VPNs, so that the systems are (appropriately) not accessible if outside this VPN (e.g., PAVICs). This makes it challenging (impossible) to, while on PAVICs, 'pull' as-needed files from these VPN'd servers to PAVICs via any method (e.g. an scp call from PAVICs to VPN'd server to access a file). The alternative option would be for us to 'push' files out from these VPN'd clusters to PAVICs (e.g. an scp call from VPN'd server to PAVICs to push a file). But in this case we'd need a 'landing spot' in our writeable-workspace or similar for these files, so we could use command line tools, e.g. scp, to make this transfer directly, with defined PAVICs-side server/file path information. Does such a space exist, and/or is this possible to do (or implement) in PAVICs so that users (power users?) have capability to push files to a PAVICs workspace? Lacking this, our practical approach to as-needed file sharing across platforms is unfortunately so far to route files to our local laptops from our VPN'd servers, and then from our laptops to PAVICs. Which bottlenecks collaborations on data projects between these two HPC systems by home internet speeds and local laptop disk spaces, which isn't particularly graceful (let alone scriptable/trackable/reproducible). Thoughts welcome on how we could improve this flow!
Sincerely,
@JeremyFyke @cpomer10