search

Parsl / parsl

Parsl - a Python parallel scripting library
http://parsl-project.org
Apache License 2.0
507 stars 196 forks source link

Implement local file staging #344

Open annawoodard opened 6 years ago

annawoodard commented 6 years ago

Currently, in order to have parsl transfer a local file for you, you must either:

1) set up an FTP server locally and use File(’ftp:...), 2) set up an HTTPS server locally and use File(’https:...), 3) set up a GridFTP server locally (e.g. Globus Connect Server or Globus Connect Personal) and use File(’globus:...).

We need a simple way of transferring a file that exists locally which does not require setting up a server. Probably using channel.push_file and channel.pull_file would cover 90% of our use cases (the caveat would be it wouldn't work if workers did not have shared filesystem access). It also has the advantage of already being authenticated.

benclifford commented 5 years ago

This is doable with rsync as of #1378.

There might still be value in a channel based provider so I'll leave this open.