search

cooperative-computing-lab / cctools

The Cooperative Computing Tools (cctools) enable large scale distributed computations to harness hundreds to thousands of machines from clusters, clouds, and grids.
http://ccl.cse.nd.edu
Other
131 stars 111 forks source link

Vine: Remote Transfer Fallback Mode #3281

Open dthain opened 1 year ago

dthain commented 1 year ago

In some environments, it will not be possible for workers to perform URL downloads directly. Instead, the manager must perform the download and substitute the URL with a vine_file_local. Add an option to the manager so the user can enable this mode in one place easily.

dthain commented 2 weeks ago

This is similar to #3756 but backwards: when encountering a VINE_FILE_URL, the manager downloads it to the (manager-side) cache, and then substitutes an appropriate VINE_FILE instead.