search

saga-project / BigJob

SAGA-based Pilot-Job Implementation for Compute and Data
http://saga-project.github.com/BigJob/
Other
19 stars 8 forks source link

Need to export/stage to remote data source #73

Open JackS9 opened 11 years ago

JackS9 commented 11 years ago

Not being able to export to or stage back a remote source has a tremendous impact on orchestrating workflows with chained data dependencies when the data source (and sink) is remote. Internal chaining from the output DU task to the input DU of a later task helps, but ultimately it would be best to stage/export files back to the original data source for subsequent dependent tasks. They would automatically get staged in from the remote source. We really need a full round-trip mechanism to do effective workflow with data chaining. This would also eliminate re-sync afterward, especially when dealing with very large files.

A hybrid method could use DU-to-DU chaining to effectively cache large files locally and avoid copying chained files twice, but it would still be useful to stage files to the remote source incrementally instead all at once at the end. This is especially important if the BigJob fails mid-stream for some reason. Restarting from orphaned DUs would be nightmare to manage.

drelu commented 11 years ago

The aim of the abstraction is to work on Pilot Data level. You can stage out files to a remote Pilot Data (e.g. on your submission machine). From there you can export it (which is just a symlink by default).

melrom commented 11 years ago

Hi Andre,

Can we capture a remote data flow example in our example scripts?

Is it possible to spawn the Pilot Data on the remote resource with the same BigJob script that you are using to execute your jobs? Or do you need a separate script that runs on the remote resource itself which starts a Pilot Data? Does this imply that BigJob must be installed on the remote resource?

I understand what you're saying by create the Pilot Data on the remote resource, move the DUs to that Pilot Data, then use export to get the DUs wherever you want on the local filesystem. I just had the above technical questions, so that I can help Jack understand the data movement better and also improve the Sphinx documentation.

Thank you! -Melissa

melrom commented 11 years ago

Created: http://saga-project.github.com/BigJob/sphinxdoc/patterns/pdata.html to try to help Jack.