search

ATLAS-Titan / PanDA-NGE

Integration between PanDA and Next Generation Executer (NGE)
0 stars 0 forks source link

Data placement and movement #15

Open mturilli opened 7 years ago

mturilli commented 7 years ago

PanDA Pilot expects input and output data to reside on OLCF Lustre fs:

  1. PanDA Pilot includes location of input/output files in the task description pushed to the integration DB via the REST integration API
  2. NGE pulls the task description from the DB and uses input/output paths for the CU
  3. RP uses the path in the CU to stage the input files from Lustre into the pilot sandbox (Andre to confirm)
  4. RP executes the CU
  5. RP stages the output files of the CU to the path on Lustre indicated in the CU description
  6. RP returns handle to NGE
  7. NGE returns handle to PanDA Pilot
  8. PanDA Pilot finds the output files of the task on the Lustre fs.
mturilli commented 7 years ago

Andre to provide an updated API by Aug 8

andre-merzky commented 7 years ago

update: the API did not change for the data staging, but the NGE server now has the right hooks in place to do the staging expected by the workload (3, 5). Well, 'hooks' means really just 'FIXME' comments, as I am not sure how the staging directives look like, but that is a minor details ;) As soon as we start to deploy, we'll flesh this out with actual staging code (which are one-liners to translate the Panda directives into RP pilot staging directives).

mturilli commented 7 years ago

Depends on #16 and #24