search

radical-collaboration / QCArchive

2 stars 0 forks source link

File staging #11

Closed mturilli closed 5 years ago

mturilli commented 5 years ago

QA would like to have RP to do staging of output files on an arbitrary location or, if possible, via MongoDB (see how FireWorks does it). Staging out onto a remote machine would be also a good option. A third option would be to use a REST interface to manage file staging (as QA does it at the moment).

mturilli commented 5 years ago

RADICAL is happy to use the REST interface. We would like to see the code of QA REST interface where data staging is handled.

mturilli commented 5 years ago

Andre to have a look at fireworks.

dgasmith commented 5 years ago

This line tells Fireworks to store the result, which is then pulled out here.

andre-merzky commented 5 years ago

Thanks!

mturilli commented 5 years ago

Rutgers ready to support this. Will be coded once support for task has been developed.

mturilli commented 5 years ago

To be implemented after #8

Lnaden commented 5 years ago

~QCA will try to limit to about 10 tasks/second~

Redundant with comment below which is more detailed

mturilli commented 5 years ago

RP to stage out files to QCA. RADICAL to optimize output staging as done for input staging. Currently, we should limit output staging to around 10 tasks/s. Required by #13

mturilli commented 5 years ago

Json is enough and already supported.