MWATelescope / manta-ray-client

Python API and helper scripts to interact with the MWA ASVO.
9 stars 7 forks source link

allow mwa_client to submit jobs and start downloads in separate calls #13

Closed PaulHancock closed 6 years ago

PaulHancock commented 6 years ago

The help documentation states that the mwa_client will submit a conversion job, and then block until the data are converted and downloaded. In practice on the pawsey machines this means that a job will sit on the Zeus copyq doing nothing whilst the ASVO machines retrieve data from the archive and run cotter. This is a poor use of the Zeus copyq.

It would be nice to allow the mwa_client to act more like the ASVO website so that a user can submit a list of jobs and then disconnect. At a later time the user can then rejoin and request the status of each job, and download when the jobs are complete. It would also make it possible for the job submission scripts to be run from any where, and only the actually downloads need to be run on the copyq.

davepallot commented 6 years ago

It's pretty straight forward to separate submit from download. In fact the API is there if someone whats to have a go at it.