need ability to specify walltime for Dask

falkamelung commented 5 years ago

Hi @2gotgrossman : Pysar with Dask works great. Thank you!

Issues:

for longer jobs we need longer walltime then 30 min specified in the dash_pysar.yaml
its location in ~/.config/dask is not good if it is updated for different runs.

Suggestion: For each Dask invocation we copy the /PYSAR/defaults/dask_pysar.yaml into the projectname/PYSAR dir and modify it for the specific case. The walltime could be adjusted as a PySAR option:

pysar.networkInversion.parallel     = yes              # [yes / no], auto for no, parallel 
pysar.networkInversion.parallel.walltime   = 3:00         #  walltime in HH:MM, auto for 0:30
processing using dask

Are there any other options we may want to modify (e.g. number of workers? - I don't see it).

Any thoughts?

Others:

Some documentation on it on the pysar page would be great, including how to assess progress.
I did not see messages such as 'inversion split into 200 chunks' and information when chunks are finished.
there are two incidences of walltime. Which one is used: jobqueue or Ingram_inversion?
Who can/like/will do this?

yunjunz commented 5 years ago

Hi @falkamelung,

The number of workers can be specified through pysar.networkInversion.numWorker: https://github.com/insarlab/PySAR/blob/master/pysar/defaults/pysarApp_template.txt#L109

falkamelung commented 5 years ago

Great! Then we just need an option to specify the wall time.

falkamelung commented 5 years ago

I see that LSFCluster has a walltime option. So it seems pretty straightforward to add a walltime similar to the number of workers. To be consistent with that option, it probably should be pysar.networkInversion.walltime. @yunjunz @mirzaees

yunjunz commented 5 years ago

Feel free to try it when you got a chance @falkamelung. I guess this should be easy for someone who is familiar with dask_jobqueue too.

falkamelung commented 5 years ago

Given that mintpy is more widely used than minsar or minopy I will rename the dask_pysar.yaml to dask.yaml and adjust the minsar DASK_CONFIG accordingly. @yunjunz @mirzaees please yell if that is not a good idea.

And remove the specifics for ifgram_inversion.py as they are not used anyway.

Actually, the minsar desk has three files (see below) , whereas pysar has only one. I suppose we need all three? @yunjunz @mirzaees //login3/nethome/famelung/test/development/rsmas_insar/minsar/defaults/dask[1060] ls * dask.yaml distributed.yaml jobqueue.yaml

yunjunz commented 5 years ago

Test before your pull request please.

Riley-coder7 commented 5 years ago

there are some questions when i install the software on ubuntu16.04

yunjunz commented 5 years ago

there are some questions when i install the software on ubuntu16.04

Hi @Keplere, please post your installation related question in a separated issue or in the google group, with more details, to help the diagnose.

insarlab / MintPy

need ability to specify walltime for Dask #59