search

geodesymiami / rsmas_insar

RSMAS InSAR code
https://rsmas-insar.readthedocs.io/
GNU General Public License v3.0
58 stars 22 forks source link

updating wall time if timeout #497

Closed mirzaees closed 3 years ago

mirzaees commented 3 years ago

Recently the updating wall time in case a job is timed out, does not work . I checked update_walltime.py script and it works but the problem is from run_workflow.bash and how it is implemented there.

@falkamelung , @Ovec8hkin you might have a quick solution

https://github.com/geodesymiami/rsmas_insar/blob/549cd1e42829395751e23f2855c38f31b08076f9/minsar/run_workflow.bash#L297-L310

Ovec8hkin commented 3 years ago

Im gonna need more information than that. How many times did the TIMEOUT occur consecutively? Did it succeed in updating wall time 1+ times and then fail afterwards, or did it fail immediately once a TIMEOUT occured? Does this error always happen with the given dataset/runfile or is it a one off failure?

I haven't experience any issues for a while with this functionality, and it hasn't been updated recently.

mirzaees commented 3 years ago

Thanks for replying @Ovec8hkin! you are right, it is working. It was my mistake...

I am closing the issue