search

geodesymiami / rsmas_insar

RSMAS InSAR code
https://rsmas-insar.readthedocs.io/
GNU General Public License v3.0
60 stars 23 forks source link

Moving all job submission functions to job_submission.py (currently create_batch.py) #154

Closed falkamelung closed 5 years ago

falkamelung commented 5 years ago

Hi @mirzaees I would suggest to move the submit_run_jobs function into create_batch.py and import form there. https://github.com/geodesymiami/rsmas_insar/blob/fe56fac6aa0752c806a90dbab9456a44a8613475/rinsar/execute_runfiles.py#L44

Lily's blurb regarding renaming of create_batch is below. She suggest to use job_submission.py and it would use defaults/job_submission.cfg (currently job_defaults.cfg). Sara and @2gotgrossman , how do you think?

Lily: "As for renaming create_batch.py, it also has functions for submitting a single job (including a script like process_rsmas or download_rsmas) so I think maybe something like job_submission.py might be more appropriate?"

mirzaees commented 5 years ago

Hi @falkamelung and @2gotgrossman

I agree with job_submission.py I have another question before renaming I want to use create_batch.py as a module to submit a batch_file (each run_file). for that I have to run following lines from create_batch.py right?

JOBS = write_batch_job_files(PARAMS.file, PARAMS.outdir, memory=PARAMS.memory, walltime=PARAMS.wall, queue=PARAMS.queue)
submit_batch_jobs(JOBS, PARAMS.file, PARAMS.outdir)

Is it ok if I create a function and move these lines there instead of having them under if __name__ == "__main__":

or is there a better way of doing that?

Right now, in my submit_run_jobs (from execute_runfiles.py) I am calling create_batch with subprocess:

cmd = 'create_batch.py ' + cwd + '/' + item + ' --memory=' + memorymax + ' --walltime=' + walltimelimit + \
               ' --queuename ' + queuename + ' --outdir ' + outdir
status = subprocess.Popen(cmd, shell=True).wait()

I need to use create_batch.py in the same way as you sumbit a single script. then I can remove submit_run_jobs from execute_runfiles.py

falkamelung commented 5 years ago

I don’t know, unfortunately. I never really looked at it ad It just worked. You may want to ask Lily.

Falk Amelung Professor Department of Marine Geosciences Rosenstiel School of Marine and Atmospheric Sciences University of Miami 4600 Rickenbacker Causeway Miami, FL 33149 USA Tel: 305 421 4949 E-mail: famelung@rsmas.miami.edumailto:famelung@rsmas.miami.edu Web: http://insar.rsmas.miami.edu InSAR data: http://insarmaps.miami.edu

On May 2, 2019, at 11:08 PM, Sara Mirzaee notifications@github.com<mailto:notifications@github.com> wrote:

Hi @falkamelunghttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffalkamelung&data=02%7C01%7Cfamelung%40rsmas.miami.edu%7Cdc724549a38646c573af08d6cf10155d%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C636924065327009751&sdata=W1j4%2B360O5%2FhiOwkEQb%2BLf52x1QJEcYk03eTSDiMSJ0%3D&reserved=0 and @2gotgrossmanhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2F2gotgrossman&data=02%7C01%7Cfamelung%40rsmas.miami.edu%7Cdc724549a38646c573af08d6cf10155d%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C636924065327009751&sdata=XSSeZ0VQJKn0Z718ATTQ9OK5Sa6ita7VrEjDPzsx%2B30%3D&reserved=0

I agree with job_submission.py I have another question before renaming I want to use create_batch.py as a module to submit a batch_file (each run_file). for that I have to run following lines fr:om create_batch.py right?

JOBS = write_batch_job_files(PARAMS.file, PARAMS.outdir, memory=PARAMS.memory, walltime=PARAMS.wall, queue=PARAMS.queue) submit_batch_jobs(JOBS, PARAMS.file, PARAMS.outdir)

Is it ok if I create a function and move these lines there instead of having them under if name == "main":

or is there a better way of doing that?

Right now, in my submit_run_jobs (from execute_runfiles.py) I am calling create_batch with subprocess :

cmd = 'create_batch.py ' + cwd + '/' + item + ' --memory=' + memorymax + ' --walltime=' + walltimelimit + \ ' --queuename ' + queuename + ' --outdir ' + outdir status = subprocess.Popen(cmd, shell=True).wait()

I need to use create_batch.py in the same way as you sumbit a single script. then I can remove submit_run_jobs from execute_runfiles.py

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgeodesymiami%2Frsmas_insar%2Fissues%2F154%23issuecomment-488711110&data=02%7C01%7Cfamelung%40rsmas.miami.edu%7Cdc724549a38646c573af08d6cf10155d%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C636924065327019756&sdata=9K%2B2O%2BZW9bJ7sXwmKgHuD86j5F74YWCeJyxb7xTGNxY%3D&reserved=0, or mute the threadhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACVFHXD5HUD5NQFPT34H553PTL7YDANCNFSM4HJ3NKHQ&data=02%7C01%7Cfamelung%40rsmas.miami.edu%7Cdc724549a38646c573af08d6cf10155d%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C636924065327019756&sdata=1abos9SbTJI%2BozAUOWGSoz1cdDPQdS1KshzhCgBZgyM%3D&reserved=0.

mirzaees commented 5 years ago

Hi @lily-wittle

Could you help me with this?