falkamelung
commented
4 years ago I tried the parallel queue. It works fine! (see below) The jobs need an ‘&' at the end of the line.
So I suggest to check for the existence of the environment variable below and do the sixteen-jobs-per-node only if the variable exists, as indicated by NEW below. Once this is confirmed to be working we change the rest, which obviously a more significant step.
If we do it this way, it can be in master without affecting anything (unless the variable is set)
-rw-rw--w-+ 1 dwg11 0 Mar 8 13:29 run_9_16parallel_parallel_23348462.e
-rw-rw--w-+ 1 dwg11 71601 Mar 8 13:29 run_9_16parallel_parallel_23348462.o
-rw-rw--w-+ 1 dwg11 0 Mar 8 14:16 run_9_5parallel_parallel_23348482.e
-rw-rw--w-+ 1 dwg11 6283 Mar 8 14:16 run_9_5parallel_parallel_23348482.o
-rw-rw--w-+ 1 dwg11 2266 Mar 8 14:42 run_9_10parallel_parallel.job
-rw-rw--w-+ 1 dwg11 2285 Mar 8 14:46 run_9_16parallel_parallel.job
-rw-rw--w-+ 1 dwg11 0 Mar 8 17:57 run_9_10parallel_parallel_23348605.e
-rw-rw--w-+ 1 dwg11 46207 Mar 8 17:57 run_9_10parallel_parallel_23348605.o
-rw-rw--w-+ 1 dwg11 0 Mar 8 19:41 run_9_16parallel_parallel_allthreads_.23348606.e
-rw-rw--w-+ 1 dwg11 71638 Mar 8 19:41 run_9_16parallel_parallel_allthreads_23348606.o if os.getenv('JOBSCHEDULER') in ['SLURM', 'sge']:
submit_job_with_launcher(batch_file=batch_file, out_dir=out_dir, memory=maxmemory, walltime=wall_time,
number_of_threads=num_threads, queue=queue)
else:
NEW if os.getenv('JOB_SUBMISSION_SCHEME'):
NEW jobs = submit_sixteen_jobs_per_node(batch_file=batch_file, out_dir=out_dir,
NEW else:
jobs = submit_jobs_individually(batch_file=batch_file, out_dir=out_dir, memory=maxmemory,
walltime=wall_time, queue=queue)
Ovec8hkin
JOB_SUBMISSION_SCHEME= RUNFILETYPE_METHOD_SCHEDULER_QUEUENAME
These environment variables should be read at the beginning job_submission.py :
and will be available as PARAMS.scheduler, PARAMS.queue, PARAMS.number_of_cores_per_node, etc (Sara, we have PARAMS already, shall we use something else?)
We need these functions (run from execute_runfiles.py):
submit_launcher_jobs (currently: submit_with_launcher) submit_serial_jobs (currently: submit_jobs_individually) submit_parallel_jobs
write_job_file ( run_file_type=singlefile, method= serial)
(currently we have for method=serial write_batch_job_files and for method=launcher the lines of submit_with_launcher until 'job_number = submit_single_job...' )
get_job_header_lines() get_job_file_lines(run_file_list)
cat /nethome/sxh733/development/falk/rsmas_insar/sources/roipac/INT_SCR/split_jobs.py cat /nethome/sxh733/development/falkold_keep/rsmas_insar/sources/roipac/INT_SCR/split_igramJobs_filt.py cat /nethome/sxh733/development/rsmas_insar/sources/roipac/INT_SCR/runJobs.py