Improved job_defaults.cfg format

[falkamelung]

Hi @Ovec8hkin @mirzaees
We currently have our job_defaults.cfg in the row-type format below. It is clumsy. Adding a new field is an effort. It would be preferred to have a column-type format as shown here. Are there any modules around that we could use for this? I am cc'ing @yunjunz maybe he has some idea.

name              walltime  adjust  num_threads  memory
DEFAULT           02:00     True    1            4000
process_rsmas     02:00     True    1            4000
create_runfiles   02:00     True    1            4000

//login3/home1/05861/tg851601/test/operations/rsmas_insar/minsar/defaults[1023] cat job_defaults.cfg
## HPC job setting
##-----------------------------  ISCE2/topsStack  ------------------------------##

[DEFAULT]
walltime = 10:00
memory = 3000
adjust = Fale
num_threads = 4

[process_rsmas]
walltime = 20:00
memory = 1000
adjust = False
num_threads = 4

[create_runfiles]
walltime = 00:10
memory = 1000
adjust = False
num_threads = 4

[execute_runfiles]
walltime = 20:00
memory = 1000
adjust = False
num_threads = 4

### ISCE topsSatck #####
#[unpack_slc_topo_master]
#walltime = 00:30
#memory = 4000
#adjust = True
#num_threads = 16

[unpack_topo_master]
#sara walltime 00:03
walltime = 00:03
memory = 4000
adjust = True
num_threads = 4

[unpack_slave_slc]
walltime = 00:10
memory = 4000
adjust = False
num_threads = 2

[average_baseline]
walltime = 00:06
memory = 1000
adjust = False
num_threads = 2

[extract_burst_overlaps]
walltime = 00:03
memory = 4000
adjust = False
num_threads = 16

[overlap_geo2rdr_resample]
walltime = 00:04
memory = 4000
adjust = True
num_threads = 2

[pairs_misreg]
# from sara
#walltime = 00:03
# from BalotschistanSenDT run_6_pairs_misreg  8   01:18:00   00:12:21   00:08:31   00:00:19
# but Kilauea 00:01:15
walltime = 00:01
memory = 4000
adjust = True
num_threads = 2

[timeseries_misreg]
walltime = 0:02
memory = 4000
adjust = False
num_threads = 16

[geo2rdr_resample]
# was at 00:08. may want to go to 00:05
walltime = 00:04
memory = 5000
adjust = True
num_threads = 4

[extract_stack_valid_region]
#sara adjust false 00:10
walltime = 00:01
memory = 4000
adjust = True
num_threads = 4

[merge_master_slave_slc]
walltime = 00:20
memory = 4000
adjust = False
num_threads = 2

[merge_burst_igram]
#sara: adjust=False 00:30
walltime = 00:02
memory = 4000
adjust = True
num_threads = 2

[filter_coherence]
walltime = 0:02
memory = 6000
adjust = True
num_threads = 2

[unwrap]
walltime = 0:01
memory = 5000
adjust = True
num_threads = 2
#### end ISCE topsStack #####

# jobs since 2020
#run_10_merge_master_slave_slc           1   00:20:00   00:02:49   00:01:02   00:00:02
#run_11_merge_burst_igram                1   00:52:00   00:02:20   00:07:07   00:00:16
#run_12_filter_coherence                 1   00:52:00   00:01:53   00:08:34   00:00:19
#run_13_unwrap                           1   00:26:00   00:02:00   00:01:04   00:00:02

[smallbaseline_wrapper]
walltime = 5:00
memory = 2000
adjust = False
num_threads = 4

[grid_baseline]
walltime = 00:10
memory = 4000
adjust = True
num_threads = 4

[generate_igram]
walltime = 00:10
memory = 4000
adjust = True
num_threads = 4

[single_master_interferograms]
walltime = 2:00
memory = 4000
adjust = False
num_threads = 4

[amplitude_ortho_geo]
walltime = 00:30
memory = 4000
adjust = False
num_threads = 4

[export_ortho_geo]
walltime = 01:00
memory = 1000
adjust = False
num_threads = 4

[ingest_insarmaps]
walltime = 01:00
memory = 1000
adjust = False
num_threads = 4

[download_rsmas]
walltime = 03:00
memory = 5000
adjust = False
num_threads = 4

[minopy_wrapper]
walltime = 12:00
memory = 5000
adjust = False
num_threads = 4

[crop_sentinel]
walltime = 1:00
memory = 6000
adjust = False
num_threads = 4

[create_patch]
walltime = 1:00
memory = 4000
adjust = False
num_threads = 4

[phase_linking]
walltime = 12:00
memory = 1000
adjust = False
num_threads = 4

[patch_inversion]
walltime = 6:00
memory = 3000
adjust = False
num_threads = 4

[ifgramStack_to_ifgram_and_coherence]
walltime = 1:00
memory = 4000
adjust = False
num_threads = 4

[timeseries_corrections]
walltime = 4:00
memory = 8000
adjust = False
num_threads = 4

#### mooved to end ####
[merge]
walltime = 02:00
memory = 4000
adjust = False
num_threads = 4

[Ovec8hkin]

You just make the conversion to that format by hand. There’s no library to reformat config files for you.

[falkamelung]

The format conversion is easy. Are there canned functions that we could use to read table formats? Or should we just write ourselves?

[falkamelung]

I reformatted already and it is in master as new_job_defaults.cfg. It is so much easier to read ! The moment we are ready to rewrite the reading function we should switch

https://github.com/geodesymiami/rsmas_insar/blob/master/minsar/defaults/new_job_defaults.cfg

[mirzaees]

This is implemented. I am closing the issue