falkamelung
commented
5 years ago The memory assignment is currently done by a function in execute_stacksentinel_runfiles.py in teh lines below. In a similar way we would like to have the ability to set the walltimelimit. Most run files will have the same default walltimelimit but there are a few runfiles that need longer walltimelimit. In the same way we want the ability to set other job parameters as needed. (Looking forward: we may want the ability to set these parameters for different platforms and queues. So this default function should be ready to understand platform, queue variables.)
*Important: we need for all these parameters the ability to overwrite defaults by setting the option in the `.template file`.**
def set_memory_defaults():
""" Sets an optimized memory value for each job. """
memoryuse = {'unpack_slc_topo_master':'3700',
'average_baseline':'3700',
'extract_burst_overlaps':'3700',
'overlap_geo2rdr_resample':'4000',
'pairs_misreg':'3700',
'timeseries_misreg':'3700',
'geo2rdr_resample':'5000',
'extract_stack_valid_region':'3700',
'merge':'3700',
'merge_burst_igram': '3700',
'grid_baseline':'3700',
'generate_igram':'3700',
'filter_coherence':'6000',
'merge_master_slave_slc':'3700',
'unwrap':'3700'}
return memoryuse
lily-wittle
and it would be great to be able to use the parallel queue on pegasus. I am trying to get on XSEDE's STAMPEDE and I was told that the queue is similar to the pegasus parallel queue (but there are 64 workers compared to 16 on pegasus). The way to switch between queues would be for now by changing the environment variable
QUEUENAME=parallel.And it would be good to have a way to define the default job submission parameters for different schedulers and queues not as defaults in the parser but in a dedicated module or text file (similar functionality as
platforms.bash). We currently have 2-3 different ways to set default parameters. I like the first but I am flexible. We should decide about a default way to set defaults and then replace platforms.bash with this as well. ### NOTE March 1: I think we should follow the PYSAR style shown at the end in a defaults directory.or:
or:
import os import re import glob import numpy as np
Auto setting for file structure of Univ. of Miami, as shown below.
It required 3 conditions: 1) autoPath = True
2) $SCRATCHDIR is defined in environmental variable
3) input custom template with basename same as project_name
Change it to False if you are not using it.
autoPath = True
Default path of data files from different InSAR processors to be loaded into PySAR
isceAutoPath = '''##----------Default file path of ISCE-topsStack products pysar.load.processor = isce pysar.load.metaFile = ${PROJECT_DIR}/master/IW*.xml pysar.load.baselineDir = ${PROJECT_DIR}/baselines
pysar.load.unwFile = ${PROJECT_DIR}/merged/interferograms//filt.unw pysar.load.corFile = ${PROJECT_DIR}/merged/interferograms//filt.cor pysar.load.connCompFile = ${PROJECT_DIR}/merged/interferograms//filt.unw.conncomp pysar.load.ionoFile = None pysar.load.intFile = None