search

geodesymiami / rsmas_insar

RSMAS InSAR code
https://rsmas-insar.readthedocs.io/
GNU General Public License v3.0
59 stars 23 forks source link

copy reference to /tmp #453

Open falkamelung opened 3 years ago

falkamelung commented 3 years ago 
##########################
run_03_average_baseline   
--> limit to 500 simultaneous tasks
##########################
cat /scratch/05861/tg851601/unittestGalapagosSenDT128/configs/config_baseline_20160629
[Function-1]
computeBaseline : 
reference : /scratch/05861/tg851601/unittestGalapagosSenDT128/reference/
secondary : /scratch/05861/tg851601/unittestGalapagosSenDT128/secondarys/20160629
baseline_file : /scratch/05861/tg851601/unittestGalapagosSenDT128/baselines/20160605_20160629/20160605_20160629.txt

##########################
run_04_fullBurst_geo2rdr
--> copy '/reference' and '/geom_reference' to /tmp
##########################
cat /scratch/05861/tg851601/unittestGalapagosSenDT128/configs/config_fullBurst_geo2rdr_20160828

[Function-1]
geo2rdr :
secondary : /scratch/05861/tg851601/unittestGalapagosSenDT128/secondarys/20160828
reference : /scratch/05861/tg851601/unittestGalapagosSenDT128/reference
geom_referenceDir : /scratch/05861/tg851601/unittestGalapagosSenDT128/geom_reference
coregSLCdir : /scratch/05861/tg851601/unittestGalapagosSenDT128/coreg_secondarys/20160828
overlap : False
useGPU : False
azimuth_misreg : /scratch/05861/tg851601/unittestGalapagosSenDT128/misreg/azimuth/dates/20160828.txt
range_misreg : /scratch/05861/tg851601/unittestGalapagosSenDT128/misreg/range/dates/20160828.txt

##########################
run_05_fullBurst_resample
--> copy `/reference` to /tmp
#########################
cat /scratch/05861/tg851601/unittestGalapagosSenDT128/configs/config_fullBurst_resample_20160629
[Function-1]
resamp_withCarrier : 
secondary : /scratch/05861/tg851601/unittestGalapagosSenDT128/secondarys/20160629
reference : /scratch/05861/tg851601/unittestGalapagosSenDT128/reference
coregdir : /scratch/05861/tg851601/unittestGalapagosSenDT128/coreg_secondarys/20160629
overlap : False
azimuth_misreg : /scratch/05861/tg851601/unittestGalapagosSenDT128/misreg/azimuth/dates/20160629.txt
range_misreg : /scratch/05861/tg851601/unittestGalapagosSenDT128/misreg/range/dates/20160629.txt

##########################
run_06_extract_stack_valid_region
-->  modify to allow specification of `/stack`
##########################
cat /scratch/05861/tg851601/unittestGalapagosSenDT128/run_files/run_06_extract_stack_valid_region
extractCommonValidRegion.py -m /scratch/05861/tg851601/unittestGalapagosSenDT128/reference -s /scratch/05861/tg851601/unittestGalapagosSenDT128/coreg_secondarys

##########################
run_07_merge_reference_secondary_slc
OK
##########################

Using reference only for merging the reference itself:

SentinelWrapper.py -c /scratch/05861/tg851601/unittestGalapagosSenDT128/configs/config_merge_20160605
SentinelWrapper.py -c /scratch/05861/tg851601/unittestGalapagosSenDT128/configs/config_merge_lat
SentinelWrapper.py -c /scratch/05861/tg851601/unittestGalapagosSenDT128/configs/config_merge_lon
SentinelWrapper.py -c /scratch/05861/tg851601/unittestGalapagosSenDT128/configs/config_merge_los
SentinelWrapper.py -c /scratch/05861/tg851601/unittestGalapagosSenDT128/configs/config_merge_hgt
SentinelWrapper.py -c /scratch/05861/tg851601/unittestGalapagosSenDT128/configs/config_merge_shadowMask
SentinelWrapper.py -c /scratch/05861/tg851601/unittestGalapagosSenDT128/configs/config_merge_incLocal

all other commands don't use reference

##########################
run_08_generate_burst_igram
--> copy `/refereence` to `/tmp`
##########################
cat /scratch/05861/tg851601/unittestGalapagosSenDT128/configs/config_generate_igram_20160605_20160629

[Function-1]
generateIgram : 
reference : /scratch/05861/tg851601/unittestGalapagosSenDT128/reference
secondary : /scratch/05861/tg851601/unittestGalapagosSenDT128/coreg_secondarys/20160629
interferogram : /scratch/05861/tg851601/unittestGalapagosSenDT128/interferograms/20160605_20160629
flatten : False
interferogram_prefix : fine
overlap : False

##########################
run_09_merge_burst_igram
--> copy `/stack` to `/tmp`
##########################
cat /scratch/05861/tg851601/unittestGalapagosSenDT128/configs/config_merge_igram_20160605_20160629

[Function-1]
mergeBursts : 
stack : /scratch/05861/tg851601/unittestGalapagosSenDT128/stack
inp_reference : /scratch/05861/tg851601/unittestGalapagosSenDT128/interferograms/20160605_20160629
dirname : /scratch/05861/tg851601/unittestGalapagosSenDT128/interferograms/20160605_20160629
name_pattern : fine*int
outfile : /scratch/05861/tg851601/unittestGalapagosSenDT128/merged/interferograms/20160605_20160629/fine.int
method : top
aligned : True
valid_only : True
use_virtual_files : True
multilook : True
range_looks : 15
azimuth_looks : 5

--> uses /stack

##########################
run_10_filter_coherence
##########################
Does not use reference

##########################
run_11_unwrap
--> copy `/reference` to `/tmp`
##########################
cat /scratch/05861/tg851601/unittestGalapagosSenDT128/configs/config_igram_unw_20160605_20160629

[Function-1]
unwrap : 
ifg : /scratch/05861/tg851601/unittestGalapagosSenDT128/merged/interferograms/20160605_20160629/filt_fine.int
unw : /scratch/05861/tg851601/unittestGalapagosSenDT128/merged/interferograms/20160605_20160629/filt_fine.unw
coh : /scratch/05861/tg851601/unittestGalapagosSenDT128/merged/interferograms/20160605_20160629/filt_fine.cor
nomcf : False
reference : /scratch/05861/tg851601/unittestGalapagosSenDT128/reference
defomax : 2
rlks : 15
alks : 5
rmfilter : False
method : snaphu
falkamelung commented 3 years ago

This comment is not correct. This function creates the /stack which can be done on /scratch (21 Jan 21)

Difficulty to implement reading from local /tmp for frequently accessed file: extractCommonValidRegion.py currently does not use a /configs/* file:

cat run_06_extract_stack_valid_region
extractCommonValidRegion.py -m /scratch/05861/tg851601/unittestGalapagosSenDT128/reference -s /scratch/05861/tg851601/unittestGalapagosSenDT128/coreg_secondarys

The out directory /stack is hardwired into the code. There is also no option to specify the location of /stack

extractCommonValidRegion.py --help
This is the Open Source version of ISCE.
Some of the workflows depend on a separate licensed package.
To obtain the licensed package, please make a request for ISCE
through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.
Alternatively, if you are a member, or can become a member of WinSAR
you may be able to obtain access to a version of the licensed sofware at
https://winsar.unavco.org/software/isce
usage: extractCommonValidRegion.py [-h] -m REFERENCE -s SECONDARY

Extract valid overlap region for the stack

optional arguments:
  -h, --help            show this help message and exit
  -m REFERENCE, --reference REFERENCE
                        Directory with reference acquisition
  -s SECONDARY, --secondary SECONDARY
                        Directory with secondary acquisition

We need to modify so that

SentinelWrapper.py -c /scratch/05861/tg851601/unittestGalapagosSenDT128/configs/config_extract_common_valid_region

with /configs/config_extract_common_valid_region containing (among others).

reference : /scratch/05861/tg851601/unittestGalapagosSenDT128/reference
coregdirBase : /scratch/05861/tg851601/unittestGalapagosSenDT128/coreg_secondarys
stack : /scratch/05861/tg851601/unittestGalapagosSenDT128/stack

(Heresh, any name suggestions for coregdirBase ? I don't see it used without the date anywhere)

To what I see, the hardwiring is done here: https://github.com/isce-framework/isce2/blob/0d5b94efadbc1dcfcb7fea91e6152db7cfdb3e8d/contrib/stack/topsStack/extractCommonValidRegion.py#L105

Another problem with the script is that it fails if /stack (or files within /stack) exist. We should modify so that it overwrites.

A more complex issue is that it should run in parallel. Currently it uses only one core. That is not urgent.