search

insarlab / MiaplPy

MIAmi Phase Linking in PYthon
Other
92 stars 23 forks source link

out of data range error although subset is within SLC area #30

Open falkamelung opened 2 years ago

falkamelung commented 2 years ago

I am getting an out of data range error, although I have selected a subset that is fully within the SLCs. It turns out that the LENGTH and WIDTH in the *.rsc files are too small. Some multilooking seems to have been applied and this is what causes the problem? I darkly remember that we had this before, but I don't remember details. Any idea?

miaplpyApp.py /work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/samples/unittestGalapagosSenDT128.template --dir miaplpy --start load_data
--------------------------------------------------

_________________________________________________      

  /##      /## /##                     / ## /#######
 | ###    /###|__/|                    | ##| ##__  ##
 | ####  /#### /##|/ ##### /##         | ##| ##  \ ## /##   /##
 | ## ##/## ##| ##| ##__  ### /####### | ##| #######/| ##  | ##
 | ##  ###| ##| ##| ##__  ###| ##__  ##| ##| ##      | ##  | ##
 | ##\  # | ##| ##| ##   \###| ##   \##| ##| ##      | ##  | ##
 | ## \/  | ##| ##|  ##### ##| #######/| ##| ##      |  #######
 |__/     |__/|__/| \____/|_/| ##____/ |__/|__/       \____  ##
                             | ##                     /##  | ## 
                             | ##                    |  ######/         
                             |__/                     \______/
 Miami Non-Linear Phase Linking software in Python   
          MiaplPy 0.2.0, 2021-09-14
_________________________________________________

--RUN-at-2022-10-14 21:04:15.426400--
Current directory: /scratch/05861/tg851601/unittestGalapagosSenDT128
Run routine processing with arg_parser.py on steps: ['load_data', 'phase_linking', 'concatenate_patches', 'generate_ifgram', 'unwrap_ifgram', 'load_ifgram', 'ifgram_correction', 'invert_network', 'timeseries_correction']
Remaining steps: ['phase_linking', 'concatenate_patches', 'generate_ifgram', 'unwrap_ifgram', 'load_ifgram', 'ifgram_correction', 'invert_network', 'timeseries_correction']
Project name: unittestGalapagosSenDT128
--------------------------------------------------
20221014:210415 * miaplpyApp.py /work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/samples/unittestGalapagosSenDT128.template --dir miaplpy --start load_data
Project name: unittestGalapagosSenDT128
Go to work directory: /scratch/05861/tg851601/unittestGalapagosSenDT128/miaplpy
copy default template file /work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MintPy/mintpy/defaults/smallbaselineApp.cfg to work directory
read custom template file: /work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/samples/unittestGalapagosSenDT128.template
update default template based on input custom template
    mintpy.compute.cluster: auto --> local
    mintpy.compute.numWorker: auto --> 6
    mintpy.load.processor: auto --> isce
    mintpy.load.autoPath: auto --> yes
    mintpy.subset.lalo: auto --> -0.86:-0.81,-91.19:-91.13
    mintpy.reference.lalo: auto --> -0.82,-91.14
    mintpy.troposphericDelay.method: auto --> no
    mintpy.save.kmz: auto --> yes
    mintpy.save.hdfEos5: auto --> yes
    mintpy.save.hdfEos5.update: auto --> yes
copy unittestGalapagosSenDT128.template to inputs   directory for backup.
copy smallbaselineApp.cfg               to inputs   directory for backup.
copy unittestGalapagosSenDT128.template to pic      directory for backup.
copy smallbaselineApp.cfg               to pic      directory for backup.
read default template file: /scratch/05861/tg851601/unittestGalapagosSenDT128/miaplpy/smallbaselineApp.cfg
copy default template file /work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MiaplPy/miaplpy/objects/../defaults/miaplpyApp.cfg to work directory
update default template based on input custom template
    miaplpy.load.processor: auto --> isce
    miaplpy.load.autoPath: auto --> yes
No new option value found, skip updating /scratch/05861/tg851601/unittestGalapagosSenDT128/miaplpy/custom_smallbaselineApp.cfg
copy miaplpyApp.cfg to inputs   directory for backup.
read default template file: /scratch/05861/tg851601/unittestGalapagosSenDT128/miaplpy/miaplpyApp.cfg
    miaplpy.load.compression: auto --> no
    miaplpy.load.autoPath: yes --> yes
    miaplpy.subset.lalo: auto --> -0.86:-0.81,-91.19:-91.13
SAR platform/sensor : Sen
processor: isce
check auto path setting for Univ of Miami users for processor: isce
--------------------------------------------------
prepare metadata files for isce products
prep_slc_isce.py -s /scratch/05861/tg851601/unittestGalapagosSenDT128/merged/SLC -f *.slc.full -m /scratch/05861/tg851601/unittestGalapagosSenDT128/reference/IW1.xml -b /scratch/05861/tg851601/unittestGalapagosSenDT128/baselines -g /scratch/05861/tg851601/unittestGalapagosSenDT128/merged/geom_reference
['/scratch/05861/tg851601/unittestGalapagosSenDT128/reference/data.rsc'] exists and is newer than ['/scratch/05861/tg851601/unittestGalapagosSenDT128/reference/IW1.xml'] --> skip.
prepare .rsc file for geometry files
read perp baseline time-series from /scratch/05861/tg851601/unittestGalapagosSenDT128/baselines
preparing RSC file for  *.slc.full
[==================================================] 20160605_20160828    0s /     0s
Done.
Traceback (most recent call last):
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MiaplPy/miaplpy/miaplpyApp.py", line 801, in <module>
    main()
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MiaplPy/miaplpy/miaplpyApp.py", line 82, in main
    app.open()
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MiaplPy/miaplpy/miaplpyApp.py", line 176, in open
    self.date_list, self.num_pixels, self.metadata = read_initial_info(self.workDir, self.templateFile)
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MiaplPy/miaplpy/objects/utils.py", line 1223, in read_initial_info
    metadata = read_subset_box(iDict)
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MiaplPy/miaplpy/objects/utils.py", line 1156, in read_subset_box
    pix_box = coord.check_box_within_data_coverage(pix_box)
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MintPy/mintpy/objects/coord.py", line 477, in check_box_within_data_coverage
    raise ValueError(msg)
ValueError: ERROR: input index is out of data range!
data   range in (x0,y0,x1,y1): (0, 0, 1364, 565)
subset range in (x0,y0,x1,y1): (7009, 1486, 8912, 2010)
data   range in (W, N, E, S): None
subset range in (W, N, E, S): None
unittestGalapagosSenDT128/merged/SLC/20160605[1121] cat *.full.rsc | grep -E 'LENGTH|length'
FILE_LENGTH               565
LENGTH                    565
length                    2826

For comparison, here for a different dataset which works fine:

MiamiSenAT48/merged/SLC/20150921[1029] cat *.full.rsc | grep -E 'LENGTH|length'
FILE_LENGTH               5531
LENGTH                    5531
length                    5531

Here the *template file for the problem case:

cat /work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/samples/unittestGalapagosSenDT128.template 
######################################################
cleanopt                          = 0   # [ 0 / 1 / 2 / 3 / 4]   0,1: none 2: keep merged,geom_master,SLC 3: keep MINTPY 4: everything
processor                         = isce
ssaraopt.platform                 = SENTINEL-1A,SENTINEL-1B
ssaraopt.relativeOrbit            = 128
ssaraopt.startDate                = 20160601
ssaraopt.endDate                  = 20160831
insarmaps_flag                    = False
image_products_flag               = False

######################################################

topsStack.boundingBox             = -1 -0.6 -91.9 -90.7      # -1 0.15 -91.9 -90.6
topsStack.boundingBox             = -0.81 -0.80 -90.9 -90.86    # -1 0.15 -91.9 -90.6
topsStack.subswath                = 1    # '1 2'
topsStack.numConnections          = 3    # comment
topsStack.azimuthLooks            = 5    # comment
topsStack.rangeLooks              = 15   # comment
topsStack.filtStrength            = 0.2  # comment
topsStack.unwMethod               = snaphu  # comment
topsStack.coregistration          = auto  # [NESD geometry], auto for NESD
topsStack.slcDir                  = $TESTDATA_ISCE/unittestGalapagosSenDT128/SLC

######################################################

mintpy.load.autoPath              = yes
mintpy.compute.cluster            = local #[local / slurm / pbs / lsf / none], auto for none, cluster type
mintpy.compute.numWorker          = 6 #[int > 1 / all], auto for 4 (local) or 40 (non-local), num of workers
mintpy.reference.lalo             = -0.82,-91.14     # S of SN
mintpy.networkInversion.parallel  = yes  #[yes / no], auto for no, parallel processing using dask
mintpy.troposphericDelay.method   = no    # pyaps  #[pyaps / height_correlation / base_trop_cor / no], auto for pyaps
mintpy.save.hdfEos5               = yes   #[yes / update / no], auto for no, save timeseries to UNAVCO InSAR Archive format
mintpy.save.hdfEos5.update        = yes   #[yes / no], auto for no, put XXXXXXXX as endDate in output filename
mintpy.save.hdfEos5.subset        = yes   #[yes / no], auto for no, put XXXXXXXX as endDate in output filename
mintpy.save.kmz                   = yes  #[yes / no], auto for yes, save geocoded velocity to Google Earth KMZ file

######################################################

miaplpy.load.processor      = isce  
miaplpy.load.autoPath       = yes
miaplpy.compute.num_workers = 46
miaplpy.subset.lalo        = -0.86:-0.81,-91.19:-91.13    #[31.5:32.5,130.5:131.0 / no], auto for no
miaplpy.interferograms.networkType = sequential     # [single_reference, sequential, combine, list] default: single_reference
mirzaees commented 2 years ago

@falkamelung I don't understand why you are comparing data length for unittestGalapagosSenDT128 and MiamiSenAT48. They are definitely different

I run unittestGalapagosSenDT128 data without problem and did not get this error. You might not be using the last version of MiaplPy or your DEM is not downloaded properly:

unittestGalapagosSenDT128/merged/SLC/20160605[1070] cat *.full.rsc | grep -E 'LENGTH|length'
FILE_LENGTH              2826
LENGTH                   2826
length                   2826

Also these 2 options do not have overlap:

topsStack.boundingBox             = -0.81 -0.80 -90.9 -90.86 
miaplpy.subset.lalo        = -0.86:-0.81,-91.19:-91.13

however it is not a big problem for this dataset because isce uses a larger subset than the bbox in template and still covers the subset defined for miaplpy

falkamelung commented 2 years ago

Hi @mirzaees Thank you for running this. I am glad this it is still working for you! Are you using the committed unittestGalapagosSenDT128.template? If not, can you please post yours?

I ran with a new DEM and I am still getting the same error and have multi looked LENGTH.

/unittestGalapagosSenDT128/merged/SLC/20160605[1097] 
grep -E 'LENGTH|length' *rsc
FILE_LENGTH               565
LENGTH                    565
length                    2826

I did check for latest versions. So maybe my installation got screwed and I should do a new one.

mirzaees commented 2 years ago

@falkamelung , Yes I used the same template that is committed

falkamelung commented 2 years ago

Hi @mirzaees I found the problem. The issue occurs if you run MintPy prior to running MiaplPy.

The Mintpy prep_isce module creates an /reference/data.rsc. If that exists then MiaplPy has a problem. I did not look at the code but it looks that the MiaplPy prep_isce module does not create it if it exists, or similar, which looks like an easy fix.

The offending attribute seems to be WIDTH which is created by the Mintpy module but not by the MiaplPy module.

unittestGalapagosSenDT128

grep WIDTH reference/data.rsc mintpy_data.rsc 
mintpy_data.rsc:WIDTH                     1364

If both MiaplPy and Mintpy always overwrite all *rsc files there should not be an issue.