search

geodesymiami / rsmas_insar

RSMAS InSAR code
https://rsmas-insar.readthedocs.io/
GNU General Public License v3.0
62 stars 23 forks source link

topsStack bug: number of bursts varies between latitude chunks #431

Closed falkamelung closed 3 years ago

falkamelung commented 3 years ago

The merge command below does not work for lat30-31, but it works for lat29-30 and for lat36-37 and lat37-38.

For lat31-32, lat32-33 lat34-35, lat35-36 there is no file produced. Have to why lat33-34 did not work

SentinelWrapper.py -c /scratch/05861/tg851601/IsraelBig30SenDT21/configs/config_merge_20160318
Completed Parsing the Configuration file
Functions to be executed:
['Function-1']
Running: mergeBursts
['--stack', '/scratch/05861/tg851601/IsraelBig30SenDT21/stack', '--inp_reference', '/scratch/05861/tg851601/IsraelBig30SenDT21/coreg_secondarys/20160318', '--dirname', '/scratch/05861/tg851601/IsraelBig30SenDT21/coreg_secondarys/20160318', '--name_pattern', 'burst*slc', '--outfile', '/scratch/05861/tg851601/IsraelBig30SenDT21/merged/SLC/20160318/20160318.slc', '--method', 'top', '--aligned', '--valid_only', '--use_virtual_files', '--range_looks', '38', '--azimuth_looks', '14']
This is the Open Source version of ISCE.
Some of the workflows depend on a separate licensed package.
To obtain the licensed package, please make a request for ISCE
through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.
Alternatively, if you are a member, or can become a member of WinSAR
you may be able to obtain access to a version of the licensed sofware at
https://winsar.unavco.org/software/isce
Traceback (most recent call last):
  File "/scratch/05861/tg851601/code2/rsmas_insar/sources/isce2/contrib/stack/topsStack/SentinelWrapper.py", line 251, in <module>
    main(args.start,args.end)
  File "/scratch/05861/tg851601/code2/rsmas_insar/sources/isce2/contrib/stack/topsStack/SentinelWrapper.py", line 242, in main
    cfgParser.runCmd()
  File "/scratch/05861/tg851601/code2/rsmas_insar/sources/isce2/contrib/stack/topsStack/SentinelWrapper.py", line 54, in runCmd
    func_modules.main(self.funcParams[section])
  File "/scratch/05861/tg851601/code2/rsmas_insar/sources/isce2/contrib/stack/topsStack/mergeBursts.py", line 369, in main
    inps.reference, ifg.numberOfBursts, reference.numberOfBursts))
ValueError: /scratch/05861/tg851601/IsraelBig30SenDT21/coreg_secondarys/20160318 has different number of bursts (7) than the reference (8)
 SentinelWrapper.py -c /scratch/05861/tg851601/IsraelBig29SenDT21/configs/config_merge_20160318                                                                                                            
Completed Parsing the Configuration file
Functions to be executed:
['Function-1']
Running: mergeBursts
['--stack', '/scratch/05861/tg851601/IsraelBig29SenDT21/stack', '--inp_reference', '/scratch/05861/tg851601/IsraelBig29SenDT21/coreg_secondarys/20160318', '--dirname', '/scratch/05861/tg851601/IsraelBig29SenDT21/coreg_secondarys/20160318', '--name_pattern', 'burst*slc', '--outfile', '/scratch/05861/tg851601/IsraelBig29SenDT21/merged/SLC/20160318/20160318.slc', '--method', 'top', '--aligned', '--valid_only', '--use_virtual_files', '--range_looks', '38', '--azimuth_looks', '14']
This is the Open Source version of ISCE.
Some of the workflows depend on a separate licensed package.
To obtain the licensed package, please make a request for ISCE
through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.
Alternatively, if you are a member, or can become a member of WinSAR
you may be able to obtain access to a version of the licensed sofware at
https://winsar.unavco.org/software/isce
Updating the valid region of each burst to the common valid region of the stack
bursts:  1 8
Updating the valid region of each burst to the common valid region of the stack
bursts:  1 8
Updating the valid region of each burst to the common valid region of the stack
bursts:  1 8
False
Skipping multi-looking ....

It turns out that in coreg_secondarys and secondarys there are a few files missing in the lat30-31 chunk (14 IW1 files): (burst_08.slc* is missing)

//c506-094/scratch/05861/tg851601/IsraelBig30SenDT21/coreg_secondarys/20160318[1010] countfiles
./IW1 : 63 file(s)
./IW3 : 72 file(s)
./IW2 : 72 file(s)
./overlap : 366 file(s)

//c506-094/scratch/05861/tg851601/IsraelBig30SenDT21/coreg_secondarys/20160318[1012] cd ../../secondarys/20160318
//c506-094/scratch/05861/tg851601/IsraelBig30SenDT21/secondarys/20160318[1013] countfiles
./IW1 : 14 file(s)
./IW3 : 16 file(s)
./IW2 : 16 file(s)

//c506-094/scratch/05861/tg851601/IsraelBig30SenDT21/secondarys/20160318[1014] ls IW1
burst_01.slc.vrt  burst_02.slc.vrt  burst_03.slc.vrt  burst_04.slc.vrt  burst_05.slc.vrt  burst_06.slc.vrt  burst_07.slc.vrt
burst_01.slc.xml  burst_02.slc.xml  burst_03.slc.xml  burst_04.slc.xml  burst_05.slc.xml  burst_06.slc.xml  burst_07.slc.xml

compared to 16 IW1 files in the lat29-30 chunk:

/c506-094/scratch/05861/tg851601/IsraelBig29SenDT21/coreg_secondarys/20160318[1014] countfiles
./IW1 : 72 file(s)
./IW3 : 72 file(s)
./IW2 : 72 file(s)
./overlap : 384 file(s)
//c506-094/scratch/05861/tg851601/IsraelBig29SenDT21/coreg_secondarys/20160318[1015] cd ../../secondarys/20160318
//c506-094/scratch/05861/tg851601/IsraelBig29SenDT21/secondarys/20160318[1016] !cou
countfiles
./IW1 : 16 file(s)
./IW3 : 16 file(s)
./IW2 : 16 file(s)

//c506-094/scratch/05861/tg851601/IsraelBig29SenDT21/secondarys/20160318[1017] ls IW1
burst_01.slc.vrt  burst_02.slc.vrt  burst_03.slc.vrt  burst_04.slc.vrt  burst_05.slc.vrt  burst_06.slc.vrt  burst_07.slc.vrt  burst_08.slc.vrt
burst_01.slc.xml  burst_02.slc.xml  burst_03.slc.xml  burst_04.slc.xml  burst_05.slc.xml  burst_06.slc.xml  burst_07.slc.xml  burst_08.slc.xml

Unpacking in the lat30-31 chunk gives only 7 bursts for IW1:

/login3/scratch/05861/tg851601/IsraelBig30SenDT21/run_files[1011] SentinelWrapper.py -c /scratch/05861/tg851601/IsraelBig30SenDT21/configs/config_secondary_20160318
Completed Parsing the Configuration file
Functions to be executed:
['Function-1']
Running: Sentinel1_TOPS
['--dirname', '/scratch/05861/tg851601/IsraelbigSenDT21/SLC/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.zip', '--swaths', '1 2 3', '--orbitdir', '/work/05861/tg851601/stampede2/insarlab/S1orbits', '--outdir', '/scratch/05861/tg851601/IsraelBig30SenDT21/secondarys/20160318', '--auxdir', '/work/05861/tg851601/stampede2/insarlab/S1aux', '--bbox', '30.0 31.0 32.0 39.5', '--pol', 'vv']
This is the Open Source version of ISCE.
Some of the workflows depend on a separate licensed package.
To obtain the licensed package, please make a request for ISCE
through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.
Alternatively, if you are a member, or can become a member of WinSAR
you may be able to obtain access to a version of the licensed sofware at
https://winsar.unavco.org/software/isce
Input XML files:  ['/vsizip//scratch/05861/tg851601/IsraelbigSenDT21/SLC/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.zip/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.SAFE/annotation/s1a-iw1-slc-vv-20160318t033551-20160318t033619-010418-00f71e-004.xml']
Input TIFF files:  ['/vsizip//scratch/05861/tg851601/IsraelbigSenDT21/SLC/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.zip/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.SAFE/measurement/s1a-iw1-slc-vv-20160318t033551-20160318t033619-010418-00f71e-004.tiff']
Manifest files:  ['/vsizip//scratch/05861/tg851601/IsraelbigSenDT21/SLC/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.zip/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.SAFE/manifest.safe']
MANS:  /scratch/05861/tg851601/IsraelbigSenDT21/SLC/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.zip S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.SAFE/manifest.safe
Setting IPF version to :  002.62
Extracting orbit from Orbit File:  /work/05861/tg851601/stampede2/insarlab/S1orbits/S1A_OPER_AUX_POEORB_OPOD_20160407T121449_V20160317T225943_20160319T005943.EOF
Number of Bursts before cropping:  10
Number of Bursts after cropping:  7
Output directory exists. Overwriting ...
Updating burst number from 1 to 1
Updating burst number from 2 to 2
Updating burst number from 3 to 3
Updating burst number from 4 to 4
Updating burst number from 5 to 5
Updating burst number from 6 to 6
Updating burst number from 7 to 7
Input XML files:  ['/vsizip//scratch/05861/tg851601/IsraelbigSenDT21/SLC/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.zip/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.SAFE/annotation/s1a-iw2-slc-vv-20160318t033552-20160318t033618-010418-00f71e-005.xml']

Whereas unpacking in the lat29-30 chunk gives 8 bursts for IW1:

Completed Parsing the Configuration file
Functions to be executed:
['Function-1']
Running: Sentinel1_TOPS
['--dirname', '/scratch/05861/tg851601/IsraelbigSenDT21/SLC/S1A_IW_SLC__1SDV_20160318T033617_20160318T033644_010418_00F71E_E89A.zip /scratch/05861/tg851601/IsraelbigSenDT21/SLC/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.zip', '--swaths', '1 2 3', '--orbitdir', '/work/05861/tg851601/stampede2/insarlab/S1orbits', '--outdir', '/scratch/05861/tg851601/IsraelBig29SenDT21/secondarys/20160318', '--auxdir', '/work/05861/tg851601/stampede2/insarlab/S1aux', '--bbox', '29.0 30.0 32.0 39.5', '--pol', 'vv']
This is the Open Source version of ISCE.
Some of the workflows depend on a separate licensed package.
To obtain the licensed package, please make a request for ISCE
through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.
Alternatively, if you are a member, or can become a member of WinSAR
you may be able to obtain access to a version of the licensed sofware at
https://winsar.unavco.org/software/isce
Input XML files:  ['/vsizip//scratch/05861/tg851601/IsraelbigSenDT21/SLC/S1A_IW_SLC__1SDV_20160318T033617_20160318T033644_010418_00F71E_E89A.zip/S1A_IW_SLC__1SDV_20160318T033617_20160318T033644_010418_00F71E_E89A.SAFE/annotation/s1a-iw1-slc-vv-20160318t033619-20160318t033644-010418-00f71e-004.xml', '/vsizip//scratch/05861/tg851601/IsraelbigSenDT21/SLC/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.zip/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.SAFE/annotation/s1a-iw1-slc-vv-20160318t033551-20160318t033619-010418-00f71e-004.xml']
Input TIFF files:  ['/vsizip//scratch/05861/tg851601/IsraelbigSenDT21/SLC/S1A_IW_SLC__1SDV_20160318T033617_20160318T033644_010418_00F71E_E89A.zip/S1A_IW_SLC__1SDV_20160318T033617_20160318T033644_010418_00F71E_E89A.SAFE/measurement/s1a-iw1-slc-vv-20160318t033619-20160318t033644-010418-00f71e-004.tiff', '/vsizip//scratch/05861/tg851601/IsraelbigSenDT21/SLC/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.zip/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.SAFE/measurement/s1a-iw1-slc-vv-20160318t033551-20160318t033619-010418-00f71e-004.tiff']
Manifest files:  ['/vsizip//scratch/05861/tg851601/IsraelbigSenDT21/SLC/S1A_IW_SLC__1SDV_20160318T033617_20160318T033644_010418_00F71E_E89A.zip/S1A_IW_SLC__1SDV_20160318T033617_20160318T033644_010418_00F71E_E89A.SAFE/manifest.safe', '/vsizip//scratch/05861/tg851601/IsraelbigSenDT21/SLC/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.zip/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.SAFE/manifest.safe']
Input XML files:  ['/vsizip//scratch/05861/tg851601/IsraelbigSenDT21/SLC/S1A_IW_SLC__1SDV_20160318T033617_20160318T033644_010418_00F71E_E89A.zip/S1A_IW_SLC__1SDV_20160318T033617_20160318T033644_010418_00F71E_E89A.SAFE/annotation/s1a-iw1-slc-vv-20160318t033619-20160318t033644-010418-00f71e-004.xml']
Input TIFF files:  ['/vsizip//scratch/05861/tg851601/IsraelbigSenDT21/SLC/S1A_IW_SLC__1SDV_20160318T033617_20160318T033644_010418_00F71E_E89A.zip/S1A_IW_SLC__1SDV_20160318T033617_20160318T033644_010418_00F71E_E89A.SAFE/measurement/s1a-iw1-slc-vv-20160318t033619-20160318t033644-010418-00f71e-004.tiff']
MANS:  /scratch/05861/tg851601/IsraelbigSenDT21/SLC/S1A_IW_SLC__1SDV_20160318T033617_20160318T033644_010418_00F71E_E89A.zip S1A_IW_SLC__1SDV_20160318T033617_20160318T033644_010418_00F71E_E89A.SAFE/manifest.safe
Setting IPF version to :  002.62
Extracting orbit from Orbit File:  /work/05861/tg851601/stampede2/insarlab/S1orbits/S1A_OPER_AUX_POEORB_OPOD_20160407T121449_V20160317T225943_20160319T005943.EOF
Number of Bursts before cropping:  9
Number of Bursts after cropping:  3
Input XML files:  ['/vsizip//scratch/05861/tg851601/IsraelbigSenDT21/SLC/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.zip/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.SAFE/annotation/s1a-iw1-slc-vv-20160318t033551-20160318t033619-010418-00f71e-004.xml']
Input TIFF files:  ['/vsizip//scratch/05861/tg851601/IsraelbigSenDT21/SLC/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.zip/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.SAFE/measurement/s1a-iw1-slc-vv-20160318t033551-20160318t033619-010418-00f71e-004.tiff']
MANS:  /scratch/05861/tg851601/IsraelbigSenDT21/SLC/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.zip S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.SAFE/manifest.safe
Setting IPF version to :  002.62
Extracting orbit from Orbit File:  /work/05861/tg851601/stampede2/insarlab/S1orbits/S1A_OPER_AUX_POEORB_OPOD_20160407T121449_V20160317T225943_20160319T005943.EOF
Number of Bursts before cropping:  10
Number of Bursts after cropping:  5
Stitching 2 slices together
COMBINED VERSION:  002.62
Output directory exists. Overwriting ...
Updating burst number from 6 to 1
Updating burst number from 7 to 2
Updating burst number from 8 to 3
Updating burst number from 9 to 4
Updating burst number from 10 to 5
Updating burst number from 1 to 6
Updating burst number from 2 to 7
Updating burst number from 3 to 8
Input XML files:  ['/vsizip//scratch/05861/tg851601/IsraelbigSenDT21/SLC/S1A_IW_SLC__1SDV_20160318T033617_20160318T033644_010418_00F71E_E89A.zip/S1A_IW_SLC__1SDV_20160318T033617_20160318T033644_010418_00F71E_E89A.SAFE/annotation/s1a-iw2-slc-vv-20160318t033617-20160318t033642-010418-00f71e-005.xml', '/vsizip//scratch/05861/tg851601/IsraelbigSenDT21/SLC/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.zip/S1A_IW_SLC__1SDV_20160318T033551_20160318T033619_010418_00F71E_81BB.SAFE/annotation/s1a-iw2-slc-vv-20160318t033552-20160318t033618-010418-00f71e-005.xml']

Here all the dirs where this orbit exists:

//login3/scratch/05861/tg851601[1024] ls -d1 IsraelBig{2,3}*/secondarys/20160318
IsraelBig29SenDT21/secondarys/20160318
IsraelBig30SenDT21/secondarys/20160318
IsraelBig33SenDT21/secondarys/20160318
IsraelBig36SenDT21/secondarys/20160318
IsraelBig37SenDT21/secondarys/20160318
IsraelBig38SenDT21/secondarys/20160318
IsraelBig39SenDT21/secondarys/20160318
///login3/scratch/05861/tg851601/IsraelBig29SenDT21/secondarys/20160318[1040] !cou
countfiles
./IW1 : 16 file(s)
./IW3 : 16 file(s)
./IW2 : 16 file(s)

//login3/scratch/05861/tg851601/IsraelBig30SenDT21/secondarys/20160318[1042] countfiles
./IW1 : 14 file(s)
./IW3 : 16 file(s)
./IW2 : 16 file(s)

//login3/scratch/05861/tg851601/IsraelBig36SenDT21/secondarys/20160318[1045] countfiles
./IW1 : 16 file(s)
./IW3 : 16 file(s)
./IW2 : 18 file(s)

//login3/scratch/05861/tg851601/IsraelBig37SenDT21/secondarys/20160318[1047] !cou
countfiles
./IW1 : 16 file(s)
./IW3 : 16 file(s)
./IW2 : 16 file(s)

//login3/scratch/05861/tg851601/IsraelBig38SenDT21/secondarys/20160318[1049] countfiles
./IW1 : 16 file(s)
./IW3 : 16 file(s)
./IW2 : 16 file(s)

//login3/scratch/05861/tg851601/IsraelBig39SenDT21/secondarys/20160318[1051] countfiles
./IW1 : 16 file(s)
./IW3 : 16 file(s)
./IW2 : 16 file(s)