search

geodesymiami / rsmas_insar

RSMAS InSAR code
https://rsmas-insar.readthedocs.io/
GNU General Public License v3.0
59 stars 23 forks source link

Troubleshooting Sentinel: FileNotFoundError /stack/IW3.xml #382

Closed falkamelung closed 3 years ago

falkamelung commented 4 years ago

Occasionally I see this error. Produced in run_11 by extractCommonValidRegion.py. It turns out that there is a variability in the number of bursts for master (see below).

Did I select a too small area or did data not get properly unpacked?

//login3/scratch/05861/tg851601/BogdSenDT135[1009] cat out_run_11_extract_stack_valid_region.e
#########################
#### /scratch/05861/tg851601/BogdSenDT135/run_files/run_11_extract_stack_valid_region_6024429.e 
#########################
using /tmp/launcher.6024429.hostlist.i6pJW6eO to get hosts
starting job on c477-063
#########################
#### /scratch/05861/tg851601/BogdSenDT135/run_files/run_11_extract_stack_valid_region_1.e 
#########################
Traceback (most recent call last):
  File "/home1/05861/tg851601/test/operations/rsmas_insar/sources/isce2/contrib/stack/topsStack/extractCommonValidRegion.py", line 109, in <module>
    main()
  File "/home1/05861/tg851601/test/operations/rsmas_insar/sources/isce2/contrib/stack/topsStack/extractCommonValidRegion.py", line 95, in main
    topMaster = ut.loadProduct(os.path.join(inps.master , 'IW{0}.xml'.format(swath)))
  File "/home1/05861/tg851601/test/operations/rsmas_insar/sources/isce2/contrib/stack/topsStack/s1a_isce_utils.py", line 26, in loadProduct
    obj = pm.loadProduct(xmlname)
  File "/home1/05861/tg851601/test/operations/rsmas_insar/3rdparty/miniconda3/lib/python3.7/site-packages/isce/components/iscesys/Component/ProductManager.py", line 25, in loadProduct
    self.load(filename)
  File "/home1/05861/tg851601/test/operations/rsmas_insar/3rdparty/miniconda3/lib/python3.7/site-packages/isce/components/iscesys/Component/Configurable.py", line 1407, in load
    tmpProp, tmpFact, tmpMisc = FP.parse(filename)
  File "/home1/05861/tg851601/test/operations/rsmas_insar/3rdparty/miniconda3/lib/python3.7/site-packages/isce/components/iscesys/Parsers/XmlParser.py", line 41, in parse
    root = ET.parse(filename)
  File "/home1/05861/tg851601/test/operations/rsmas_insar/3rdparty/miniconda3/lib/python3.7/xml/etree/ElementTree.py", line 1197, in parse
    tree.parse(source, parser)
  File "/home1/05861/tg851601/test/operations/rsmas_insar/3rdparty/miniconda3/lib/python3.7/xml/etree/ElementTree.py", line 587, in parse
    source = open(source, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/05861/tg851601/BogdSenDT135/stack/IW3.xml'
extractCommonValidRegion.py -m /scratch/05861/tg851601/BogdSenDT135/master -s /scratch/05861/tg851601/BogdSenDT135/coreg_slaves

This is the Open Source version of ISCE.
Some of the workflows depend on a separate licensed package.
To obtain the licensed package, please make a request for ISCE
through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.
Alternatively, if you are a member, or can become a member of WinSAR
you may be able to obtain access to a version of the licensed sofware at
https://winsar.unavco.org/software/isce
/scratch/05861/tg851601/BogdSenDT135/stack  already exists.
Replacing master with existing stack.
updating the valid overlap region of:
/scratch/05861/tg851601/BogdSenDT135/stack
******************
swath:  1
minSlave,maxSlave 1 4
minMaster,maxMaster 1 4
minBurst, maxBurst:  1 4
minSlave,maxSlave 1 4
minMaster,maxMaster 1 4
...
...
minMaster,maxMaster 1 4
minBurst, maxBurst:  1 4
writing  /scratch/05861/tg851601/BogdSenDT135/stack/IW1.xml
******************
swath:  2
minSlave,maxSlave 3 18
minMaster,maxMaster 1 18
minBurst, maxBurst:  3 18
...
...
...
minBurst, maxBurst:  3 18
minSlave,maxSlave 3 18
minMaster,maxMaster 1 18
minBurst, maxBurst:  3 18
writing  /scratch/05861/tg851601/BogdSenDT135/stack/IW2.xml
******************
swath:  3
Traceback (most recent call last):
  File "/home1/05861/tg851601/test/operations/rsmas_insar/sources/isce2/contrib/stack/topsStack/extractCommonValidRegion.py", line 109, in <module>
    main()
  File "/home1/05861/tg851601/test/operations/rsmas_insar/sources/isce2/contrib/stack/topsStack/extractCommonValidRegion.py", line 95, in main
    topMaster = ut.loadProduct(os.path.join(inps.master , 'IW{0}.xml'.format(swath)))
  File "/home1/05861/tg851601/test/operations/rsmas_insar/sources/isce2/contrib/stack/topsStack/s1a_isce_utils.py", line 26, in loadProduct
    obj = pm.loadProduct(xmlname)
  File "/home1/05861/tg851601/test/operations/rsmas_insar/3rdparty/miniconda3/lib/python3.7/site-packages/isce/components/iscesys/Component/ProductManager.py", line 25, in loadProduct
    self.load(filename)
  File "/home1/05861/tg851601/test/operations/rsmas_insar/3rdparty/miniconda3/lib/python3.7/site-packages/isce/components/iscesys/Component/Configurable.py", line 1407, in load
    tmpProp, tmpFact, tmpMisc = FP.parse(filename)
  File "/home1/05861/tg851601/test/operations/rsmas_insar/3rdparty/miniconda3/lib/python3.7/site-packages/isce/components/iscesys/Parsers/XmlParser.py", line 41, in parse
    root = ET.parse(filename)
  File "/home1/05861/tg851601/test/operations/rsmas_insar/3rdparty/miniconda3/lib/python3.7/xml/etree/ElementTree.py", line 1197, in parse
    tree.parse(source, parser)
  File "/home1/05861/tg851601/test/operations/rsmas_insar/3rdparty/miniconda3/lib/python3.7/xml/etree/ElementTree.py", line 587, in parse
    source = open(source, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/05861/tg851601/BogdSenDT135/stack/IW3.xml'
ls master/*
master/IW1.xml  master/IW2.xml  master/IW3.xml

master/IW1:
burst_01.slc.vrt  burst_01.slc.xml  burst_02.slc.vrt  burst_02.slc.xml  burst_03.slc.vrt  burst_03.slc.xml  burst_04.slc.vrt  burst_04.slc.xml

master/IW2:
burst_01.slc.vrt  burst_03.slc.vrt  burst_05.slc.vrt  burst_07.slc.vrt  burst_09.slc.vrt  burst_11.slc.vrt  burst_13.slc.vrt  burst_15.slc.vrt  burst_17.slc.vrt
burst_01.slc.xml  burst_03.slc.xml  burst_05.slc.xml  burst_07.slc.xml  burst_09.slc.xml  burst_11.slc.xml  burst_13.slc.xml  burst_15.slc.xml  burst_17.slc.xml
burst_02.slc.vrt  burst_04.slc.vrt  burst_06.slc.vrt  burst_08.slc.vrt  burst_10.slc.vrt  burst_12.slc.vrt  burst_14.slc.vrt  burst_16.slc.vrt  burst_18.slc.vrt
burst_02.slc.xml  burst_04.slc.xml  burst_06.slc.xml  burst_08.slc.xml  burst_10.slc.xml  burst_12.slc.xml  burst_14.slc.xml  burst_16.slc.xml  burst_18.slc.xml

master/IW3:
burst_01.slc.vrt  burst_03.slc.vrt  burst_05.slc.vrt  burst_07.slc.vrt  burst_09.slc.vrt  burst_11.slc.vrt  burst_13.slc.vrt  burst_15.slc.vrt  burst_17.slc.vrt
burst_01.slc.xml  burst_03.slc.xml  burst_05.slc.xml  burst_07.slc.xml  burst_09.slc.xml  burst_11.slc.xml  burst_13.slc.xml  burst_15.slc.xml  burst_17.slc.xml
burst_02.slc.vrt  burst_04.slc.vrt  burst_06.slc.vrt  burst_08.slc.vrt  burst_10.slc.vrt  burst_12.slc.vrt  burst_14.slc.vrt  burst_16.slc.vrt  burst_18.slc.vrt
burst_02.slc.xml  burst_04.slc.xml  burst_06.slc.xml  burst_08.slc.xml  burst_10.slc.xml  burst_12.slc.xml  burst_14.slc.xml  burst_16.slc.xml  burst_18.slc.xml

master/overlap:
IW1  IW1_bottom.xml  IW1_top.xml  IW2  IW2_bottom.xml  IW2_top.xml  IW3  IW3_bottom.xml  IW3_top.xml
//login3/scratch/05861/tg851601/BogdSenDT135/slaves/20170425[1037] ls *
IW1.xml  IW2.xml  IW3.xml

IW1:
burst_01.slc.vrt  burst_01.slc.xml  burst_02.slc.vrt  burst_02.slc.xml  burst_03.slc.vrt  burst_03.slc.xml  burst_04.slc.vrt  burst_04.slc.xml

IW2:
burst_01.slc.vrt  burst_03.slc.vrt  burst_05.slc.vrt  burst_07.slc.vrt  burst_09.slc.vrt  burst_11.slc.vrt  burst_13.slc.vrt  burst_15.slc.vrt
burst_01.slc.xml  burst_03.slc.xml  burst_05.slc.xml  burst_07.slc.xml  burst_09.slc.xml  burst_11.slc.xml  burst_13.slc.xml  burst_15.slc.xml
burst_02.slc.vrt  burst_04.slc.vrt  burst_06.slc.vrt  burst_08.slc.vrt  burst_10.slc.vrt  burst_12.slc.vrt  burst_14.slc.vrt  burst_16.slc.vrt
burst_02.slc.xml  burst_04.slc.xml  burst_06.slc.xml  burst_08.slc.xml  burst_10.slc.xml  burst_12.slc.xml  burst_14.slc.xml  burst_16.slc.xml

IW3:
burst_01.slc.vrt  burst_03.slc.vrt  burst_05.slc.vrt  burst_07.slc.vrt  burst_09.slc.vrt  burst_11.slc.vrt  burst_13.slc.vrt  burst_15.slc.vrt
burst_01.slc.xml  burst_03.slc.xml  burst_05.slc.xml  burst_07.slc.xml  burst_09.slc.xml  burst_11.slc.xml  burst_13.slc.xml  burst_15.slc.xml
burst_02.slc.vrt  burst_04.slc.vrt  burst_06.slc.vrt  burst_08.slc.vrt  burst_10.slc.vrt  burst_12.slc.vrt  burst_14.slc.vrt  burst_16.slc.vrt
burst_02.slc.xml  burst_04.slc.xml  burst_06.slc.xml  burst_08.slc.xml  burst_10.slc.xml  burst_12.slc.xml  burst_14.slc.xml  burst_16.slc.xml
create_runfiles.py /work/05861/tg851601/stampede2/insarlab/infiles/famelung/TEMPLATES/BogdSenDT135.template
This is the Open Source version of ISCE.
Some of the workflows depend on a separate licensed package.
To obtain the licensed package, please make a request for ISCE
through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.
Alternatively, if you are a member, or can become a member of WinSAR
you may be able to obtain access to a version of the licensed sofware at
https://winsar.unavco.org/software/isce

*************** Template Options ****************
Custom Template File:  /work/05861/tg851601/stampede2/insarlab/infiles/famelung/TEMPLATES/BogdSenDT135.template
Project Name:  BogdSenDT135
Work Dir:  /scratch/05861/tg851601/BogdSenDT135
1 2 3-->'1 2 3'
43.7 46.3 94.0 105.0-->'43.7 46.3 94.0 105.0'
WARNING: "acquisition_mode" is not given --> default: tops   (available options: tops, stripmap)
auto-->auto
None-->None
1 2 3-->'1 2 3'
43.7 46.3 94.0 105.0-->'43.7 46.3 94.0 105.0'
template file exists: /scratch/05861/tg851601/BogdSenDT135/BogdSenDT135.template, no updates
20200710:143420 * create_runfiles.py /work/05861/tg851601/stampede2/insarlab/infiles/famelung/TEMPLATES/BogdSenDT135.template
20200710:143420 * stackSentinel.py --slc_directory /scratch/05861/tg851601/BogdSenDT135/SLC --orbit_directory /work/05861/tg851601/stampede2/insarlab/S1orbits --aux_directory /work/05861/tg851601/stampede2/insarlab/S1aux --working_directory /scratch/05861/tg851601/BogdSenDT135 --dem DEM/demLat_N43_N47_Lon_E093_E106.dem.wgs84 --num_connections 4 --num_overlap_connections 3 --swath_num "1 2 3" --bbox "43.7 46.3 94.0 105.0" --azimuth_looks 14 --range_looks 38 --filter_strength 0.2 --esd_coherence_threshold 0.85 --snr_misreg_threshold 10 --unw_method snaphu --polarization vv --coregistration NESD --workflow interferogram
This is the Open Source version of ISCE.
Some of the workflows depend on a separate licensed package.
To obtain the licensed package, please make a request for ISCE
through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.
Alternatively, if you are a member, or can become a member of WinSAR
you may be able to obtain access to a version of the licensed sofware at
https://winsar.unavco.org/software/isce
Number of SAFE files found: 192
Number of SAFE files to be used (cover BBOX): 192
*****************************************
Number of dates : 129
List of dates : 
['20150130', '20150530', '20150623', '20150810', '20150927', '20151208', '20160101', '20160125', '20160218', '20160313', '20160430', '20160524', '20160804', '20160828', '20160921', '20160927', '20161009', '20161015', '20161102', '20161108', '20161126', '20161220', '20170101', '20170113', '20170119', '20170125', '20170206', '20170212', '20170218', '20170224', '20170302', '20170308', '20170314', '20170320', '20170326', '20170401', '20170407', '20170413', '20170419', '20170425', '20170507', '20170513', '20170519', '20170525', '20170531', '20170606', '20170612', '20170618', '20170624', '20170630', '20170712', '20170718', '20170724', '20170730', '20170805', '20170811', '20170817', '20170823', '20170829', '20170910', '20170916', '20170928', '20171004', '20171010', '20171016', '20171022', '20171028', '20171103', '20171109', '20171115', '20171121', '20171127', '20171203', '20171209', '20171215', '20171221', '20171227', '20180102', '20180108', '20180114', '20180120', '20180126', '20180201', '20180207', '20180213', '20180219', '20180225', '20180309', '20180321', '20180402', '20180414', '20180426', '20180508', '20180520', '20180601', '20180613', '20180625', '20180707', '20180719', '20180731', '20180812', '20180824', '20180905', '20180917', '20180929', '20181011', '20181023', '20181116', '20181128', '20181210', '20181222', '20190103', '20190115', '20190127', '20190208', '20190220', '20190304', '20190316', '20190328', '20190409', '20190421', '20190503', '20190515', '20190527', '20190608', '20190702', '20190714', '20190726', '20200614']
date      south      north
20150130 42.663147 47.708752
20150530 42.380985 44.401375
20150623 42.049942 45.723671
20150810 42.054905 45.724674
20150927 42.05072 45.724258
20151208 42.333191 46.054699
20160101 42.835926 47.818859
20160125 42.333359 46.055355
20160218 42.333599 46.055241
20160313 42.333122 46.054676
20160430 42.334438 46.055832
....
....
20190702 45.2033 47.597679
20190714 45.203365 47.597393
20190726 45.203197 47.597519
20200614 45.203598 47.598007
*****************************************
The overlap region among all dates (based on the preview kml files):
 South   North   East  West 
45.534962 44.401375 103.398727 106.238602
*****************************************
WARNING:
           There might not be overlap between some dates
*****************************************
All dates (129)
['20150130', '20150530', '20150623', '20150810', '20150927', '20151208', '20160101', '20160125', '20160218', '20160313', '20160430', '20160524', '20160804', '20160828', '20160921', '20160927', '20161009', '20161015', '20161102', '20161108', '20161126', '20161220', '20170101', '20170113', '20170119', '20170125', '20170206', '20170212', '20170218', '20170224', '20170302', '20170308', '20170314', '20170320', '20170326', '20170401', '20170407', '20170413', '20170419', '20170425', '20170507', '20170513', '20170519', '20170525', '20170531', '20170606', '20170612', '20170618', '20170624', '20170630', '20170712', '20170718', '20170724', '20170730', '20170805', '20170811', '20170817', '20170823', '20170829', '20170910', '20170916', '20170928', '20171004', '20171010', '20171016', '20171022', '20171028', '20171103', '20171109', '20171115', '20171121', '20171127', '20171203', '20171209', '20171215', '20171221', '20171227', '20180102', '20180108', '20180114', '20180120', '20180126', '20180201', '20180207', '20180213', '20180219', '20180225', '20180309', '20180321', '20180402', '20180414', '20180426', '20180508', '20180520', '20180601', '20180613', '20180625', '20180707', '20180719', '20180731', '20180812', '20180824', '20180905', '20180917', '20180929', '20181011', '20181023', '20181116', '20181128', '20181210', '20181222', '20190103', '20190115', '20190127', '20190208', '20190220', '20190304', '20190316', '20190328', '20190409', '20190421', '20190503', '20190515', '20190527', '20190608', '20190702', '20190714', '20190726', '20200614']

dates covering the bbox (35)
['20150130', '20160101', '20160804', '20161015', '20161108', '20170101', '20170119', '20170125', '20170218', '20170320', '20170401', '20170413', '20170425', '20170507', '20170519', '20170531', '20170612', '20170624', '20170718', '20170730', '20170811', '20170823', '20170916', '20170928', '20171010', '20171022', '20171103', '20171115', '20171209', '20171221', '20180102', '20180114', '20180126', '20180207', '20180219']

The master date was not chosen. The first date is considered as master date.

All SLCs will be coregistered to : 20150130
slave dates :
['20160101', '20160804', '20161015', '20161108', '20170101', '20170119', '20170125', '20170218', '20170320', '20170401', '20170413', '20170425', '20170507', '20170519', '20170531', '20170612', '20170624', '20170718', '20170730', '20170811', '20170823', '20170916', '20170928', '20171010', '20171022', '20171103', '20171115', '20171209', '20171221', '20180102', '20180114', '20180126', '20180207', '20180219']

No existing stack was identified. A new stack will be generated.
*****************************************
Coregistration method:  NESD
Workflow:  interferogram
*****************************************
falkamelung commented 3 years ago

seems resolved