insarlab / MintPy

Miami InSAR time-series software in Python
https://mintpy.readthedocs.io
Other
584 stars 253 forks source link

timeseries2velocity list index out of range #493

Closed hardreddata closed 3 years ago

hardreddata commented 3 years ago

Thanks for making this great tool available open source.

Description of the problem

I have prepared my first ISCE Sentinel stack using topsStack.py and I am trying to run MintPy for the first time also. The smallbaselineApp fails during the velocity calculation.

Note that I only have three dates in the stack. I am assuming this is sufficient to test the workflow end to end.

I based the configuration on the example https://mintpy.readthedocs.io/en/stable/dir_structure/

Full script that generated the error

The full output of smallbaselineApp.py is below, including the error

-RUN-at-2020-12-27 18:26:45.259118--
Current directory: /media/storage/isce_stack
Run routine processing with smallbaselineApp.py on steps: ['load_data', 'modify_network', 'reference_point', 'quick_overview', 'correct_unwrap_error', 'invert_network', 'correct_LOD', 'correct_troposphere', 'deramp', 'correct_topography', 'residual_RMS', 'reference_date', 'velocity', 'geocode', 'google_earth', 'hdfeos5']
Remaining steps: ['modify_network', 'reference_point', 'quick_overview', 'correct_unwrap_error', 'invert_network', 'correct_LOD', 'correct_troposphere', 'deramp', 'correct_topography', 'residual_RMS', 'reference_date', 'velocity', 'geocode', 'google_earth', 'hdfeos5']
--------------------------------------------------
Go to work directory: /media/storage/isce_stack
read default template file: /media/storage/isce_stack/smallbaselineApp.cfg

******************** step - load_data ********************

load_data.py --template /media/storage/isce_stack/smallbaselineApp.cfg
processor : isce
SAR platform/sensor : unknown from project name "None"
--------------------------------------------------
prepare metadata files for isce products
prep_isce.py -m /media/storage/isce_stack/reference/IW1.xml -g /media/storage/isce_stack/merged/geom_reference -b /media/storage/isce_stack/baselines -d /media/storage/isce_stack/merged/interferograms -f filt_*.unw
['/media/storage/isce_stack/reference/data.rsc'] exists and is newer than ['/media/storage/isce_stack/reference/IW1.xml'] --> skip.
prepare .rsc file for geometry files
read perp baseline time-series from /media/storage/isce_stack/baselines
prepare .rsc file for  filt_*.unw
[==================================================] 20201209_20201221    0s /     0s
Done.
--------------------------------------------------
searching interferometric pairs info
input data files:
unwrapPhase     : /media/storage/isce_stack/merged/interferograms/*/filt_*.unw
coherence       : /media/storage/isce_stack/merged/interferograms/*/filt_*.cor
connectComponent: /media/storage/isce_stack/merged/interferograms/*/filt_*.unw.conncomp
number of unwrapPhase     : 2
number of coherence       : 2
number of connectComponent: 2
--------------------------------------------------
searching geometry files info
input data files:
height          : /media/storage/isce_stack/merged/geom_reference/hgt.rdr
latitude        : /media/storage/isce_stack/merged/geom_reference/lat.rdr
longitude       : /media/storage/isce_stack/merged/geom_reference/lon.rdr
incidenceAngle  : /media/storage/isce_stack/merged/geom_reference/los.rdr
azimuthAngle    : /media/storage/isce_stack/merged/geom_reference/los.rdr
shadowMask      : /media/storage/isce_stack/merged/geom_reference/shadowMask.rdr
--------------------------------------------------
updateMode : True
compression: None
x/ystep: 1/1
All date12   exists in file ifgramStack.h5 with same size as required, no need to re-load.
All datasets exists in file geometryRadar.h5 with same size as required, no need to re-load.
time used: 00 mins 0.9 secs.

Loaded dataset are processed by InSAR software: isce
Loaded dataset is in RADAR coordinates
Interferograms Stack: /media/storage/isce_stack/inputs/ifgramStack.h5
Geometry File       : /media/storage/isce_stack/inputs/geometryRadar.h5
Lookup Table File   : /media/storage/isce_stack/inputs/geometryRadar.h5
--------------------------------------------------
All data needed found/loaded/copied. Processed 2-pass InSAR data can be removed.
--------------------------------------------------

******************** step - modify_network ********************

modify_network.py /media/storage/isce_stack/inputs/ifgramStack.h5 -t /media/storage/isce_stack/smallbaselineApp.cfg
read options from template file: smallbaselineApp.cfg
No input option found to remove interferogram
Keep all interferograms by enable --reset option
--------------------------------------------------
reset dataset 'dropIfgram' to True for all interferograms for file: /media/storage/isce_stack/inputs/ifgramStack.h5
All dropIfgram are already True, no need to reset.

plot_network.py /media/storage/isce_stack/inputs/ifgramStack.h5 -t /media/storage/isce_stack/smallbaselineApp.cfg --nodisplay -d coherence -v 0.2 1.0
['network.pdf'] exists and is newer than ['/media/storage/isce_stack/inputs/ifgramStack.h5', 'coherenceSpatialAvg.txt', '/media/storage/isce_stack/smallbaselineApp.cfg'] --> skip.

******************** step - reference_point ********************

generate_mask.py /media/storage/isce_stack/inputs/ifgramStack.h5 --nonzero -o maskConnComp.h5 --update
input ifgramStack file: /media/storage/isce_stack/inputs/ifgramStack.h5
--------------------------------------------------
update mode: ON
1) output file maskConnComp.h5 already exists.
2) output file is newer than input dataset: connectComponent.
run or skip: skip.

temporal_average.py /media/storage/isce_stack/inputs/ifgramStack.h5 --dataset coherence -o avgSpatialCoh.h5 --update
output file: avgSpatialCoh.h5
--------------------------------------------------
update mode: ON
1) output file avgSpatialCoh.h5 already exists.
2) output file is newer than input dataset: coherence.
run or skip: skip.

reference_point.py /media/storage/isce_stack/inputs/ifgramStack.h5 -t /media/storage/isce_stack/smallbaselineApp.cfg -c avgSpatialCoh.h5
--------------------------------------------------
reading reference info from template: /media/storage/isce_stack/smallbaselineApp.cfg
no input reference y/x.
REF_Y/X exists in input file, skip updating.
REF_Y: 967
REF_X: 3026
reference point selection method: None
--------------------------------------------------
Done.

******************** step - quick_overview ********************

temporal_average.py /media/storage/isce_stack/inputs/ifgramStack.h5 --dataset unwrapPhase -o avgPhaseVelocity.h5 --update
output file: avgPhaseVelocity.h5
--------------------------------------------------
update mode: ON
1) output file avgPhaseVelocity.h5 already exists.
2) output file is newer than input dataset: unwrapPhase.
run or skip: skip.

unwrap_error_phase_closure.py /media/storage/isce_stack/inputs/ifgramStack.h5 --water-mask waterMask.h5 --action calculate --update
open ifgramStack file: ifgramStack.h5

WARNING: No triangles found from input date12_list:
['20201127_20201209', '20201209_20201221']!

No triangles found from ifgramStack file: /media/storage/isce_stack/inputs/ifgramStack.h5!
    Skip calculating the number of triplets with non-zero integer ambiguity.
time used: 00 mins 0.0 secs
Done.

******************** step - correct_unwrap_error ********************
phase-unwrapping error correction is OFF.

******************** step - invert_network ********************

ifgram_inversion.py /media/storage/isce_stack/inputs/ifgramStack.h5 -t /media/storage/isce_stack/smallbaselineApp.cfg --update
use dataset "unwrapPhase" by default
--------------------------------------------------
update mode: ON
1) output files already exist: ['timeseries.h5', 'temporalCoherence.h5', 'numInvIfgram.h5'].
2) output dataset is newer than input dataset: unwrapPhase.
3) all key configuration parameters are the same: ['obsDatasetName', 'numIfgram', 'weightFunc', 'maskDataset', 'maskThreshold', 'minRedundancy', 'minNormVelocity'].
run or skip: skip.

generate_mask.py temporalCoherence.h5 -m 0.7 -o maskTempCoh.h5 --base /media/storage/isce_stack/inputs/geometryRadar.h5 --base-dataset shadowMask --base-value 1
update mode: ON
1) output file: maskTempCoh.h5 already exists and newer than input file: temporalCoherence.h5
2) all key configuration parameters are the same: ['mintpy.networkInversion.minTempCoh', 'mintpy.networkInversion.shadowMask']
run or skip: skip
number of reliable pixels: 6722627

******************** step - correct_LOD ********************
No local oscillator drift correction is needed for sen.

******************** step - correct_troposphere ********************
tropospheric delay correction with height-correlation approach

tropo_phase_elevation.py timeseries.h5 -g /media/storage/isce_stack/inputs/geometryRadar.h5 -p 1 -m maskTempCoh.h5 -o timeseries_tropHgt.h5 -l 8 -t 0
['timeseries_tropHgt.h5'] exists and is newer than ['timeseries.h5'] --> skip.

******************** step - deramp ********************
No phase ramp removal.

******************** step - correct_topography ********************

dem_error.py timeseries_tropHgt.h5 -t /media/storage/isce_stack/smallbaselineApp.cfg -o timeseries_tropHgt_demErr.h5 --update -g /media/storage/isce_stack/inputs/geometryRadar.h5
read options from template file: smallbaselineApp.cfg
--------------------------------------------------
update mode: ON
1) output file timeseries_tropHgt_demErr.h5 already exists.
2) output file is newer than input file: ['timeseries_tropHgt.h5', '/media/storage/isce_stack/inputs/geometryRadar.h5'].
3) all key configuration parameters are the same:['polyOrder', 'phaseVelocity', 'stepFuncDate', 'excludeDate']
run or skip: skip.

******************** step - residual_RMS ********************

timeseries_rms.py timeseriesResidual.h5 -t /media/storage/isce_stack/smallbaselineApp.cfg
read options from template file: smallbaselineApp.cfg
['/media/storage/isce_stack/rms_timeseriesResidual_ramp.txt'] exists and is newer than ['timeseriesResidual_ramp.h5', 'maskTempCoh.h5'] --> skip.
read timeseries residual RMS from file: /media/storage/isce_stack/rms_timeseriesResidual_ramp.txt
--------------------------------------------------
date with min RMS: 20201209 - 0.0000
['reference_date.txt'] exists and is newer than ['timeseriesResidual.h5', 'maskTempCoh.h5', '/media/storage/isce_stack/smallbaselineApp.cfg'] --> skip.
--------------------------------------------------
date(s) with RMS > 3.0 * median RMS (0.0000)
None.
create figure in size: [5.0, 3.0]
save figure to file: /media/storage/isce_stack/rms_timeseriesResidual_ramp.pdf

******************** step - reference_date ********************

reference_date.py -t /media/storage/isce_stack/smallbaselineApp.cfg timeseries.h5 timeseries_tropHgt.h5 timeseries_tropHgt_demErr.h5
input reference date: 20201127
--------------------------------------------------
change reference date for file: /media/storage/isce_stack/timeseries.h5
same reference date chosen as existing reference date.
Nothing to be done.
--------------------------------------------------
change reference date for file: /media/storage/isce_stack/timeseries_tropHgt.h5
same reference date chosen as existing reference date.
Nothing to be done.
--------------------------------------------------
change reference date for file: /media/storage/isce_stack/timeseries_tropHgt_demErr.h5
same reference date chosen as existing reference date.
Nothing to be done.
time used: 00 mins 3.0 secs.

******************** step - velocity ********************

timeseries2velocity.py timeseries_tropHgt_demErr.h5 -t /media/storage/isce_stack/smallbaselineApp.cfg -o velocity.h5 --update
read options from template file: smallbaselineApp.cfg
open timeseries file: timeseries_tropHgt_demErr.h5
--------------------------------------------------
dates from input file: 3
['20201127', '20201209', '20201221']
--------------------------------------------------
using all dates to calculate the velocity
--------------------------------------------------
update mode: ON
1) output file velocity.h5 already exists.
2) output file is newer than input file: timeseries_tropHgt_demErr.h5.
3) all key configuration parameters are the same: ['startDate', 'endDate', 'excludeDate', 'bootstrap', 'bootstrapCount'].
run or skip: skip.

timeseries2velocity.py ./inputs/ERA5.h5 -t /media/storage/isce_stack/smallbaselineApp.cfg -o velocityERA5.h5 --update
read options from template file: smallbaselineApp.cfg
open timeseries file: ERA5.h5
Traceback (most recent call last):
  File "/home/russell/tools/MintPy/mintpy/smallbaselineApp.py", line 1133, in <module>
    main(sys.argv[1:])
  File "/home/russell/tools/MintPy/mintpy/smallbaselineApp.py", line 1123, in main
    app.run(steps=inps.runSteps, plot=inps.plot)
  File "/home/russell/tools/MintPy/mintpy/smallbaselineApp.py", line 1088, in run
    self.run_timeseries2velocity(sname)
  File "/home/russell/tools/MintPy/mintpy/smallbaselineApp.py", line 901, in run_timeseries2velocity
    mintpy.timeseries2velocity.main(iargs)
  File "/home/russell/tools/MintPy/mintpy/timeseries2velocity.py", line 695, in main
    inps = read_date_info(inps)
  File "/home/russell/tools/MintPy/mintpy/timeseries2velocity.py", line 261, in read_date_info
    tsobj.open()
  File "/home/russell/tools/MintPy/mintpy/objects/stack.py", line 173, in open
    self.get_metadata()
  File "/home/russell/tools/MintPy/mintpy/objects/stack.py", line 214, in get_metadata
    self.metadata['REF_DATE'] = dateList[0]
IndexError: list index out of range

Full error message

See the bottom of the code snippet above.

I tried forcing the reference.date and velocity dates but it isn't working. I note that referenced_date.txt contains the middle of my three dates and not the first.

System information

########## 1. load_data

a. autoPath - automatic path pattern defined in mintpy.defaults.auto_path.AUTOPATH*

b. load_data.py -H to check more details and example inputs.

c. compression to save disk usage for ifgramStack.h5 file:

no - save 0% disk usage, fast [default]

lzf - save ~57% disk usage, relative slow

gzip - save ~62% disk usage, very slow [not recommend]

mintpy.load.processor = isce #[isce, aria, snap, gamma, roipac], auto for isce mintpy.load.autoPath = auto #[yes / no], auto for no, use pre-defined auto path mintpy.load.updateMode = auto #[yes / no], auto for yes, skip re-loading if HDF5 files are complete mintpy.load.compression = auto #[gzip / lzf / no], auto for no.

---------for ISCE only:

mintpy.load.metaFile = /media/storage/isce_stack/reference/IW*.xml #[path of common metadata file for the stack], i.e.: ./reference/IW1.xml, ./referenceShelve/data.dat mintpy.load.baselineDir = /media/storage/isce_stack/baselines #[path of the baseline dir], i.e.: ./baselines

---------interferogram datasets:

mintpy.load.unwFile = /media/storage/iscestack/merged/interferograms/*/filt.unw #[path pattern of unwrapped interferogram files] mintpy.load.corFile = /media/storage/isce_stack/merged/interferograms//filt_.cor #[path pattern of spatial coherence files] mintpy.load.connCompFile = /media/storage/isce_stack/merged/interferograms//filt_*.unw.conncomp #[path pattern of connected components files], optional but recommend mintpy.load.intFile = no #[path pattern of wrapped interferogram files], optional mintpy.load.ionoFile = no #[path pattern of ionospheric delay files], optional mintpy.load.magFile = no #[path pattern of interferogram magnitude files], optional

---------offset datasets (optional):

mintpy.load.azOffFile = auto #[path pattern of azimuth offset file], optional mintpy.load.rgOffFile = auto #[path pattern of range offset file], optional mintpy.load.offSnrFile = auto #[path pattern of offset signal-to-noise ratio file], optional

---------geometry datasets:

mintpy.load.demFile = /media/storage/isce_stack/merged/geom_reference/hgt.rdr #[path of DEM file] mintpy.load.lookupYFile = /media/storage/isce_stack/merged/geom_reference/lat.rdr #[path of latitude /row /y coordinate file], not required for geocoded data mintpy.load.lookupXFile = /media/storage/isce_stack/merged/geom_reference/lon.rdr #[path of longitude/column/x coordinate file], not required for geocoded data mintpy.load.incAngleFile = /media/storage/isce_stack/merged/geom_reference/los.rdr #[path of incidence angle file], optional but recommend mintpy.load.azAngleFile = /media/storage/isce_stack/merged/geom_reference/los.rdr #[path of azimuth angle file], optional mintpy.load.shadowMaskFile = /media/storage/isce_stack/merged/geom_reference/shadowMask.rdr #[path of shadow mask file], optional but recommend mintpy.load.waterMaskFile = auto #[path of water mask file], optional but recommend mintpy.load.bperpFile = auto #[path pattern of 2D perpendicular baseline file], optional

---------multilook (optional):

multilook while loading data with nearest interpolation, to reduce dataset size

mintpy.load.ystep = auto #[int >= 1], auto for 1 - no multilooking mintpy.load.xstep = auto #[int >= 1], auto for 1 - no multilooking

---------subset (optional):

if both yx and lalo are specified, use lalo option unless a) no lookup file AND b) dataset is in radar coord

mintpy.subset.yx = auto #[y0:y1,x0:x1 / no], auto for no mintpy.subset.lalo = auto #[lat0:lat1,lon0:lon1 / no], auto for no

########## 2. modify_network

reference: Yunjun et al. (2019, section 4.2 and 5.3.1); Chaussard et al. (2015, GRL)

1) Coherence-based network modification = (threshold + MST) by default

It calculates a average coherence value for each interferogram using spatial coherence and input mask (with AOI)

Then it finds a minimum spanning tree (MST) network with inverse of average coherence as weight (keepMinSpanTree)

For all interferograms except for MST's, exclude those with average coherence < minCoherence.

mintpy.network.coherenceBased = auto #[yes / no], auto for no, exclude interferograms with coherence < minCoherence mintpy.network.keepMinSpanTree = auto #[yes / no], auto for yes, keep interferograms in Min Span Tree network mintpy.network.minCoherence = auto #[0.0-1.0], auto for 0.7 mintpy.network.maskFile = auto #[file name, no], auto for waterMask.h5 or no [if no waterMask.h5 found] mintpy.network.aoiYX = auto #[y0:y1,x0:x1 / no], auto for no, area of interest for coherence calculation mintpy.network.aoiLALO = auto #[lat0:lat1,lon0:lon1 / no], auto for no - use the whole area

2) Network modification based on temporal/perpendicular baselines, date etc.

mintpy.network.tempBaseMax = auto #[1-inf, no], auto for no, max temporal baseline in days mintpy.network.perpBaseMax = auto #[1-inf, no], auto for no, max perpendicular spatial baseline in meter mintpy.network.connNumMax = auto #[1-inf, no], auto for no, max number of neighbors for each acquisition mintpy.network.startDate = auto #[20090101 / no], auto for no mintpy.network.endDate = auto #[20110101 / no], auto for no mintpy.network.excludeDate = auto #[20080520,20090817 / no], auto for no mintpy.network.excludeIfgIndex = auto #[1:5,25 / no], auto for no, list of ifg index (start from 0) mintpy.network.referenceFile = auto #[date12_list.txt / ifgramStack.h5 / no], auto for no

########## 3. reference_point

Reference all interferograms to one common point in space

auto - randomly select a pixel with coherence > minCoherence

however, manually specify using prior knowledge of the study area is highly recommended

with the following guideline (section 4.3 in Yunjun et al., 2019):

1) located in a coherence area, to minimize the decorrelation effect.

2) not affected by strong atmospheric turbulence, i.e. ionospheric streaks

3) close to and with similar elevation as the AOI, to minimize the impact of spatially correlated atmospheric delay

mintpy.reference.yx = auto #[257,151 / auto] mintpy.reference.lalo = auto #[31.8,130.8 / auto] mintpy.reference.maskFile = auto #[filename / no], auto for maskConnComp.h5 mintpy.reference.coherenceFile = auto #[filename], auto for avgSpatialCoh.h5 mintpy.reference.minCoherence = auto #[0.0-1.0], auto for 0.85, minimum coherence for auto method

########## quick_overview

A quick assessment of:

1) possible groud deformation

using the velocity from the traditional interferogram stacking

reference: Zebker et al. (1997, JGR)

2) distribution of phase unwrapping error

from the number of interferogram triplets with non-zero integer ambiguity of closue phase

reference: T_int in Yunjun et al. (2019, CAGEO). Related to section 3.2, equation (8-9) and Fig. 3d-e.

########## 4. correct_unwrap_error (optional)

connected components (mintpy.load.connCompFile) are required for this step.

reference: Yunjun et al. (2019, section 3)

supported methods:

a. phase_closure - suitable for highly redundant network

b. bridging - suitable for regions separated by narrow decorrelated features, e.g. rivers, narrow water bodies

c. bridging+phase_closure

mintpy.unwrapError.method = auto #[bridging / phase_closure / bridging+phase_closure / no], auto for no mintpy.unwrapError.waterMaskFile = auto #[waterMask.h5 / no], auto for waterMask.h5 or no [if not found]

briding options:

ramp - a phase ramp could be estimated based on the largest reliable region, removed from the entire interferogram

before estimating the phase difference between reliable regions and added back after the correction.

bridgePtsRadius - half size of the window used to calculate the median value of phase difference

mintpy.unwrapError.ramp = auto #[linear / quadratic], auto for no; recommend linear for L-band data mintpy.unwrapError.bridgePtsRadius = auto #[1-inf], auto for 50, half size of the window around end points

########## 5. invert_network

Invert network of interferograms into time-series using weighted least sqaure (WLS) estimator.

weighting options for least square inversion [fast option available but not best]:

a. var - use inverse of covariance as weight (Tough et al., 1995; Guarnieri & Tebaldini, 2008) [recommended]

b. fim - use Fisher Information Matrix as weight (Seymour & Cumming, 1994; Samiei-Esfahany et al., 2016).

c. coh - use coherence as weight (Perissin & Wang, 2012)

d. no - uniform weight (Berardino et al., 2002) [fast]

SBAS (Berardino et al., 2002) = minNormVelocity (yes) + weightFunc (no)

mintpy.networkInversion.weightFunc = auto #[var / fim / coh / no], auto for var mintpy.networkInversion.waterMaskFile = auto #[filename / no], auto for waterMask.h5 or no [if not found] mintpy.networkInversion.minNormVelocity = auto #[yes / no], auto for yes, min-norm deformation velocity / phase mintpy.networkInversion.residualNorm = auto #[L2 ], auto for L2, norm minimization solution

mask options for unwrapPhase of each interferogram before inversion (recommed if weightFunct=no):

a. coherence - mask out pixels with spatial coherence < maskThreshold

b. connectComponent - mask out pixels with False/0 value

c. no - no masking [recommended].

d. offsetSNR - mask out pixels with offset SNR < maskThreshold [for offset]

mintpy.networkInversion.maskDataset = auto #[coherence / connectComponent / offsetSNR / no], auto for no mintpy.networkInversion.maskThreshold = auto #[0-inf], auto for 0.4 mintpy.networkInversion.minRedundancy = auto #[1-inf], auto for 1.0, min num_ifgram for every SAR acquisition

Temporal coherence is calculated and used to generate the mask as the reliability measure

reference: Pepe & Lanari (2006, IEEE-TGRS)

mintpy.networkInversion.minTempCoh = auto #[0.0-1.0], auto for 0.7, min temporal coherence for mask mintpy.networkInversion.minNumPixel = auto #[int > 1], auto for 100, min number of pixels in mask above mintpy.networkInversion.shadowMask = auto #[yes / no], auto for yes [if shadowMask is in geometry file] or no.

########## correct_LOD

Local Oscillator Drift (LOD) correction (for Envisat only)

reference: Marinkovic and Larsen (2013, Proc. LPS)

automatically applied to Envisat data (identified via PLATFORM attribute)

and skipped for all the other satellites.

########## 6. correct_troposphere (optional and recommended)

correct tropospheric delay using the following methods:

a. height_correlation - correct stratified tropospheric delay (Doin et al., 2009, J Applied Geop)

b. pyaps - use Global Atmospheric Models (GAMs) data (Jolivet et al., 2011; 2014)

ERA5 - ERA-5 from ECMWF [need to install pyaps3 on GitHub; recommended and turn ON by default]

MERRA - MERRA-2 from NASA [need to install pyaps on Caltech/EarthDef]

NARR - NARR from NOAA [need to install pyaps on Caltech/EarthDef; recommended for N America]

c. gacos - use GACOS with the iterative tropospheric decomposition model (Yu et al., 2017, JGR; 2018, JGR; 2018, RSE)

Manually download GACOS products at http://www.gacos.net/index.html for all acquisitions before running this step.

mintpy.troposphericDelay.method = height_correlation #[pyaps / height_correlation / gacos / no], auto for pyaps

Notes for pyaps:

a. GAM data latency: with the most recent SAR data, there will be GAM data missing, the correction

will be applied to dates with GAM data available and skipped for the others.

b. WEATHER_DIR: if you define an environmental variable named WEATHER_DIR to contain the path to a

directory, then MintPy applications will download the GAM files into the indicated directory.

MintPy application will look for the GAM files in the directory before downloading a new one to

prevent downloading multiple copies if you work with different dataset that cover the same date/time.

mintpy.troposphericDelay.weatherModel = auto #[ERA5 / MERRA / NARR], auto for ERA5 mintpy.troposphericDelay.weatherDir = auto #[path2directory], auto for WEATHER_DIR or "./"

Notes for height_correlation:

Extra multilooking is applied to estimate the empirical phase/elevation ratio ONLY.

For an dataset with 5 by 15 looks, looks=8 will generate phase with (58) by (158) looks

to estimate the empirical parameter; then apply the correction to original phase (with 5 by 15 looks),

if the phase/elevation correlation is larger than minCorrelation.

mintpy.troposphericDelay.polyOrder = auto #[1 / 2 / 3], auto for 1 mintpy.troposphericDelay.looks = auto #[1-inf], auto for 8, extra multilooking num mintpy.troposphericDelay.minCorrelation = auto #[0.0-1.0], auto for 0

Notes for gacos:

Set the path below to directory that contains the downloaded .ztd and .ztd.rsc files

mintpy.troposphericDelay.gacosDir = auto # [path2directory], auto for "./GACOS"

########## 7. deramp (optional)

Estimate and remove a phase ramp for each acquisition based on the reliable pixels.

Recommended for localized deformation signals, i.e. volcanic deformation, landslide and land subsidence, etc.

NOT recommended for long spatial wavelength deformation signals, i.e. co-, post- and inter-seimic deformation.

mintpy.deramp = auto #[no / linear / quadratic], auto for no - no ramp will be removed mintpy.deramp.maskFile = auto #[filename / no], auto for maskTempCoh.h5, mask file for ramp estimation

########## 8. correct_topography (optional and recommended)

Topographic residual (DEM error) correction

reference: Fattahi and Amelung (2013, IEEE-TGRS)

stepFuncDate - Specify stepFuncDate option if you know there are sudden displacement jump in your area,

i.e. volcanic eruption, or earthquake, and check timeseriesStepModel.h5 afterward for their estimation.

excludeDate - Dates excluded for error estimation only

pixelwiseGeometry - Use pixel-wise geometry info, i.e. incidence angle and slant range distance

yes - use pixel-wise geometry when they are available [slow; used by default]

no - use mean geometry [fast]

mintpy.topographicResidual = auto #[yes / no], auto for yes mintpy.topographicResidual.polyOrder = auto #[1-inf], auto for 2, poly order of temporal deformation model mintpy.topographicResidual.phaseVelocity = auto #[yes / no], auto for no - phase, use phase velocity for minimization mintpy.topographicResidual.stepFuncDate = auto #[20080529,20100611 / no], auto for no, date of step jump mintpy.topographicResidual.excludeDate = auto #[20070321 / txtFile / no], auto for exclude_date.txt mintpy.topographicResidual.pixelwiseGeometry = auto #[yes / no], auto for yes, use pixel-wise geometry info

########## 9.1 residual_RMS (root mean squares for noise evaluation)

Calculate the Root Mean Square (RMS) of residual phase time-series for each acquisition

reference: Yunjun et al. (2019, section 4.9 and 5.4)

To get rid of long wavelength component in space, a ramp is removed for each acquisition

Set optimal reference date to date with min RMS

Set exclude dates (outliers) to dates with RMS > cutoff * median RMS (Median Absolute Deviation)

mintpy.residualRMS.maskFile = auto #[file name / no], auto for maskTempCoh.h5, mask for ramp estimation mintpy.residualRMS.deramp = auto #[quadratic / linear / no], auto for quadratic mintpy.residualRMS.cutoff = auto #[0.0-inf], auto for 3

########## 9.2 reference_date

Reference all time-series to one date in time

reference: Yunjun et al. (2019, section 4.9)

no - do not change the default reference date (1st date)

mintpy.reference.date = 20201127 #[reference_date.txt / 20090214 / no], auto for reference_date.txt

########## 10. velocity

Estimate linear velocity and its standard deviation from time-series

and from tropospheric delay file if exists.

reference: Fattahi and Amelung (2015, JGR)

mintpy.velocity.excludeDate = auto #[exclude_date.txt / 20080520,20090817 / no], auto for exclude_date.txt mintpy.velocity.startDate = auto #[20070101 / no], auto for no mintpy.velocity.endDate = auto #[20101230 / no], auto for no

Bootstrapping

refernce: Efron and Tibshirani (1986, Stat. Sci.)

mintpy.velocity.bootstrap = auto #[yes / no], auto for no, use bootstrap mintpy.velocity.bootstrapCount = auto #[int>1], auto for 400, number of iterations for bootstrapping

########## 11.1 geocode (post-processing)

for input dataset in radar coordinates only

commonly used resolution in meters and in degrees (on equator)

100, 60, 50, 30, 20, 10

0.000925926, 0.000555556, 0.000462963, 0.000277778, 0.000185185, 0.000092593

mintpy.geocode = auto #[yes / no], auto for yes mintpy.geocode.SNWE = auto #[-1.2,0.5,-92,-91 / none ], auto for none, output extent in degree mintpy.geocode.laloStep = auto #[-0.000555556,0.000555556 / None], auto for None, output resolution in degree mintpy.geocode.interpMethod = auto #[nearest], auto for nearest, interpolation method mintpy.geocode.fillValue = auto #[np.nan, 0, ...], auto for np.nan, fill value for outliers.

########## 11.2 google_earth (post-processing) mintpy.save.kmz = auto #[yes / no], auto for yes, save geocoded velocity to Google Earth KMZ file

########## 11.3 hdfeos5 (post-processing) mintpy.save.hdfEos5 = auto #[yes / no], auto for no, save time-series to HDF-EOS5 format mintpy.save.hdfEos5.update = auto #[yes / no], auto for no, put XXXXXXXX as endDate in output filename mintpy.save.hdfEos5.subset = auto #[yes / no], auto for no, put subset range info in output filename

########## 11.4 plot mintpy.plot = auto #[yes / no], auto for yes, plot files generated by default processing to pic folder



I note that `h5dump avgSpatialCoh.h5` returns data so some of the earlier steps are working ok. The pdf file outputs look real. 

I also read elsewhere that topsStack should share the platform details with MintPy, so I wonder about `SAR platform/sensor : unknown from project name "None"`.

Any advice is very welcome.

Thanks.
welcome[bot] commented 3 years ago

👋 Thanks for opening your first issue here! Please make sure you filled out the template with as much detail as possible. We appreciate that you took the time to contribute! Make sure you read our contributing guidelines

yunjunz commented 3 years ago

It seems like something wrong in the ./inputs/ERA5.h5 file, could you plot it using view.py to see how it looks like?

If there are issues with this file, it's probably caused by the previously unsuccessful run with mintpy.troposphericDelay.method = pyaps, delete the ERA5.h5 file and re-run should solve the issue.

The "unknown SAR platform" msg is fine since it is before running prep_isce.py, which extracts the metadata from ISCE/topsStack.

hardreddata commented 3 years ago

Hi @yunjunz

I was unable to view.py the ERA.h5 file and it failed with the same error. I deleted it and run again and this all works now.

I think the problem was my first smallbaselineApp.py run failed when looking for ECMWF data (no credentials) and after setting mintpy.troposphericDelay.method = height_correlation I should have deleted a bunch of stuff.

How can I prepare plots like the below? I wondered if they should already be in my pic folder?

image

This works view.py timeseries.h5 but the below does not

tsview.py timeseries.h5
open timeseries file: timeseries.h5
data   coverage in y/x: (0, 0, 5033, 2133)
subset coverage in y/x: (0, 0, 5033, 2133)
data   coverage in lat/lon: None
subset coverage in lat/lon: None
------------------------------------------------------------------------
ASCENDING orbit -> flip up-down
reading timeseries from file timeseries.h5 ...
reference to pixel: [677, 2775]
Traceback (most recent call last):
  File "/home/russell/tools/MintPy/mintpy/tsview.py", line 826, in <module>
    main(sys.argv[1:])
  File "/home/russell/tools/MintPy/mintpy/tsview.py", line 819, in main
    obj.plot()
  File "/home/russell/tools/MintPy/mintpy/tsview.py", line 585, in plot
    self.ts_data, self.mask = read_timeseries_data(self)[0:2]
  File "/home/russell/tools/MintPy/mintpy/tsview.py", line 332, in read_timeseries_data
    vprint('reference to date: {}'.format(inps.date_list[inps.ref_idx]))
IndexError: list index out of range

I tried deleting the timeseries*.hdf and then later starting fresh but am not winning.

Thanks for the help.

yunjunz commented 3 years ago

I prepared the figure using view.py and tsview.py, as you have figured out.

For the error, could you run info.py timeseries.h5 to see the metadata? It seems REF_DATE is missing. I would recommend debugging in tsview.py#L332 if you are familiar with python.

hardreddata commented 3 years ago

Thanks again. The info.py output:

******************** Basic File Info ************************
file name: /media/storage/isce_stack/timeseries.h5
file type: timeseries
coordinates : RADAR

******************** Date Stat Info *************************
Start Date: 20201127
End   Date: 20201221
Number of acquisitions    : 3
Std. of acquisition times : 0.03 yeras
----------------------
List of dates:
['20201127', '20201209', '20201221']
----------------------
List of dates in years:
[2020.9062286105407, 2020.9390828199862, 2020.971937029432]

******************** HDF5 File Structure ********************
Attributes in / level:
  ALOOKS                                     3
  ANTENNA_SIDE                               -1
  AZIMUTH_PIXEL_SIZE                         46.77144780487454
  CENTER_INCIDENCE_ANGLE                     36.16769
  CENTER_LINE_UTC                            31180.0
  DATA_TYPE                                  float32
  DATE12                                     201127-201209
  EARTH_RADIUS                               6357725.271374925
  END_DATE                                   20201221
  FILE_LENGTH                                2133
  FILE_PATH                                  /media/storage/isce_stack/merged/interferograms/20201127_20201209/filt_fine.unw
  FILE_TYPE                                  timeseries
  HEADING                                    -13.12479484007516
  HEIGHT                                     709385.3632063618
  ISCE_VERSION                               Release: 2.3, svn-2531, 20190112. Current: svn-Unknown.
  LAT_REF1                                   -34.14015098133027
  LAT_REF2                                   -33.699326972753035
  LAT_REF3                                   -33.89960237436327
  LAT_REF4                                   -33.46126666596746
  LENGTH                                     2133
  LON_REF1                                   149.92407844015443
  LON_REF2                                   151.79302589403324
  LON_REF3                                   149.85289248127688
  LON_REF4                                   151.71116794676286
  NCORRLOOKS                                 16.14177671949939
  ORBIT_DIRECTION                            ASCENDING
  PLATFORM                                   sen
  POLARIZATION                               VV
  PRF                                        1717.128973878037
  PROCESSOR                                  isce
  P_BASELINE_BOTTOM_HDR                      49.5275994851483
  P_BASELINE_TOP_HDR                         49.5275994851483
  RANGE_PIXEL_SIZE                           20.966059032437908
  REF_DATE                                   20201127
  REF_X                                      2775
  REF_Y                                      677
  RLOOKS                                     9
  STARTING_RANGE                             803512.7992238925
  START_DATE                                 20201127
  UNIT                                       m
  WAVELENGTH                                 0.05546576
  WIDTH                                      5033
  access_mode                                read
  altitude                                   709385.3632063618
  azimuthPixelSize                           46.77144780487454
  azimuthResolution                          22.5
  beam_mode                                  IW
  beam_swath                                 12
  byte_order                                 l
  data_type                                  float
  earthRadius                                6357725.271374925
  family                                     unwimage
  file_name                                  /media/storage/isce_stack/merged/interferograms/20201127_20201209/filt_fine.unw
  firstFrameNumber                           1069
  first_frame                                1069
  image_type                                 unw
  lastFrameNumber                            1070
  last_frame                                 1070
  length                                     2133
  mintpy.networkInversion.maskDataset        False
  mintpy.networkInversion.maskThreshold      0.4
  mintpy.networkInversion.minNormVelocity    True
  mintpy.networkInversion.minRedundancy      1.0
  mintpy.networkInversion.numIfgram          2
  mintpy.networkInversion.obsDatasetName     unwrapPhase
  mintpy.networkInversion.weightFunc         var
  name                                       unwimage_name
  number_bands                               2
  orbitNumber                                35431
  passDirection                              ASCENDING
  polarization                               VV
  prf                                        1717.128973878037
  radarWavelength                            0.05546576
  rangePixelSize                             20.966059032437908
  rangeResolution                            2.7
  relative_orbit                             9
  scheme                                     BIL
  startUTC                                   2020-11-27 08:39:40.205553
  startingRange                              803512.7992238925
  stopUTC                                    2020-11-27 08:39:46.592164
  swathNumber                                12
  trackNumber                                9
  width                                      5033
  xmax                                       5033
  xmin                                       0

HDF5 dataset "/bperp               ": shape (3,)                , dtype <float32>
HDF5 dataset "/date                ": shape (3,)                , dtype <|S8>
HDF5 dataset "/timeseries          ": shape (3, 2133, 5033)     , dtype <float32>

It seems to know where to begin

******************** Date Stat Info *************************
Start Date: 20201127
End   Date: 20201221
Number of acquisitions    : 3
Std. of acquisition times : 0.03 yeras
----------------------
List of dates:
['20201127', '20201209', '20201221']

but then arrives at things like

  startUTC                                   2020-11-27 08:39:40.205553
  startingRange                              803512.7992238925
  stopUTC                                    2020-11-27 08:39:46.592164

I will take a look in Python but I am very open to any advice based on the above. REF_DATE looks good to me.

yunjunz commented 3 years ago

@RussellGrew The cause is the default ref_idx value of 3, which is larger than the number of acquisitions in the file you have.

Change lin 188-197 from:

    if inps.ref_date:
        inps.ref_idx = inps.date_list.index(inps.ref_date)
    else:
        inps.ref_idx = 3

    if not inps.idx:
        if inps.ref_idx < inps.num_date / 2.:
            inps.idx = inps.num_date - 3
        else:
            inps.idx = 3

to

    if inps.ref_date:
        inps.ref_idx = inps.date_list.index(inps.ref_date)
    else:
        inps.ref_idx = 1

    if not inps.idx:
        if inps.ref_idx < inps.num_date / 2.:
            inps.idx = inps.num_date - 2
        else:
            inps.idx = 2

should work, could you confirm? If so, please consider issuing a PR for it.

hardreddata commented 3 years ago

I was just reviewing that section of code when I saw your message.

The program no longer fails and displays the cumulative displacement map with slider, and a blank point displacement time series axis. This second plot only shows data after I click a point in the map on the first plot. I am not sure if this is intended?

image

If this is looking good I can submit the pull request for you.

Cheers.

yunjunz commented 3 years ago

Great. This is working as expected. Do so pls.