search

insarlab / MintPy

Miami InSAR time-series software in Python
https://mintpy.readthedocs.io
Other
597 stars 256 forks source link

ALOS1 mintpy load_data.py error #688

Closed zhuhaxixiong closed 2 years ago

zhuhaxixiong commented 2 years ago

Description of the problem

i generate the stack of alos1 (level 1.0 ) data by isce 2.5.3 stripMapStack.py,: stackStripMap.py -s /public1/home/scf0015/mazhangfeng/SLC -d /public1/home/scf0015/mazhangfeng/DEM/demLat_N04_N06_Lon_E095_E097.dem.wgs84/ -m 20080916 -W interferogram -t 500 -b 1000 -a 1 -r 2 -u snaphu --nofocus when i process SBAS in mintpy ,all the parameter set as in the following txt: but when i do the first step of smallbaseline program (load data),the error occurs:

Full script that generated the error

# vim: set filetype=cfg:
##------------------------ smallbaselineApp.cfg ------------------------##
########## computing resource configuration
mintpy.compute.maxMemory = auto #[float > 0.0], auto for 4, max memory to allocate in GB
## parallel processing with dask
## currently apply to steps: invert_network, correct_topography
## cluster   = none to turn off the parallel computing
## numWorker = all  to use all locally available cores (for cluster = local only)
## config    = none to rollback to the default name (same as the cluster type; for cluster != local)
mintpy.compute.cluster   = auto #[local / slurm / pbs / lsf / none], auto for none, cluster type
mintpy.compute.numWorker = auto #[int > 1 / all], auto for 4 (local) or 40 (non-local), num of workers
mintpy.compute.config    = auto #[none / slurm / pbs / lsf ], auto for none (same as cluster), config name

########## 1. load_data
##---------add attributes manually
## MintPy requires attributes listed at: https://mintpy.readthedocs.io/en/latest/api/attributes/
## Missing attributes can be added below manually (uncomment #), e.g.
# ORBIT_DIRECTION = ascending
# PLATFORM = CSK
# ...
## a. autoPath - automatic path pattern defined in mintpy.defaults.auto_path.AUTO_PATH_*
## b. load_data.py -H to check more details and example inputs.
## c. compression to save disk usage for ifgramStack.h5 file:
## no   - save   0% disk usage, fast [default]
## lzf  - save ~57% disk usage, relative slow
## gzip - save ~62% disk usage, very slow [not recommend]
mintpy.load.processor      = auto  #[isce, aria, snap, gamma, roipac, hyp3], auto for isce
mintpy.load.autoPath       = auto  #[yes / no], auto for no, use pre-defined auto path
mintpy.load.updateMode     = auto  #[yes / no], auto for yes, skip re-loading if HDF5 files are complete
mintpy.load.compression    = auto  #[gzip / lzf / no], auto for no.
##---------for ISCE only:
mintpy.load.metaFile       = /mnt/merged/SLC/20080916/referenceShelve/data.dat  #[path of common metadata file for the stack], i.e.: ./reference/IW1.xml, ./referenceShelve/data.dat
mintpy.load.baselineDir    = /mnt/baselines  #[path of the baseline dir], i.e.: ./baselines
##---------interferogram datasets:
mintpy.load.unwFile        = /mnt/Igrams/*/filt_*.unw  #[path pattern of unwrapped interferogram files]
mintpy.load.corFile        = /mnt/Igrams/*/filt_*.cor  #[path pattern of spatial coherence       files]
mintpy.load.connCompFile   = /mnt/Igrams/*/filt_*.unw.conncomp  #[path pattern of connected components    files], optional but recommend
mintpy.load.intFile        = auto  #[path pattern of wrapped interferogram   files], optional
mintpy.load.ionoFile       = auto  #[path pattern of ionospheric delay       files], optional
mintpy.load.magFile        = auto  #[path pattern of interferogram magnitude files], optional
##---------offset datasets (optional):
mintpy.load.azOffFile      = auto  #[path pattern of azimuth offset file], optional
mintpy.load.rgOffFile      = auto  #[path pattern of range   offset file], optional
mintpy.load.offSnrFile     = auto  #[path pattern of offset signal-to-noise ratio file], optional
##---------geometry datasets:
mintpy.load.demFile        = /mnt/geom_reference/hgt.rdr  #[path of DEM file]
mintpy.load.lookupYFile    = /mnt/geom_reference/lat.rdr  #[path of latitude /row   /y coordinate file], not required for geocoded data
mintpy.load.lookupXFile    = /mnt/geom_reference/lon.rdr  #[path of longitude/column/x coordinate file], not required for geocoded data
mintpy.load.incAngleFile   = /mnt/geom_reference/los.rdr  #[path of incidence angle file], optional but recommend
mintpy.load.azAngleFile    = /mnt/geom_reference/los.rdr  #[path of azimuth   angle file], optional
mintpy.load.shadowMaskFile = /mnt/geom_reference/shadowMask.hdr  #[path of shadow mask file], optional but recommend
mintpy.load.waterMaskFile  = auto  #[path of water  mask file], optional but recommend
mintpy.load.bperpFile      = auto  #[path pattern of 2D perpendicular baseline file], optional
##---------multilook (optional):
## multilook while loading data with nearest interpolation, to reduce dataset size
mintpy.load.ystep          = 1    #[int >= 1], auto for 1 - no multilooking
mintpy.load.xstep          = 1    #[int >= 1], auto for 1 - no multilooking
##---------subset (optional):
## if both yx and lalo are specified, use lalo option unless a) no lookup file AND b) dataset is in radar coord
mintpy.subset.yx           = auto    #[y0:y1,x0:x1 / no], auto for no
mintpy.subset.lalo         = auto    #[S:N,W:E / no], auto for no

########## 2. modify_network
## reference: Yunjun et al. (2019, section 4.2 and 5.3.1); Chaussard et al. (2015, GRL)
## 1) Coherence-based network modification = (threshold + MST) by default
## It calculates a average coherence value for each interferogram using spatial coherence and input mask (with AOI)
## Then it finds a minimum spanning tree (MST) network with inverse of average coherence as weight (keepMinSpanTree)
## For all interferograms except for MST's, exclude those with average coherence < minCoherence.
mintpy.network.coherenceBased  = auto  #[yes / no], auto for no, exclude interferograms with coherence < minCoherence
mintpy.network.keepMinSpanTree = auto  #[yes / no], auto for yes, keep interferograms in Min Span Tree network
mintpy.network.minCoherence    = auto  #[0.0-1.0], auto for 0.7
mintpy.network.maskFile        = auto  #[file name, no], auto for waterMask.h5 or no [if no waterMask.h5 found]
mintpy.network.aoiYX           = auto  #[y0:y1,x0:x1 / no], auto for no, area of interest for coherence calculation
mintpy.network.aoiLALO         = auto  #[S:N,W:E / no], auto for no - use the whole area

## 2) Network modification based on temporal/perpendicular baselines, date etc.
mintpy.network.tempBaseMax     = 3650  #[1-inf, no], auto for no, max temporal baseline in days
mintpy.network.perpBaseMax     = 10000  #[1-inf, no], auto for no, max perpendicular spatial baseline in meter
mintpy.network.connNumMax      = auto  #[1-inf, no], auto for no, max number of neighbors for each acquisition
mintpy.network.startDate       = auto  #[20090101 / no], auto for no
mintpy.network.endDate         = auto  #[20110101 / no], auto for no
mintpy.network.excludeDate     = auto  #[20080520,20090817 / no], auto for no
mintpy.network.excludeIfgIndex = auto  #[1:5,25 / no], auto for no, list of ifg index (start from 0)
mintpy.network.referenceFile   = auto  #[date12_list.txt / ifgramStack.h5 / no], auto for no

########## 3. reference_point
## Reference all interferograms to one common point in space
## auto - randomly select a pixel with coherence > minCoherence
## however, manually specify using prior knowledge of the study area is highly recommended
##   with the following guideline (section 4.3 in Yunjun et al., 2019):
## 1) located in a coherence area, to minimize the decorrelation effect.
## 2) not affected by strong atmospheric turbulence, i.e. ionospheric streaks
## 3) close to and with similar elevation as the AOI, to minimize the impact of spatially correlated atmospheric delay
mintpy.reference.yx            = auto   #[257,151 / auto]
mintpy.reference.lalo          = auto   #[31.8,130.8 / auto]
mintpy.reference.maskFile      = auto   #[filename / no], auto for maskConnComp.h5
mintpy.reference.coherenceFile = auto   #[filename], auto for avgSpatialCoh.h5
mintpy.reference.minCoherence  = auto   #[0.0-1.0], auto for 0.85, minimum coherence for auto method

########## quick_overview
## A quick assessment of:
## 1) possible groud deformation
##    using the velocity from the traditional interferogram stacking
##    reference: Zebker et al. (1997, JGR)
## 2) distribution of phase unwrapping error
##    from the number of interferogram triplets with non-zero integer ambiguity of closue phase
##    reference: T_int in Yunjun et al. (2019, CAGEO). Related to section 3.2, equation (8-9) and Fig. 3d-e.

########## 4. correct_unwrap_error (optional)
## connected components (mintpy.load.connCompFile) are required for this step.
## reference: Yunjun et al. (2019, section 3)
## supported methods:
## a. phase_closure           - suitable for highly redundant network
## b. bridging                - suitable for regions separated by narrow decorrelated features, e.g. rivers, narrow water bodies
## c. bridging+phase_closure
mintpy.unwrapError.method          = auto  #[bridging / phase_closure / bridging+phase_closure / no], auto for no
mintpy.unwrapError.waterMaskFile   = auto  #[waterMask.h5 / no], auto for waterMask.h5 or no [if not found]

## briding options:
## ramp - a phase ramp could be estimated based on the largest reliable region, removed from the entire interferogram
##        before estimating the phase difference between reliable regions and added back after the correction.
## bridgePtsRadius - half size of the window used to calculate the median value of phase difference
mintpy.unwrapError.ramp            = auto  #[linear / quadratic], auto for no; recommend linear for L-band data
mintpy.unwrapError.bridgePtsRadius = auto  #[1-inf], auto for 50, half size of the window around end points

########## 5. invert_network
## Invert network of interferograms into time-series using weighted least sqaure (WLS) estimator.
## weighting options for least square inversion [fast option available but not best]:
## a. var - use inverse of covariance as weight (Tough et al., 1995; Guarnieri & Tebaldini, 2008) [recommended]
## b. fim - use Fisher Information Matrix as weight (Seymour & Cumming, 1994; Samiei-Esfahany et al., 2016).
## c. coh - use coherence as weight (Perissin & Wang, 2012)
## d. no  - uniform weight (Berardino et al., 2002) [fast]
## SBAS (Berardino et al., 2002) = minNormVelocity (yes) + weightFunc (no)
mintpy.networkInversion.weightFunc      = auto #[var / fim / coh / no], auto for var
mintpy.networkInversion.waterMaskFile   = auto #[filename / no], auto for waterMask.h5 or no [if not found]
mintpy.networkInversion.minNormVelocity = auto #[yes / no], auto for yes, min-norm deformation velocity / phase
mintpy.networkInversion.residualNorm    = auto #[L2 ], auto for L2, norm minimization solution

## mask options for unwrapPhase of each interferogram before inversion (recommed if weightFunct=no):
## a. coherence        - mask out pixels with spatial coherence < maskThreshold
## b. connectComponent - mask out pixels with False/0 value
## c. no               - no masking [recommended].
## d. offsetSNR        - mask out pixels with offset SNR < maskThreshold [for offset]
mintpy.networkInversion.maskDataset   = auto #[coherence / connectComponent / offsetSNR / no], auto for no
mintpy.networkInversion.maskThreshold = auto #[0-inf], auto for 0.4
mintpy.networkInversion.minRedundancy = auto #[1-inf], auto for 1.0, min num_ifgram for every SAR acquisition

## Temporal coherence is calculated and used to generate the mask as the reliability measure
## reference: Pepe & Lanari (2006, IEEE-TGRS)
mintpy.networkInversion.minTempCoh  = auto #[0.0-1.0], auto for 0.7, min temporal coherence for mask
mintpy.networkInversion.minNumPixel = auto #[int > 1], auto for 100, min number of pixels in mask above
mintpy.networkInversion.shadowMask  = auto #[yes / no], auto for yes [if shadowMask is in geometry file] or no.

########## correct_LOD
## Local Oscillator Drift (LOD) correction (for Envisat only)
## reference: Marinkovic and Larsen (2013, Proc. LPS)
## automatically applied to Envisat data (identified via PLATFORM attribute)
## and skipped for all the other satellites.

########## correct_SET
## Solid Earth tides (SET) correction [need to install insarlab/PySolid]
## reference: Milbert (2018); Fattahi et al. (2020, AGU)
mintpy.solidEarthTides = auto #[yes / no], auto for no

########## 6. correct_troposphere (optional and recommended)
## correct tropospheric delay using the following methods:
## a. height_correlation - correct stratified tropospheric delay (Doin et al., 2009, J Applied Geop)
## b. pyaps - use Global Atmospheric Models (GAMs) data (Jolivet et al., 2011; 2014)
##      ERA5  - ERA-5       from ECMWF [need to install PyAPS from GitHub; recommended and turn ON by default]
##      MERRA - MERRA-2     from NASA  [need to install PyAPS from Caltech/EarthDef]
##      NARR  - NARR        from NOAA  [need to install PyAPS from Caltech/EarthDef; recommended for N America]
## c. gacos - use GACOS with the iterative tropospheric decomposition model (Yu et al., 2018a, RSE; 2018b, JGR)
##      need to manually download GACOS products at http://www.gacos.net for all acquisitions before running this step
mintpy.troposphericDelay.method = auto  #[pyaps / height_correlation / gacos / no], auto for pyaps

## Notes for pyaps:
## a. GAM data latency: with the most recent SAR data, there will be GAM data missing, the correction
##    will be applied to dates with GAM data available and skipped for the others.
## b. WEATHER_DIR: if you define an environment variable named WEATHER_DIR to contain the path to a
##    directory, then MintPy applications will download the GAM files into the indicated directory. 
##    MintPy application will look for the GAM files in the directory before downloading a new one to
##    prevent downloading multiple copies if you work with different dataset that cover the same date/time.
mintpy.troposphericDelay.weatherModel = auto  #[ERA5 / MERRA / NARR], auto for ERA5
mintpy.troposphericDelay.weatherDir   = auto  #[path2directory], auto for WEATHER_DIR or "./"

## Notes for height_correlation:
## Extra multilooking is applied to estimate the empirical phase/elevation ratio ONLY.
## For an dataset with 5 by 15 looks, looks=8 will generate phase with (5*8) by (15*8) looks
## to estimate the empirical parameter; then apply the correction to original phase (with 5 by 15 looks),
## if the phase/elevation correlation is larger than minCorrelation.
mintpy.troposphericDelay.polyOrder      = auto  #[1 / 2 / 3], auto for 1
mintpy.troposphericDelay.looks          = auto  #[1-inf], auto for 8, extra multilooking num
mintpy.troposphericDelay.minCorrelation = auto  #[0.0-1.0], auto for 0

## Notes for gacos:
## Set the path below to directory that contains the downloaded *.ztd* files
mintpy.troposphericDelay.gacosDir = auto # [path2directory], auto for "./GACOS"

########## 7. deramp (optional)
## Estimate and remove a phase ramp for each acquisition based on the reliable pixels.
## Recommended for localized deformation signals, i.e. volcanic deformation, landslide and land subsidence, etc.
## NOT recommended for long spatial wavelength deformation signals, i.e. co-, post- and inter-seimic deformation.
mintpy.deramp          = auto  #[no / linear / quadratic], auto for no - no ramp will be removed
mintpy.deramp.maskFile = auto  #[filename / no], auto for maskTempCoh.h5, mask file for ramp estimation

########## 8. correct_topography (optional and recommended)
## Topographic residual (DEM error) correction
## reference: Fattahi and Amelung (2013, IEEE-TGRS)
## stepFuncDate      - specify stepFuncDate option if you know there are sudden displacement jump in your area,
##                     e.g. volcanic eruption, or earthquake
## excludeDate       - dates excluded for the error estimation
## pixelwiseGeometry - use pixel-wise geometry (incidence angle & slant range distance)
##                     yes - use pixel-wise geometry if they are available [slow; used by default]
##                     no  - use the mean   geometry [fast]
mintpy.topographicResidual                   = auto  #[yes / no], auto for yes
mintpy.topographicResidual.polyOrder         = auto  #[1-inf], auto for 2, poly order of temporal deformation model
mintpy.topographicResidual.phaseVelocity     = auto  #[yes / no], auto for no - phase, use phase velocity for minimization
mintpy.topographicResidual.stepFuncDate      = auto  #[20080529,20100611 / no], auto for no, date of step jump
mintpy.topographicResidual.excludeDate       = auto  #[20070321 / txtFile / no], auto for exclude_date.txt
mintpy.topographicResidual.pixelwiseGeometry = auto  #[yes / no], auto for yes, use pixel-wise geometry info

########## 9.1 residual_RMS (root mean squares for noise evaluation)
## Calculate the Root Mean Square (RMS) of residual phase time-series for each acquisition
## reference: Yunjun et al. (2019, section 4.9 and 5.4)
## To get rid of long wavelength component in space, a ramp is removed for each acquisition
## Set optimal reference date to date with min RMS
## Set exclude dates (outliers) to dates with RMS > cutoff * median RMS (Median Absolute Deviation)
mintpy.residualRMS.maskFile = auto  #[file name / no], auto for maskTempCoh.h5, mask for ramp estimation
mintpy.residualRMS.deramp   = auto  #[quadratic / linear / no], auto for quadratic
mintpy.residualRMS.cutoff   = auto  #[0.0-inf], auto for 3

########## 9.2 reference_date
## Reference all time-series to one date in time
## reference: Yunjun et al. (2019, section 4.9)
## no     - do not change the default reference date (1st date)
mintpy.reference.date = auto   #[reference_date.txt / 20090214 / no], auto for reference_date.txt

########## 10. velocity
## Estimate linear velocity and its standard deviation from time-series
## and from tropospheric delay file if exists.
## reference: Fattahi and Amelung (2015, JGR)
mintpy.velocity.excludeDate    = auto   #[exclude_date.txt / 20080520,20090817 / no], auto for exclude_date.txt
mintpy.velocity.startDate      = auto   #[20070101 / no], auto for no
mintpy.velocity.endDate        = auto   #[20101230 / no], auto for no

## Bootstrapping
## refernce: Efron and Tibshirani (1986, Stat. Sci.)
mintpy.velocity.bootstrap      = auto   #[yes / no], auto for no, use bootstrap
mintpy.velocity.bootstrapCount = auto   #[int>1], auto for 400, number of iterations for bootstrapping

########## 11.1 geocode (post-processing)
# for input dataset in radar coordinates only
# commonly used resolution in meters and in degrees (on equator)
# 100,         60,          50,          30,          20,          10
# 0.000925926, 0.000555556, 0.000462963, 0.000277778, 0.000185185, 0.000092593
mintpy.geocode              = auto  #[yes / no], auto for yes
mintpy.geocode.SNWE         = auto  #[-1.2,0.5,-92,-91 / none ], auto for none, output extent in degree
mintpy.geocode.laloStep     = auto  #[-0.000555556,0.000555556 / None], auto for None, output resolution in degree
mintpy.geocode.interpMethod = auto  #[nearest], auto for nearest, interpolation method
mintpy.geocode.fillValue    = auto  #[np.nan, 0, ...], auto for np.nan, fill value for outliers.

########## 11.2 google_earth (post-processing)
mintpy.save.kmz             = auto   #[yes / no], auto for yes, save geocoded velocity to Google Earth KMZ file

########## 11.3 hdfeos5 (post-processing)
mintpy.save.hdfEos5         = auto   #[yes / no], auto for no, save time-series to HDF-EOS5 format
mintpy.save.hdfEos5.update  = auto   #[yes / no], auto for no, put XXXXXXXX as endDate in output filename
mintpy.save.hdfEos5.subset  = auto   #[yes / no], auto for no, put subset range info   in output filename

########## 11.4 plot
mintpy.plot = auto   #[yes / no], auto for yes, plot files generated by default processing to pic folder

Full error message

2021-10-31 21-11-27屏幕截图

PASTE ERROR MESSAGE HERE

System information

welcome[bot] commented 2 years ago

👋 Thanks for opening your first issue here! Please make sure you filled out the template with as much detail as possible. We appreciate that you took the time to contribute! Make sure you read our contributing guidelines

zhuhaxixiong commented 2 years ago

version of mintpy :1.3.1 python :3.8

yunjunz commented 2 years ago

This seems like an ISCE-2 processing issue, you might get a better answer in their repo.

water mask is optional in both isce stack processing and mintpy, so I don't think that failure will be an issue.

The error message in mintpy reports the mismatch in the data size. I would recommend you to check that for all the files you specified in the mintpy.load.* options, and/or debug to locate the cause.

zhuhaxixiong commented 2 years ago

maybe it is a isce issue,i found that isce have a python shelve issue when focus alos 1.0 raw data to slc, no .dat 、.dir、*.bak but in conda environment these file generate ?and my mintpy does not installed by conda but by sudo apt install.

zhuhaxixiong commented 2 years ago

has been solved