Processor and gdal for ARIA products

rzinke commented 3 years ago

Description of the problem I am testing the new release of MintPy for large data set processing.

So, I copied an old data set to a new directory, and attempted to processes it using the updated version of MintPy and my old setup scripts.

I first got an error that it does not recognize the processor, so I manually set the processor to aria. Now, I am getting an error that gdal is not installed.

******************** step - load_data ********************

load_data.py --template /u/sar-r2/rzinke/Tibet/T121_FullTrack_aria/121_B_Test/ML3/smallbaselineApp.cfg /u/sar-r2/rzinke/Tibet/T121_FullTrack_aria/121_B_Test/ML3/Parameters121B.txt --project Parameters121B
processor : aria
SAR platform/sensor : unknown from project name "Parameters121B"
--------------------------------------------------
prepare metadata files for aria products
Traceback (most recent call last):
  File "/u/sarh0/rzinke/tools/MintPy/mintpy/prep_aria.py", line 18, in <module>
    from osgeo import gdal
ModuleNotFoundError: No module named 'osgeo'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/u/sarh0/rzinke/tools/MintPy/mintpy/smallbaselineApp.py", line 1182, in <module>
    main(sys.argv[1:])
  File "/u/sarh0/rzinke/tools/MintPy/mintpy/smallbaselineApp.py", line 1172, in main
    app.run(steps=inps.runSteps, plot=inps.plot)
  File "/u/sarh0/rzinke/tools/MintPy/mintpy/smallbaselineApp.py", line 1098, in run
    self.run_load_data(sname)
  File "/u/sarh0/rzinke/tools/MintPy/mintpy/smallbaselineApp.py", line 364, in run_load_data
    mintpy.load_data.main(iargs)
  File "/u/sarh0/rzinke/tools/MintPy/mintpy/load_data.py", line 749, in main
    prepare_metadata(iDict)
  File "/u/sarh0/rzinke/tools/MintPy/mintpy/load_data.py", line 655, in prepare_metadata
    from mintpy import prep_aria
  File "/u/sarh0/rzinke/tools/MintPy/mintpy/prep_aria.py", line 20, in <module>
    raise ImportError('Can not import gdal [version>=3.0]!')
ImportError: Can not import gdal [version>=3.0]!

System information

Operating system: Linux-CentOS
Python environment: conda
Version of MintPy: MintPy release version v1.2.3-132, release date 2021-01-26

Your custom / default template file (if the bug is related to a specific dataset):


### Processor
intpy.load.processor = aria  #[isce, aria, snap, gamma, roipac], auto for isce

Exclude pairs

mintpy.network.excludeIfgIndex = 39,40,31,48,111,85,87,88,187,188,282,67

Reference lat/lon

mintpy.reference.lalo = 37.20219,89.06214

Inversion weighting

mintpy.networkInversion.maskDataset = no #[coherence / connectComponent / offsetSNR / no], auto for no mintpy.networkInversion.maskThreshold = 0.4 #[0-inf], auto for 0.4

Tropospheric delay

mintpy.troposphericDelay.method = no #[pyaps / height_correlation / no], auto for pyaps

Orbital ramps

mintpy.deramp = no #[no / linear / quadratic]

Topography correction

mintpy.topographicResidual = no #[yes / no], auto for yes

rzinke commented 3 years ago

Potential solution I added this to the conda_env.yml file:

  - gdal >=3.0,<=3.0.4

Submitting a pull request

yunjunz commented 3 years ago

Hi @rzinke, gdal is a big dependency and is only used in data preparation for aria and fringe, but not needed for other InSAR processors, thus, we did not add gdal to the default installation.

I would recommend creating your own environment.yml file to cover the dependencies of both ARIA-tools and MintPy to install them into one conda environment.

rzinke commented 3 years ago

Hi @yunjunz. Fair enough. Gdal is cantankerous to say the least, but what about the error message that MintPy throws? It seems that MintPy requires gdal to process ARIA data successfully.

yunjunz commented 3 years ago

mintpy uses gdal to read files produced by ARIA-tools in prep_aria.py, which is called by load_data.py and smallbaselineApp.py.

So for ARIA+MintPy, users need gdal + mintpy requirements; for ISCE+MintPy, users need isce + mintpy requirements; for GAMMA + MintPy, users need gamma + mintpy requirements, etc.

rzinke commented 3 years ago

I see, that makes sense. Thanks.

yunjunz commented 3 years ago

Shall we close this issue and related PR?

insarlab / MintPy