ASF HyP3 Support

[forrestfwilliams]

The ASF HyP3 on-demand InSAR service is great option for creating interferograms, but MintPy is not currently set up to ingest these products. I propose creating a prep_hyp3.py workflow that would allow for the ingestion of these products. I'm happy to perform this work, but could use guidance on preparing metadata.

[yunjunz]

Hi ASF team (@cirrusasf, @talogan, @asjohnston-asf, et al.), it seems like the "Heading" attribute in the example dataset that @cirrusasf put here: https://jzhu-hyp3-dev.s3.us-west-2.amazonaws.com/hyp3-mintpy/Ridgecrest.zip is not right: For Sentinel-1 descending track, it should be around -168, while the *.txt file shows 193.0934889.

Update: my miscalculation, the value is right and normal (with 360 in difference). Please ignore my above msg.

[asjohnston-asf]

FYI only, I created a sample data set for HyP3 using the same 98 S1 acquisitions (288 interferograms) as the Fernandina volcano data set for ARIA:

wget https://asj-hyp3-dev.s3.us-west-2.amazonaws.com/galapagos.zip
unzip galapagos.zip
cd galapagos
smallbaselineApp.py hyp3.cfg

Not intended to be hosted there long-term, but I've made it available for now in case anyone wants to look at a HyP3 time series for a familiar data set.

[yunjunz]

Close this issue as all the proposed changes are completed! Kudos to all!

[forrestfwilliams]

Happy to see this completed!

[jhkennedy]

:partying_face:

@yunjunz do you have plans for a release soon since a lot of this (and a lot more!) is main but not yet released? With a release we can get everything in the conda-forge install as well.

[yunjunz]

@jhkennedy yes, a release is planned for next month, for the exact reason you said.

There are a few things on the to-do list before that: https://github.com/insarlab/MintPy/wiki/To-do-list#version-132

[XJQiao]

Hi guys,

I am trying to play with Mintpy and Hyp3 on Google Colab these days.

I am using "!smallbaselineApp.py mintpy/RidgecrestSenDT71.txt", but got stuck at the step of "modify network" (as shown in the end). Any hints on this issue?

Appreciated!

Best, Jun

******************** step - modify_network ********************
WARNING: "coherence" is missing in file /content/RidgecrestSenDT71/inputs/ifgramStack.h5
Input data seems to be geocoded. Lookup file not needed.
generate /content/RidgecrestSenDT71/waterMask.h5 from /content/RidgecrestSenDT71/inputs/geometryGeo.h5 for conveniency
create HDF5 file: /content/RidgecrestSenDT71/waterMask.h5 with w mode
create dataset /waterMask of bool       in size of (1125, 1500)         with compression=None
finished writing to /content/RidgecrestSenDT71/waterMask.h5

modify_network.py /content/RidgecrestSenDT71/inputs/ifgramStack.h5 -t /content/RidgecrestSenDT71/smallbaselineApp.cfg
No lookup table info range/lat found in files.
read options from template file: smallbaselineApp.cfg
No input option found to remove interferogram
Keep all interferograms by enable --reset option
--------------------------------------------------
reset dataset 'dropIfgram' to True for all interferograms for file: /content/RidgecrestSenDT71/inputs/ifgramStack.h5
All dropIfgram are already True, no need to reset.

plot_network.py /content/RidgecrestSenDT71/inputs/ifgramStack.h5 -t /content/RidgecrestSenDT71/smallbaselineApp.cfg --nodisplay -d coherence -v 0.2 1.0
read options from template file: smallbaselineApp.cfg
read temporal/spatial baseline info from file: /content/RidgecrestSenDT71/inputs/ifgramStack.h5
calculating spatial mean of coherence in file /content/RidgecrestSenDT71/inputs/ifgramStack.h5 ...
read mask from file: waterMask.h5
Traceback (most recent call last):
  File "/usr/local/bin/smallbaselineApp.py", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/mintpy/smallbaselineApp.py", line 1270, in main
    app.run(steps=inps.runSteps)
  File "/usr/local/lib/python3.7/dist-packages/mintpy/smallbaselineApp.py", line 1038, in run
    self.run_network_modification(sname)
  File "/usr/local/lib/python3.7/dist-packages/mintpy/smallbaselineApp.py", line 434, in run_network_modification
    mintpy.plot_network.main(iargs)
  File "/usr/local/lib/python3.7/dist-packages/mintpy/plot_network.py", line 245, in main
    inps = read_network_info(inps)
  File "/usr/local/lib/python3.7/dist-packages/mintpy/plot_network.py", line 167, in read_network_info
    checkAoi=False)[0]
  File "/usr/local/lib/python3.7/dist-packages/mintpy/utils/utils1.py", line 256, in spatial_average
    threshold=threshold)
  File "/usr/local/lib/python3.7/dist-packages/mintpy/objects/stack.py", line 794, in spatial_average
    dset = f[datasetName]
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/usr/local/lib/python3.7/dist-packages/h5py/_hl/group.py", line 288, in __getitem__
    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 190, in h5py.h5o.open
KeyError: "Unable to open object (object 'coherence' doesn't exist)"

[forrestfwilliams]

It looks like your ifgramStack.h5 may have failed to load correctly and you don't have the required coherence dataset. Could you run info.py inputs/ifgramStack.h5 and paste the output here?

[XJQiao]

@forrestfwilliams Appreciated you!

After I rerun the code once again this morning, the bug was gone. I guess I passed a wrong path of input files, leading to no output file of ifgramStack.

Then, I encountered the next issue (attached in the end). I will look into this later, which seems due to no output file of avgPhaseVelocity.h5.

Have a nice day!

Best, Jun

******************** step - quick_overview ********************
Input data seems to be geocoded. Lookup file not needed.

temporal_average.py /content/RidgecrestSenDT71/inputs/ifgramStack.h5 --dataset unwrapPhase -o /content/RidgecrestSenDT71/avgPhaseVelocity.h5 --update
output file: /content/RidgecrestSenDT71/avgPhaseVelocity.h5
--------------------------------------------------
update mode: ON
1) output file /content/RidgecrestSenDT71/avgPhaseVelocity.h5 NOT exist.
run or skip: run.
calculate the temporal average of unwrapPhase in file /content/RidgecrestSenDT71/inputs/ifgramStack.h5 ...
Traceback (most recent call last):
  File "/usr/local/bin/smallbaselineApp.py", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/mintpy/smallbaselineApp.py", line 1270, in main
    app.run(steps=inps.runSteps)
  File "/usr/local/lib/python3.7/dist-packages/mintpy/smallbaselineApp.py", line 1044, in run
    self.run_quick_overview(sname)
  File "/usr/local/lib/python3.7/dist-packages/mintpy/smallbaselineApp.py", line 497, in run_quick_overview
    mintpy.temporal_average.main(iargs)
  File "/usr/local/lib/python3.7/dist-packages/mintpy/temporal_average.py", line 102, in main
    ut.temporal_average(inps.file, datasetName=inps.datasetName, outFile=inps.outfile)
  File "/usr/local/lib/python3.7/dist-packages/mintpy/utils/utils1.py", line 355, in temporal_average
    dataMean = ifgramStack(File).temporal_average(datasetName=datasetName)
  File "/usr/local/lib/python3.7/dist-packages/mintpy/objects/stack.py", line 933, in temporal_average
    ref_val = dset[ifgram_flag, self.refY, self.refX]
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/usr/local/lib/python3.7/dist-packages/h5py/_hl/dataset.py", line 777, in __getitem__
    selection = sel.select(self.shape, args, dataset=self)
  File "/usr/local/lib/python3.7/dist-packages/h5py/_hl/selections.py", line 82, in select
    return selector.make_selection(args)
  File "h5py/_selector.pyx", line 272, in h5py._selector.Selector.make_selection
  File "h5py/_selector.pyx", line 183, in h5py._selector.Selector.apply_args
TypeError: Indexing arrays must have integer dtypes