ASF HyP3 Support - Githubissues

forrestfwilliams commented 3 years ago

The ASF HyP3 on-demand InSAR service is great option for creating interferograms, but MintPy is not currently set up to ingest these products. I propose creating a prep_hyp3.py workflow that would allow for the ingestion of these products. I'm happy to perform this work, but could use guidance on preparing metadata.

forrestfwilliams commented 3 years ago

HyP3 product are projected to the local UTM zone. I imagine this could cause issues for the LAT/LON_START and LAT/LON_REF1/2/3/4 attributes. Can MintPy accommodate this, or must they all be reprojected?

yunjunz commented 3 years ago

No need for re-projection. MintPy should be able to handle X/Y_STEP = meters and its corresponding X/Y_FIRST.

Since most of the operations in mintpy do not care about coordinates, special attention to the coordinate system is only needed during geocoding and plotting in map coordinates, as far as I remember. Both of these two functionalities have been tested before, but not maintained via regular testing, so some functionalities may not work out of the box. But they should be fixed easily.

yunjunz commented 3 years ago

A HyP3 + MintPy support would be awesome @forrestfwilliams. I am not familiar with HyP3 results. Here is a general guideline to prepare datasets for MintPy: https://mintpy.readthedocs.io/en/latest/FAQs/. I am more than happy to answer any question you may have related to mintpy.

forrestfwilliams commented 3 years ago

Thanks @yunjunz! Currently HyP3 only provides the average perpendicular baseline, and not the baseline for the top and bottom of the interferogram. Should I see if I can get this data, or is it permissible to have the same average baseline for both P_BASELINE_TOP_HDR and P_BASELINE_BOTTOM_HDR?

yunjunz commented 3 years ago

Use the same value is perfectly fine.

forrestfwilliams commented 3 years ago

Hi @yunjunz, I've successfully created a prep_hyp3.py file that generates the needed rsc files for each interferogram. Hyp3 provides interferograms as tifs that can be viewed using view.py, but once an rsc file is generated for an interferogram, I believe that readfile.read assumes that the interferogram is now in a binary format, which causes an error and the message: "Unknown InSAR processor: hyp3". Do you have any thoughts on how to correct this?

forrestfwilliams commented 3 years ago

Disregard the previous comment, I was able to track down the error.

yunjunz commented 3 years ago

The discussion on the GMTSAR support (#452) could be a good reference here. We have implemented it here: https://github.com/yunjunz/MintPy/tree/gmtsar.

yunjunz commented 3 years ago

Here is the to-do list I could think of for full support of HyP3+MintPy workflow:

The 2D matrix of pixel-wised incidence angle [required for tropospheric correction and DEM error correction].
The connected components for each interferogram [optional but recommended; required for unwrapping error correctoin]
A 2D matrix of water/land mask [optional but recommended]
Add attributes: HEIGHT (satellite altitude in m) and EARTH_RADIUS, both of them are currently hardwired in the code here.

talogan commented 3 years ago

Thanks for the to-do list. We are going to start looking into this soon.

I note that the first link in this issue is to HyP3 version 1, which is our beta system. We've recently released our InSAR GAMMA process in HyP3 version 2, a more stable system that can be accessed through the ASF Data Search engine Vertex, through an API, or though a python SDK.

Documentation for HyP3 v2 is available at the HyP3 docs website.

forrestfwilliams commented 3 years ago

@talogan and @yunjunz, thank you for continuing this discussion! I'm looking forward to fully integrating HyP3 products into MintPy, and am happy to keep contributing as we continue this process.

yunjunz commented 3 years ago

Thank you @talogan for pointing out the link, it hasd been fixed in #545. Please feel free to add more detailed descriptions/tutorials/links via PRs. Both the example directory structure and the prep_hyp3.py help msg are good places for them.

talogan commented 3 years ago

ASF is working in the following:

we've added the spacecraft height and earth radius at the nadir to our metadata
we're adding the option to include an incidence angle map in the product bundle
we're adding the option to select water masking
we're adding the option to include the DEM into the product bundle

Unless I missed something, the only items we're missing are the connected components and the slant range distance. I have questions about these:

GAMMA's MCF phase unwrapping implementation doesn't mention connected components. Does that apply to MCF? I know that it does for snaphu, but I couldn't find any mention in the GAMMA documentation of connected components.
For the slant range distance, you need only the first slant range and the slant range spacing, I believe. Do you actually need a raster image, or would the start, [middle, end] slant range plus the range pixel spacing suffice? It seems to me that the slant range will be the same for every line in the SAR file.

yunjunz commented 3 years ago

You are right. There is no connected components product from GAMMA phase unwrapping, as far as I know.

As for the slant range distance, we could calculate it based on the 2D incidence angle, accurate spacecraft height and earth radius (https://github.com/insarlab/MintPy/blob/main/mintpy/utils/utils0.py#L139). So it's okay to not have it.

For further clarification, we are using the geocoded products from HyP3, so the pixel-wised 2D slant range distance can not be calculated/predicted from the few parameters. However, it will still be good to have those parameters available for other purposes.

talogan commented 3 years ago

As of today, the HyP3 InSAR products include the following (the options are not yet available in the Vertex, but are accessible via the other interfaces):

Spacecraft height, Earth radius, Slant range near, Slant range center, and Slant range far are now in the .txt metadata file
an option to include the incidence angle map into the product bundle now exists
an option to include the DEM file used during processing now exists.

We're still working on water masking.

One point of clarification @forrestfwilliams - would you prefer that the water mask be applied to the InSAR pairs when they are processed by HyP3, or would you prefer to just get a mask file and not have it already applied to the data?

yunjunz commented 3 years ago

This is great @talogan!

Just jump in for the water mask :)

I would think an option to download a water mask is definitely desired. This would be similar to the DEM and incidence angle.

A separate option in the processing to apply the water mask before phase unwrapping would be great too.

forrestfwilliams commented 3 years ago

Hi @talogan this is all great, thanks! Echoing @yunjunz, I think it would be great to have the water mask applied before unwrapping, but just receiving the water mask would be great as well!

forrestfwilliams commented 3 years ago

Hi @yunjunz and @talogan, I've taken a quick look at updating MintPy so that it takes full advantages of the updates to HyP3, but some other projects have taken priority and I won't be able to devote much time to this until June. Hope this isn't too long for you all!

yunjunz commented 3 years ago

Thank you for looking into this again @forrestfwilliams.

cirrusasf commented 3 years ago

I am trying to make the mintpy work against hyp3. @forrestfwilliams mentioned you have already did a prep_hyp3. May I take a look at it. I followed the guide to arrange the hyp3 output. When I run, I got stuck at step “correct_troposphere”. It complains that Exception: Assertion failed: bbox.north() <= Latitude::NORTH_POLE in check

, which comes from the statement result = self._api("%s/resources/%s" % (self.url, name), request, "POST"). The dem.tif and hyp3 tif files have the projection "WGS 84 / UTM zone 53N". Does this error means that we need longitude/latitude in degree?

yunjunz commented 3 years ago

Hi @cirrusasf, the error you show seems from the PyAPS code, I assume that you have successfully ingested incidenceAngle to geometryGeo.h5 file.

To your question, yes, PyAPS works on lat/lon only (tropo_pyaps3.py#L714-L716). So some changes are needed in tropo_paps3.py to prepare the pixelwised lat/lon. It's should straightforward to grab the UTM coordinates of the 4 corners, convert them to lat/lon and generate the pixelwised lat/lon from it.

cirrusasf commented 3 years ago

Hi Insarlab/Mintpy,

Thank you for your reply. I noticed that there is prep_hyp3.py. I want to play this first to see if I can make mintPy work against hyp3 data.

Jiang

On Tue, May 18, 2021 at 6:00 PM Zhang Yunjun @.***> wrote:

Hi @cirrusasf https://github.com/cirrusasf, the error you show seems from the PyAPS code, I assume that you have successfully ingested incidenceAngle to geometryGeo.h5 file.

To your question, yes, PyAPS works on lat/lon only ( tropo_pyaps3.py#L714-L716 https://github.com/insarlab/MintPy/blob/main/mintpy/tropo_pyaps3.py#L714-L716). So some changes are needed in tropo_paps3.py to prepare the pixelwised lat/lon. It's should straightforward to grab the UTM coordinates of the 4 corners, convert them to lat/lon and generate the pixelwised lat/lon from it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/insarlab/MintPy/issues/540#issuecomment-843686734, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO3CPWHUQIC2MMES4IK3SCDTOMLTPANCNFSM4ZT26Z7A .

yunjunz commented 3 years ago

@cirrusasf It's committed and available already.

cirrusasf commented 3 years ago

Hi Insarlab/Mintpy,

I got stuck at

pa.ECMWFdload(date_list2dload, hour, grib_dir, model=tropo_model, snwe=snwe, flist=grib_files2dload)

it can not download ERA5 data. Perhaps snake needs in degree, not in meter?

Jiang

On Wed, May 19, 2021 at 9:57 AM Zhang Yunjun @.***> wrote:

It's committed and available already.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/insarlab/MintPy/issues/540#issuecomment-844336929, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO3CPWBNO3AA2IHLUGLSBBLTOP3XTANCNFSM4ZT26Z7A .

cirrusasf commented 3 years ago

Hi Insarlab/Mintpy,

I try to make the mintpy work with hyp3 output data. I modifed the script, now I can download the ERA5 data. I got the next problem. It complains that it can not find the incangle, error messge: KeyError: "Unable to open object (object 'incidenceAngle' doesn't exist)". I produce the incangle.tif files, How do I configure the custom.cfg to add the incAngeleFile?

Thank you very much!

Jiang

On Wed, May 19, 2021 at 11:30 AM Jiang Zhu @.***> wrote:

Hi Insarlab/Mintpy,

I got stuck at

pa.ECMWFdload(date_list2dload, hour, grib_dir, model=tropo_model, snwe=snwe, flist=grib_files2dload)

it can not download ERA5 data. Perhaps snake needs in degree, not in meter?

Jiang

On Wed, May 19, 2021 at 9:57 AM Zhang Yunjun @.***> wrote:

It's committed and available already.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/insarlab/MintPy/issues/540#issuecomment-844336929, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO3CPWBNO3AA2IHLUGLSBBLTOP3XTANCNFSM4ZT26Z7A .

yunjunz commented 3 years ago

By setting mintpy.load.incAngleFile = {path_to_incangle.tif_file}.

cirrusasf commented 3 years ago

Hi Yunjun,

Thank you for your reply! I produce an incangle tif file for each interferogram, and I copy one such file into incangle.tif, and I put it in {DATA_DIR}. I edit my custom.txt file by adding mintpy.load.incAngleFile ={DATA_DIR}/incangle.tif. When I run smallbaselineApp.py custom.cfg, it complains that it needs inangle.txt, not inangle.tif. How do I create a metadata txt file named incangle.txt?

Jiang

The error is as follows: ... mintpy.load_data.main(iargs) File "/home/jzhu4/projects/work/hyp3/hyp3-mintpy/MintPy/mintpy/load_data.py", line 822, in main prepare_metadata(iDict) File "/home/jzhu4/projects/work/hyp3/hyp3-mintpy/MintPy/mintpy/load_data.py", line 651, in prepare_metadata prep_module.main(iargs) File "/home/jzhu4/projects/work/hyp3/hyp3-mintpy/MintPy/mintpy/prep_hyp3.py", line 213, in main meta = add_hyp3_metadata(fname, meta, is_ifg=False) File "/home/jzhu4/projects/work/hyp3/hyp3-mintpy/MintPy/mintpy/prep_hyp3.py", line 140, in add_hyp3_metadata with open(meta_file, 'r') as f: FileNotFoundError: [Errno 2] No such file or directory: '/media/jzhu4/data/hyp3-mintpy/case2/incangle.txt'

On Fri, May 21, 2021 at 5:50 PM Zhang Yunjun @.***> wrote:

By setting mintpy.load.incAngleFile = {path_to_incangle.tif_file}.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/insarlab/MintPy/issues/540#issuecomment-846330737, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO3CPWB4TU6QFHUBEPLTJCDTO4EVLANCNFSM4ZT26Z7A .

yunjunz commented 3 years ago

@cirrusasf Could you please 1) add some information on creating the incidence angle file in HyP3 and 2) update the example template option for loading the incidence angle file in the example dir structure page? I believe that would answer the question in the user forum (https://groups.google.com/g/mintpy/c/FSdi1PQjnEU).

cirrusasf commented 3 years ago

Hi Yunjun,

Thank you for adopting our code. Yes, We (ASF) would like to provide an example. We also want to provide a notebook for tutorial purposes.

Jiang

On Wed, Jun 2, 2021 at 9:54 PM Zhang Yunjun @.***> wrote:

@cirrusasf https://github.com/cirrusasf Could you please 1) add some information on creating the incidence angle file in HyP3 and 2) update the example template option for loading the incidence angle file in the example dir structure page https://mintpy.readthedocs.io/en/latest/dir_structure/#asf_hyp3? I believe that would answer the question in the user forum ( https://groups.google.com/g/mintpy/c/FSdi1PQjnEU).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/insarlab/MintPy/issues/540#issuecomment-853586207, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO3CPWG2GAKNZBFHX6IO3XTTQ4KI7ANCNFSM4ZT26Z7A .

yunjunz commented 3 years ago

Hi @talogan, from a quick look at https://hyp3-docs.asf.alaska.edu/guides/insar_product_guide/#processing-options (great descriptions by the way), I think there may be a confusion on the incidence angle: for tropospheric correction, we need the simple incidence angle of the LOS vector with respect to the zenith/verical direction, WITHOUT considering the slope of the local topography.

talogan commented 3 years ago

@yunjunz The Hyp3 development team is discussing how we will fix this issue. Most likely, we will either replace the local incidence angle file currently produced with the ellipsoidal incidence angle file you requested, or offer both the local and ellipsoidal incidence angle files.

cirrusasf commented 3 years ago

Hi Insarlab/Mintpy,

I can run smallbaselineApp.py successfully against the ASF HypP3 product with the following custom.cfg. The samllbaselineApp.py includes the tropospheric correction. Does this mean that the program automatically correctly get the LOS vector information from the *inc_map_clip.tif?

The content of the custom.cfg mintpy.load.processor = hyp3

---------interferogram datasets:

mintpy.load.unwFile = /media/jzhu4/data/hyp3-mintpy/case4//unw_phase_clip.tif mintpy.load.corFile = /media/jzhu4/data/hyp3-mintpy/case4//corr_clip.tif

---------geometry datasets:

mintpy.load.demFile = /media/jzhu4/data/hyp3-mintpy/case4/DEM/dem_clip.tif

---------incAngleFile:

mintpy.load.incAngleFile = /media/jzhu4/data/hyp3-mintpy/case4//inc_map_clip.tif

Jiang

On Wed, Jun 9, 2021 at 11:53 PM Tom Logan @.***> wrote:

@yunjunz https://github.com/yunjunz The Hyp3 development team is discussing how we will fix this issue. Most likely, we will either replace the local incidence angle file currently produced with the ellipsoidal incidence angle file you requested, or offer both the local and ellipsoidal incidence angle files.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/insarlab/MintPy/issues/540#issuecomment-858398977, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO3CPWFOOTKNNJYWXJB7UCLTSBVP3ANCNFSM4ZT26Z7A .

yunjunz commented 3 years ago

Hi @cirrusasf, the code will read and use the local incidence angle data and run without error, but it's still not correct (there is no function in mintpy to detect and convert local incidence angle to ellipsoid referenced incidence angle). I would imagine a wrong tropospheric correction from it.

cirrusasf commented 3 years ago

Hi Insarlab/Mintpy,

Looks like we need to produce the incidence angle of the LOS vector with respect to the zenith/vertical direction. I will discuss it with Tom. By the way, we are creating the notebook and modifying some words in the example_directory. After we are ready, I will push another PR.

Thank you,

Jiang

On Thu, Jun 10, 2021 at 9:37 AM Zhang Yunjun @.***> wrote:

Hi @cirrusasf https://github.com/cirrusasf, the code will read and use the local incidence angle data and run without error, but it's still not correct (there is no function in mintpy to detect and convert local incidence angle to ellipsoid referenced incidence angle). I would imagine a wrong tropospheric correction from it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/insarlab/MintPy/issues/540#issuecomment-858825130, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO3CPWBCWXYDFHYHNKTN7Z3TSDZ35ANCNFSM4ZT26Z7A .

yunjunz commented 3 years ago

Sounds great @cirrusasf. A jupyter notebook would be very helpful I believe, please push it to MintPy-tutorial repo, it's where the other notebooks are located as well.

cirrusasf commented 3 years ago

Hi Yunjun,

Ok. will do it.

Jiang

On Thu, Jun 10, 2021 at 4:12 PM Zhang Yunjun @.***> wrote:

Sounds great @cirrusasf https://github.com/cirrusasf. For jupyter notebooks, please push to MintPy-tutorial https://github.com/insarlab/MintPy-tutorial repo.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/insarlab/MintPy/issues/540#issuecomment-859167961, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO3CPWGHD6RMA5WUH5OK5ZLTSFIFBANCNFSM4ZT26Z7A .

cirrusasf commented 3 years ago

Hi Insarlab/Mintpy,

Can you tell what is the format of the inc_angle file. Does it include near and far inc angle with zenith/vertical? That is to say it should include 4 values: near_inc_angle_zenith, near_inc_angle_vertical, far_inc_angle_zenith, and far_inc_angle_vertical. Am I correct? We can output these values as a file. We just want to know the format of the file.

Thank you,

Jiang

On Mon, Jun 7, 2021 at 11:40 AM Zhang Yunjun @.***> wrote:

Hi @talogan https://github.com/talogan, from a quick look at https://hyp3-docs.asf.alaska.edu/guides/insar_product_guide/#processing-options (great descriptions by the way), I think there may be a confusion on the incidence angle: for tropospheric correction, we need the simple incidence angle of the LOS vector with respect to the zenith/verical direction, WITHOUT considering the slope of the local topography.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/insarlab/MintPy/issues/540#issuecomment-856204753, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO3CPWGI3LNW2Q57WSHHV4DTRUOA7ANCNFSM4ZT26Z7A .

yunjunz commented 3 years ago

It should be a pixel-wise incidence angle, such as the example below from the SanFranSenDT42 example dataset from ARIA. inc_angle

If you could consider the impact of elevation for each pixel, such as the example below from the FernandinaSenDT128 example dataset from ISCE/topsStack, that would be even better. inc_angle2

forrestfwilliams commented 3 years ago

@yunjunz, @talogan, @cirrusasf, the new version of prep_hyp3.py will correctly read any HyP3 tif file that begins with the HyP3 naming convention S1xy-aaaaaaaaTbbbbbb_ggggggggThhhhhh_pponnn_INTzz_u_def_ssss and ends with .tif. As long as future water mask/incidence angle/connected components files stick to this formatting, we shouldn't have to make any major changes!

cirrusasf commented 3 years ago

Hi Insarlab/Mintpy,

I am a little bit confused. The inc_map.tif that the hyp3 produced is the pixel-wise inc angle tif file. Does mintpy need the inc_angle tif files from every ifm? I put the following in the custom.cfg

---------incAngleFile:

mintpy.load.incAngleFile = /media/jzhu4/data/hyp3-mintpy/case2//inc_map_clip.tif

the smallbaselineApp can produce the output. You told me our inc_map.tif was wrong. We will check what exactly the inc angle is defined in our inc_map.tif.

Jiang

On Thu, Jun 10, 2021 at 4:44 PM Zhang Yunjun @.***> wrote:

It should be a pixel-wise incidence angle, such as the example below from the SanFranSenDT42 example dataset from ARIA https://github.com/insarlab/MintPy/blob/main/docs/demo_dataset.md#sentinel-1-on-san-francisco-bay-with-aria . [image: inc_angle] https://user-images.githubusercontent.com/13143710/121613806-30b01400-ca12-11eb-98ed-300aa54ea025.png

If you could consider the impact of elevation for each pixel, such as the example below from the FernandinaSenDT128 example dataset from ISCE/topsStack https://github.com/insarlab/MintPy/blob/main/docs/demo_dataset.md#sentinel-1-on-fernandina-with-isce, that would be even better. [image: inc_angle2] https://user-images.githubusercontent.com/13143710/121614376-61447d80-ca13-11eb-941c-e7a8cb9b873a.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/insarlab/MintPy/issues/540#issuecomment-859179202, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO3CPWAGOEQSGIGW4R24JATTSFL53ANCNFSM4ZT26Z7A .

yunjunz commented 3 years ago

There is a good description of the definition of incidence angle from UNAVCO: https://www.unavco.org/education/professional-development/short-courses/course-materials/insar/2008-insar-course-materials/sar_summary.pdf.

cirrusasf commented 3 years ago

Hi Insarlab/Mintpy,

I pushed the PR https://github.com/insarlab/MintPy-tutorial/pull/22. it is about the notebook we created to demo how the mintpy work with hyp3 data. Please take a look at it.

Thank you,

Jiang

On Thu, Jun 10, 2021 at 4:12 PM Zhang Yunjun @.***> wrote:

Sounds great @cirrusasf https://github.com/cirrusasf. For jupyter notebooks, please push to MintPy-tutorial https://github.com/insarlab/MintPy-tutorial repo.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/insarlab/MintPy/issues/540#issuecomment-859167961, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO3CPWGHD6RMA5WUH5OK5ZLTSFIFBANCNFSM4ZT26Z7A .

cirrusasf commented 3 years ago

Can anybody in MintPy team take a look at our PR insarlab/MintPy-tutorial#22? This gives a tutorial to run mintpy with HyP3 data.

rbiessel commented 3 years ago

Maybe this deserves its own issue, but there are a lot of instances where MintPy assumes a WGS84 projection regardless of the projection of the input files. For instance the gecoded outputs (Gtiff, kmz, etc) do not work correctly because they use the UTM zone's geotransform with EPSG 4326. In addition, the GPS stations will not load and the input reference point latlon is not converted. My current hack-around is to just reproject all of the HyP3 tiffs to lat/lon before running mintpy.

cirrusasf commented 3 years ago

Hi Rowan Biessel,

I am a little bit confused about what you talked about. Can you point out what the exact problem is. For example, which part of the program can not go through? What does the output giff look like?

Jiang

On Mon, Jun 21, 2021 at 11:48 AM Rowan Biessel @.***> wrote:

Maybe this deserves its own issue, but there are a lot of instances where MintpY assumes a WGS84 projection regardless of the projection of the input files. For instance the gecoded outputs (Gtiff, kmz, etc) do not work correctly because they use the UTM zone's geotransform with EPSG 4326. My current hack-around is to just reproject all of the HyP3 tiffs to lat/lon before running mintpy.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/insarlab/MintPy/issues/540#issuecomment-865298803, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO3CPWCWJBICUSWKV42UBQLTT6JPXANCNFSM4ZT26Z7A .

rbiessel commented 3 years ago

Hi Rowan Biessel, I am a little bit confused about what you talked about. Can you point out what the exact problem is. For example, which part of the program can not go through? What does the output giff look like? Jiang … On Mon, Jun 21, 2021 at 11:48 AM Rowan Biessel @.***> wrote: Maybe this deserves its own issue, but there are a lot of instances where MintpY assumes a WGS84 projection regardless of the projection of the input files. For instance the gecoded outputs (Gtiff, kmz, etc) do not work correctly because they use the UTM zone's geotransform with EPSG 4326. My current hack-around is to just reproject all of the HyP3 tiffs to lat/lon before running mintpy. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#540 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO3CPWCWJBICUSWKV42UBQLTT6JPXANCNFSM4ZT26Z7A .

A command such as save_gdal.py velocity.h5 --of Gtiff save_gdal will set the projection of the output geotiff as EPSG:4326 no matter what. Meanwhile, the geotransform associated with velocity.h5 will be in UTMs. As a result, the geotiff won't render at all in QGIS. save_gdal.py needs to read the UTM zone from velocity.h5's attributes and use that instead of 4326.

For example, something like this:

from pyproj import CRS

try:
  zone = h5_file.attrs['UTM_ZONE']
  n_or_s = 'north' if 'N' in zone else 'south'
  zone_n = zone.replace('N', '').replace('S', '')
  crs = CRS.from_string(f'+proj=utm +zone={zone_n} +{n_or_s}')
  projection = crs.to_authority()[1]
except:
  projection = 4326 # If the interferograms are  in longitude/latitude already, the UTM_ZONE attribute won't exist 

...

outRasterSRS = osr.SpatialReference()
outRasterSRS.ImportFromEPSG(int(projection))
outRaster.SetProjection(outRasterSRS.ExportToWkt())

...

It is for sure easier to just reproject HyP3 products to EPSG:4326 before running MintPy

cirrusasf commented 3 years ago

Hi Rowan,

Thank you for clarification. I suggest modifying the save_gdal.py or other conversion/display/save scripts to make sure they can intake data with geo projection other than EPSG:4326. I don't suggest converting ifm files and then ingesting them to MintPy.

Jiang

On Mon, Jun 21, 2021 at 2:46 PM Rowan Biessel @.***> wrote:

Hi Rowan Biessel, I am a little bit confused about what you talked about. Can you point out what the exact problem is. For example, which part of the program can not go through? What does the output giff look like? Jiang … <#m-2245035496201012529> On Mon, Jun 21, 2021 at 11:48 AM Rowan Biessel @.***> wrote: Maybe this deserves its own issue, but there are a lot of instances where MintpY assumes a WGS84 projection regardless of the projection of the input files. For instance the gecoded outputs (Gtiff, kmz, etc) do not work correctly because they use the UTM zone's geotransform with EPSG 4326. My current hack-around is to just reproject all of the HyP3 tiffs to lat/lon before running mintpy. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#540 (comment) https://github.com/insarlab/MintPy/issues/540#issuecomment-865298803>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO3CPWCWJBICUSWKV42UBQLTT6JPXANCNFSM4ZT26Z7A .

A command such as save_gdal.py velocity.h5 --of Gtiff save_gdal will set the projection of the output geotiff as EPSG:4326 no matter what. Meanwhile, the geotransform associated with velocity.h5 will be in UTMs. save_gdal.py needs to read the UTM zone from velocity.h5's attributes and use that instead of 4326.

For example, something like this:

from pyproj import CRS

zone = h5_file.attrs['UTM_ZONE']

n_or_s = 'north' if 'N' in zone else 'south'

zone_n = zone.replace('N', '').replace('S', '')

crs = CRS.from_string(f'+proj=utm +zone={zone_n} +{n_or_s}')

projection = crs.to_authority()[1]

...

outRasterSRS = osr.SpatialReference()

outRasterSRS.ImportFromEPSG(int(projection))

outRaster.SetProjection(outRasterSRS.ExportToWkt())

...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/insarlab/MintPy/issues/540#issuecomment-865392847, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO3CPWF6MG736DCX2EESABDTT66NZANCNFSM4ZT26Z7A .

cirrusasf commented 3 years ago

Hi Jianbao,

I believe that new mintpy version includes the fix that I provided. If you grasp the newest version of mintpy. It will works with hyp3 and download the ERA correctly.

Jiang

On Sat, Jul 10, 2021 at 7:48 AM JianbaoSun @.***> wrote:

Hi, Jiang, I got the same ERA5 download issue when using HyP3 data. Mintpy works well when GAM is not used for tropospheric corrections. Could you please let me know what is your final solution? Thanks!

Sun

WARNING: downloading failed for 3 times, stop trying and continue.

update mode: ON output file: inputs/ERA5.h5

output file either do NOT exist or is NOT newer than all GRIB files. run or skip: run open geometry file: geometryGeo.h5 reading incidenceAngle data from file: inputs/geometryGeo.h5 ... reading height data from file: inputs/geometryGeo.h5 ...

create HDF5 file: inputs/ERA5.h5 with w mode create dataset : date of |S1 in size of (0,) with compression = None create dataset : timeseries of <class 'numpy.float32'> in size of (0, 953, 1202) with compression = None close HDF5 file: inputs/ERA5.h5

calculating absolute delay for each date using PyAPS (Jolivet et al., 2011; 2014) ... number of grib files used: 0 Traceback (most recent call last): File "/home/sun/MintPy_tools/MintPy/mintpy/tropo_pyaps3.py", line 821, in main(sys.argv[1:]) File "/home/sun/MintPy_tools/MintPy/mintpy/tropo_pyaps3.py", line 793, in main calc_delay_timeseries(inps) File "/home/sun/MintPy_tools/MintPy/mintpy/tropo_pyaps3.py", line 716, in calc_delay_timeseries prog_bar = ptime.progressBar(maxValue=num_date, print_msg=~inps.verbose) File "/home/sun/MintPy_tools/MintPy/mintpy/objects/progress.py", line 53, in init self.reset() File "/home/sun/MintPy_tools/MintPy/mintpy/objects/progress.py", line 58, in reset self.update_amount(0) # Build progress bar string File "/home/sun/MintPy_tools/MintPy/mintpy/objects/progress.py", line 74, in update_amount percentDone = (diffFromMin / np.float(self.span)) * 100.0 ZeroDivisionError: float division by zero

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/insarlab/MintPy/issues/540#issuecomment-877658740, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO3CPWF6TR3IXSLH4QT3HLTTXBTUNANCNFSM4ZT26Z7A .

asjohnston-asf commented 3 years ago

@yunjunz Regarding incidence angles, after more review, the lv_theta.tif GeoTIFF we offer via the include_look_vectors option may be sufficient. That GeoTIFF is generated using GAMMA's look_vector utility:

look_vector calculates SAR look-vector direction (pointing towards SAR) in map geometry. Based on the SAR imaging and DEM geometries the look vector is calculated for each pixel of the DEM/map section specified in the DEM parameter file. The local topography is taken into account. The look vector elevation angle is defined as the angle between the horizontal surface and the look vector with positive angles indicating sensor positions above the surface.

Elevation is already taken into account, but our lv_theta angle is measuring up from horizontal, rather than down from vertical. I believe a simple pi/2 - lv_theta conversion would give the angle desired by MintPy. Is there a convenient way for MintPy to perform that conversion? Would you prefer ASF include an additional GeoTIFF with the conversion already applied?

We also notice that the impact of elevation on the angle map in your Galapagos example is significantly greater than when we see from HyP3 products over the same area. I'll post an example HyP3 product shortly.

yunjunz commented 3 years ago

@asjohnston-asf the conversion could be applied in mintpy at stackDict.py#L619 since it already handles the radian to degree conversion.

insarlab / MintPy

ASF HyP3 Support #540

---------interferogram datasets:

---------geometry datasets:

---------incAngleFile:

---------incAngleFile:

Sun

WARNING: downloading failed for 3 times, stop trying and continue.

create HDF5 file: inputs/ERA5.h5 with w mode create dataset : date of |S1 in size of (0,) with compression = None create dataset : timeseries of <class 'numpy.float32'> in size of (0, 953, 1202) with compression = None close HDF5 file: inputs/ERA5.h5