search

aria-tools / ARIA-tools

Tools for exploiting ARIA standard products
Apache License 2.0
98 stars 36 forks source link

[BUG] ARIA stitched products give zero values #232

Closed rzinke closed 4 years ago

rzinke commented 4 years ago

I am following up on a bug I originally discovered using MintPy, and reported there: https://github.com/insarlab/MintPy/issues/453

However, indications point to the bug originating in the ARIA-tools workflow. Essentially, the binary image files are stored with all nodata values. I tested this for two Tibet tracks and both turned out this way.

dbekaert commented 4 years ago

@rzinke Can you provide additional information such other can replicate your problem? Also did you re-run this from an existing tun or a clean run?

The png that you showed looks to only provide partial coverage over the region of interest. The pngs are generated after the data is written to the envi file: https://github.com/aria-tools/ARIA-tools/blob/b787e827a799fecbf45d6aa267eeb95940148d83/tools/ARIAtools/unwrapStitching.py#L300 The files itself are not touched anymore afterwards, as the stack files are merely pointing to these files.

Can you investigate further if this is applicable to all the files; e.g. .coherence, connected components, phase, geometric files etc?

rzinke commented 4 years ago

Okay. It appears there are several things going on here:

First, I want to use a practice data set for troubleshooting. I downloaded the standard products:

ariaDownload.py -t 19 -s 20190101 -e 20190601

then launched the ARIA TS setup script: 

ariaTSsetup.py -f "products/*.nc" -b 19_A_Tibet_AOI.shp -l 3 -d download -m download -nt 8

which produces the following error:

NETCDF:"/u/sar-r2/rzinke/ARIAfix/T19/products/S1-GUNW-D-R-019-tops-20190526_20190514-001637-39721N_37749N-PP-22b6-v2_0_2.nc":/science/grids/data/connectedComponents is GDAL compatible
Generating: unwrappedPhase - [======================= 98% ====================> ] 20190601_20190508  385s /     7sNETCDF:"/u/sar-r2/rzinke/ARIAfix/T19/products/S1-GUNW-D-R-019-tops-20190601_20190508-001444-44029N_42063N-PP-1f59-v2_0_2.nc":/science/grids/data/unwrappedPhase is GDAL compatible
NETCDF:"/u/sar-r2/rzinke/ARIAfix/T19/products/S1-GUNW-D-R-019-tops-20190601_20190508-001444-44029N_42063N-PP-1f59-v2_0_2.nc":/science/grids/data/connectedComponents is GDAL compatible
NETCDF:"/u/sar-r2/rzinke/ARIAfix/T19/products/S1-GUNW-D-R-019-tops-20190601_20190508-001509-42540N_40573N-PP-6554-v2_0_2.nc":/science/grids/data/unwrappedPhase is GDAL compatible
NETCDF:"/u/sar-r2/rzinke/ARIAfix/T19/products/S1-GUNW-D-R-019-tops-20190601_20190508-001509-42540N_40573N-PP-6554-v2_0_2.nc":/science/grids/data/connectedComponents is GDAL compatible
NETCDF:"/u/sar-r2/rzinke/ARIAfix/T19/products/S1-GUNW-D-R-019-tops-20190601_20190508-001533-41050N_39247N-PP-3654-v2_0_2.nc":/science/grids/data/unwrappedPhase is GDAL compatible
NETCDF:"/u/sar-r2/rzinke/ARIAfix/T19/products/S1-GUNW-D-R-019-tops-20190601_20190508-001533-41050N_39247N-PP-3654-v2_0_2.nc":/science/grids/data/connectedComponents is GDAL compatible
Generating: unwrappedPhase - [==================================================] 20190601_20190520NETCDF:"/u/sar-r2/rzinke/ARIAfix/T19/products/S1-GUNW-D-R-019-tops-20190601_20190520-001444-44029N_42063N-PP-7f33-v2_0_2.nc":/science/grids/data/unwrappedPhase is GDAL compatible
NETCDF:"/u/sar-r2/rzinke/ARIAfix/T19/products/S1-GUNW-D-R-019-tops-20190601_20190520-001444-44029N_42063N-PP-7f33-v2_0_2.nc":/science/grids/data/connectedComponents is GDAL compatible
NETCDF:"/u/sar-r2/rzinke/ARIAfix/T19/products/S1-GUNW-D-R-019-tops-20190601_20190520-001509-42540N_40573N-PP-aa33-v2_0_2.nc":/science/grids/data/unwrappedPhase is GDAL compatible
NETCDF:"/u/sar-r2/rzinke/ARIAfix/T19/products/S1-GUNW-D-R-019-tops-20190601_20190520-001509-42540N_40573N-PP-aa33-v2_0_2.nc":/science/grids/data/connectedComponents is GDAL compatible
NETCDF:"/u/sar-r2/rzinke/ARIAfix/T19/products/S1-GUNW-D-R-019-tops-20190601_20190520-001533-41050N_39247N-PP-2007-v2_0_2.nc":/science/grids/data/unwrappedPhase is GDAL compatible
NETCDF:"/u/sar-r2/rzinke/ARIAfix/T19/products/S1-GUNW-D-R-019-tops-20190601_20190520-001533-41050N_39247N-PP-2007-v2_0_2.nc":/science/grids/data/connectedComponents is GDAL compatible

Generating: coherence - [==================================================] 20190601_20190520  112s /     2s

Extracting single incidence angle, look angle and azimuth angle files valid over common interferometric grid
Generating: incidenceAngle - [==================================================] 20190114_20190102
Generating: lookAngle - [==================================================] 20190114_20190102
Generating: azimuthAngle - [==================================================] 20190114_20190102

Extracting optional, user-specified layers ['3'] for each interferogram pair
Traceback (most recent call last):
  File "/u/sar-r0/rzinke/python/miniconda3/envs/ARIA-tools/bin/ariaTSsetup.py", line 15, in <module>
    main(inps)
  File "/u/sar-r0/rzinke/python/miniconda3/envs/ARIA-tools/lib/python3.8/site-packages/ARIAtools/tsSetup.py", line 295, in main
    export_products(standardproduct_info.products[1], standardproduct_info.bbox_file, prods_TOTbbox, layers, dem=demfile, lat=Latitude, lon=Longitude, mask=inps.mask, outDir=inps.workdir, outputFormat=inps.outputFormat, stitchMethodType='overlap', verbose=inps.verbose, num_threads=inps.num_threads, multilooking=inps.multilooking)
  File "/u/sar-r0/rzinke/python/miniconda3/envs/ARIA-tools/lib/python3.8/site-packages/ARIAtools/extractProduct.py", line 304, in export_products
    product_dict=[[j[key] for j in full_product_dict], [j["pair_name"] for j in full_product_dict]]
  File "/u/sar-r0/rzinke/python/miniconda3/envs/ARIA-tools/lib/python3.8/site-packages/ARIAtools/extractProduct.py", line 304, in <listcomp>
    product_dict=[[j[key] for j in full_product_dict], [j["pair_name"] for j in full_product_dict]]
KeyError: '3'
rzinke commented 4 years ago

@rzinke Can you provide additional information such other can replicate your problem? Also did you re-run this from an existing tun or a clean run?

The png that you showed looks to only provide partial coverage over the region of interest. The pngs are generated after the data is written to the envi file:

https://github.com/aria-tools/ARIA-tools/blob/b787e827a799fecbf45d6aa267eeb95940148d83/tools/ARIAtools/unwrapStitching.py#L300

The files itself are not touched anymore afterwards, as the stack files are merely pointing to these files.

Can you investigate further if this is applicable to all the files; e.g. .coherence, connected components, phase, geometric files etc?

To more directly answer your questions: I ran from a clean run. The png covering only part of the AOI is expected, since the frames currently do not cover the whole bbox area. The coherence maps show only zeros. The connected components outline the expected extent, same as the pngs:

Screen Shot 2020-10-05 at 12 53 25 PM

sssangha commented 4 years ago

@rzinke I believe there was a typo in your input? You specified '3' with the '-l' command, which is why the program was looking for layer key '3' in the dictionary

rzinke commented 4 years ago

Thanks, @sssangha. I fixed the typo and run the script. All the maps are still zeros. So, 

ariaTSsetup.py -f "products/*.nc" -b ShapeFiles/19_A_Tibet_AOI.shp -ml 3 -d download -m download -nt 8 -mo 10000

produces the following outputs:

Screen Shot 2020-10-05 at 4 07 30 PM

dbekaert commented 4 years ago

My first suggestion would be to simplify the issue and remove the -ml option.

On Mon, 5 Oct 2020 at 16:08, rzinke notifications@github.com wrote:

Thanks, @sssangha https://github.com/sssangha. I fixed the typo and run the script. All the maps are still zeros. So,

ariaTSsetup.py -f "products/*.nc" -b ShapeFiles/19_A_Tibet_AOI.shp -ml 3 -d download -m download -nt 8 -mo 10000

produces the following outputs:

[image: Screen Shot 2020-10-05 at 4 07 30 PM] https://user-images.githubusercontent.com/48299159/95140985-e6f58800-0724-11eb-9d7a-59ed07dba72e.png

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/aria-tools/ARIA-tools/issues/232#issuecomment-703936049, or unsubscribe https://github.com/notifications/unsubscribe-auth/AESZPSNK2YWBNG3KNNI4WNTSJJGVTANCNFSM4SE64IXQ .

rzinke commented 4 years ago

I had a brief chat with @sssangha. He thinks it has to do with the watermask. I don't have time to test this at this moment, but hopefully later tonight I can set up a job without the water mask to see that solves the issue.

rzinke commented 4 years ago

That is, here is the watermask from yesterday's run:

Screen Shot 2020-10-05 at 5 57 56 PM

And here is the watermask from last May, which returned valid results:

Screen Shot 2020-10-05 at 5 58 56 PM

I think Sim's intuition was correct. Re-running my data w/out a water mask to confirm.

dbekaert commented 4 years ago

@sssangha it might be good if we add more control plots like we do for unwrapped phase for the DEM and water mask too. Perhaps add a warning if the stat of the waster mask is all zero?

rzinke commented 4 years ago

I re-ran the functions without using a water mask at all. I still get only null values.

ariaTSsetup.py -f "products/*.nc" -b ShapeFiles/19_A_Tibet_AOI.shp -ml 3 -d download -nt 8 -mo 5000

returns the following results: Screen Shot 2020-10-06 at 10 04 50 AM

Where white indicates nodata. Here again, the pngs look fine:

20190309_20190213

rzinke commented 4 years ago

Here is the shapefile:

19_A_Tibet_AOI.zip

rzinke commented 4 years ago

I did the standard timeseries generation process for Track 41. Here, there are no AOI or minimum overlap specifications necessary.

ariaDownload.py -t 41 -s 20160101 -e 20160301

ariaTSsetup.py -f "products/*.nc" -ml 3 -d download -nt 8

The PNGs show valid ifgs, but the ENVI files are all zeros.