search

aria-tools / ARIA-tools

Tools for exploiting ARIA standard products
Apache License 2.0
98 stars 36 forks source link

incidence, azimuth and look angle files are extraction problem #210

Closed ehavazli closed 4 years ago

ehavazli commented 4 years ago

ariaTSsetup.py doesn't fail but the extracted incidence, azimuth and look angle files are empty even though the file sizes are more than 10MB.

To Reproduce

ariaDownload.py --bbox "18.8 20.3 -156.1 -154.8" -t 124 --start 20181101 --end 20190120
ariaTSsetup.py -f 'products/*.nc' -b "18.8 20.3 -156.1 -154.8" --mask Download --dem Download -mo 20386.82310276927 --verbose

***Time-series Preparation Function:***
Shapefile ./user_bbox.json created for input user bounds.
Multi-core version
All (14) GUNW products meet spatial bbox criteria.
Group GUNW products into spatiotemporally continuous interferograms.
All (9) interferograms are spatially continuous.
Thread count specified for gdal multiprocessing = 2
4 out of 9 interferograms rejected for not meeting specified spatial thresholds
Download/cropping DEM

Extracting unwrapped phase, coherence, and connected components for each interferogram pair
Generating: unwrappedPhase - [==================================================] 20190115_20190103    0s /     0s
Generating: coherence - [==================================================] 20190115_20190103    1s /     0s

Extracting single incidence angle, look angle and azimuth angle files valid over common interferometric grid
Generating: incidenceAngle - [==================================================] 20181210_20181204/Users/havazli/miniconda3/envs/aria/lib/python3.8/importlib/__init__.py:127: MatplotlibDeprecationWarning: 
The matplotlib.backends.backend_qt4agg backend was deprecated in Matplotlib 3.3 and will be removed two minor releases later.
  return _bootstrap._gcd_import(name[level:], package, level)

Generating: lookAngle - [==================================================] 20181210_20181204 
Generating: azimuthAngle - [==================================================] 20181210_20181204 
Directory ./stack already exists.
Number of coherence discovered:  5
[==================================================] 20181210_20181204    0s /     0s
cohStack : stack generated
Directory ./stack already exists.
Number of connectedComponents discovered:  5
[==================================================] 20181210_20181204    0s /     0s
connCompStack : stack generated
Directory ./stack already exists.
Number of unwrapped interferograms discovered:  5
[==================================================] 20181210_20181204    0s /     0s
unwrapStack : stack generated

gdalinfo incidenceAngle/20181210_20181204 -stats

Driver: ENVI/ENVI .hdr Labelled
Files: incidenceAngle/20181210_20181204
       incidenceAngle/20181210_20181204.aux.xml
       incidenceAngle/20181210_20181204.hdr
Size is 1561, 1801
Coordinate System is:
GEOGCRS["GCS_unknown",
    DATUM["D_unnamed",
        ELLIPSOID["Spheroid",6378137,298.257223563,
            LENGTHUNIT["metre",1,
                ID["EPSG",9001]]]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["Degree",0.0174532925199433]],
    CS[ellipsoidal,2],
        AXIS["longitude",east,
            ORDER[1],
            ANGLEUNIT["Degree",0.0174532925199433]],
        AXIS["latitude",north,
            ORDER[2],
            ANGLEUNIT["Degree",0.0174532925199433]]]
Data axis to CRS axis mapping: 1,2
Origin = (-156.099999999999994,20.300833333333301)
Pixel Size = (0.000833333333333,-0.000833333333333)
Metadata:
  Band_1=Band 1
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (-156.1000000,  20.3008333) (156d 6' 0.00"W, 20d18' 3.00"N)
Lower Left  (-156.1000000,  18.8000000) (156d 6' 0.00"W, 18d48' 0.00"N)
Upper Right (-154.7991667,  20.3008333) (154d47'57.00"W, 20d18' 3.00"N)
Lower Right (-154.7991667,  18.8000000) (154d47'57.00"W, 18d48' 0.00"N)
Center      (-155.4495833,  19.5504167) (155d26'58.50"W, 19d33' 1.50"N)
Band 1 Block=1561x1 Type=Float32, ColorInterp=Undefined
ERROR 1: incidenceAngle/20181210_20181204, band 1: Failed to compute statistics, no valid pixels found in sampling.
  NoData Value=0
  Metadata:
    STATISTICS_VALID_PERCENT=0

Desktop (please complete the following information):

sssangha commented 4 years ago

@ehavazli I couldn't replicate this error.

gdalinfo incidenceAngle/20181110_20181104 -stats
Driver: ISCE/ISCE raster
Files: incidenceAngle/20181110_20181104
       incidenceAngle/20181110_20181104.aux.xml
       incidenceAngle/20181110_20181104.xml
Size is 1561, 1801
Coordinate System is:
GEOGCRS["unknown",
    DATUM["unnamed",
        ELLIPSOID["Spheroid",6378137,298.257223563,
            LENGTHUNIT["metre",1,
                ID["EPSG",9001]]]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["degree",0.0174532925199433,
            ID["EPSG",9122]]],
    CS[ellipsoidal,2],
        AXIS["latitude",north,
            ORDER[1],
            ANGLEUNIT["degree",0.0174532925199433,
                ID["EPSG",9122]]],
        AXIS["longitude",east,
            ORDER[2],
            ANGLEUNIT["degree",0.0174532925199433,
                ID["EPSG",9122]]]]
Data axis to CRS axis mapping: 2,1
Origin = (-156.100833333332986,20.300833333333301)
Pixel Size = (0.000833333333333,-0.000833333333333)
Corner Coordinates:
Upper Left  (-156.1008333,  20.3008333) (156d 6' 3.00"W, 20d18' 3.00"N)
Lower Left  (-156.1008333,  18.8000000) (156d 6' 3.00"W, 18d48' 0.00"N)
Upper Right (-154.8000000,  20.3008333) (154d48' 0.00"W, 20d18' 3.00"N)
Lower Right (-154.8000000,  18.8000000) (154d48' 0.00"W, 18d48' 0.00"N)
Center      (-155.4504167,  19.5504167) (155d27' 1.50"W, 19d33' 1.50"N)
Band 1 Block=1561x1 Type=Float32, ColorInterp=Undefined
  Minimum=36.794, Maximum=43.929, Mean=39.946, StdDev=1.647
  NoData Value=0
  Metadata:
    STATISTICS_MAXIMUM=43.929100036621
    STATISTICS_MEAN=39.946399452252
    STATISTICS_MINIMUM=36.793869018555
    STATISTICS_STDDEV=1.6473984439886
    STATISTICS_VALID_PERCENT=46.4

Perhaps some products are corrupted? I should also note that the reference interferograms chosen to extract metadata fields--always the first in the stack--for your case was 20181210_20181204, while for me it was 20181110_20181104 after following your download instructions. Can you confirm why this earlier product wasn't used as a reference on your end?

Also, it looks like you haven't checked-out the most recent revision, but that shouldn't affect the run in this way.

rzinke commented 4 years ago

Same as @sssangha, I do not get the same error. When I copy the commands exactly from the Reproduce section, I get the following error:

(ARIA-tools) [rzinke@sar T124]$ gdalinfo incidenceAngle/20181210_20181204 -stats ERROR 4: incidenceAngle/20181210_20181204: No such file or directory gdalinfo failed - unable to open 'incidenceAngle/20181210_20181204'

meaning that ARIA-tools is using slightly different files from you. It appears that AT is tossing out some duplicated files or has added new ones during download. Instead, AT generates incidence angle for 20181110_20181104, which has valid stats. Exactly like Sim.

Could you rerun the commands in a new folder and see if that works?

ehavazli commented 4 years ago

@sssangha @rzinke I was making multiple tests with different time extents to avoid possible corrupt products just like you mentioned. The discrepancy between the start date and primary date should come from me pasting the output of one of those tries.

I haven't checked out the latest version to test the latest release.

sssangha commented 4 years ago

@ehavazli I also tried just extracting the incidence angle layer for this interferogram 20181210_20181204 separately and it still isn't blank on my end:

gdalinfo -stats incidenceAngle/20181210_20181204
Driver: ISCE/ISCE raster
Files: incidenceAngle/20181210_20181204
       incidenceAngle/20181210_20181204.aux.xml
       incidenceAngle/20181210_20181204.xml
Size is 1561, 1801
Coordinate System is:
GEOGCRS["unknown",
    DATUM["unnamed",
        ELLIPSOID["Spheroid",6378137,298.257223563,
            LENGTHUNIT["metre",1,
                ID["EPSG",9001]]]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["degree",0.0174532925199433,
            ID["EPSG",9122]]],
    CS[ellipsoidal,2],
        AXIS["latitude",north,
            ORDER[1],
            ANGLEUNIT["degree",0.0174532925199433,
                ID["EPSG",9122]]],
        AXIS["longitude",east,
            ORDER[2],
            ANGLEUNIT["degree",0.0174532925199433,
                ID["EPSG",9122]]]]
Data axis to CRS axis mapping: 2,1
Origin = (-156.099999999999994,20.300833333333301)
Pixel Size = (0.000833333333333,-0.000833333333333)
Corner Coordinates:
Upper Left  (-156.1000000,  20.3008333) (156d 6' 0.00"W, 20d18' 3.00"N)
Lower Left  (-156.1000000,  18.8000000) (156d 6' 0.00"W, 18d48' 0.00"N)
Upper Right (-154.7991667,  20.3008333) (154d47'57.00"W, 20d18' 3.00"N)
Lower Right (-154.7991667,  18.8000000) (154d47'57.00"W, 18d48' 0.00"N)
Center      (-155.4495833,  19.5504167) (155d26'58.50"W, 19d33' 1.50"N)
Band 1 Block=1561x1 Type=Float32, ColorInterp=Undefined
  Minimum=36.801, Maximum=43.932, Mean=39.952, StdDev=1.646
  NoData Value=0
  Metadata:
    STATISTICS_MAXIMUM=43.931743621826
    STATISTICS_MEAN=39.951890613885
    STATISTICS_MINIMUM=36.801475524902
    STATISTICS_STDDEV=1.6459143698682
    STATISTICS_VALID_PERCENT=46.4

I would rerun in a new folder and redownload as @rzinke suggested. It's concerning though that the problem isn't captured. Did you check the other interferometric outputs though, are they all blank as well? Potentially there is an issue with the cutline being biased by a shorter IFG for which all frames hadn't been download (e.g. see #208)

dbekaert commented 4 years ago

@ehavazli suspect this might go back to the installation you are using. Can you provide reproducibility of the environment you are using too? i.e. provide the link to the exact shell script that installs the code.

ehavazli commented 4 years ago

@ehavazli suspect this might go back to the installation you are using. Can you provide reproducibility of the environment you are using too? i.e. provide the link to the exact shell script that installs the code.

https://github.com/aria-tools/ARIA-tools-docs/blob/aria_sds/JupyterDocs/ariaMintpy_install/ariaMintpy_install.sh

@sssangha @bbuzz31 @rzinke could you give it a shot?

sssangha commented 4 years ago

@ehavazli figured it out! I was able to replicate the issue and figured out it was associated with the most recent release of gdal.

I have confirmed that libgdal revision 3.0.4 is stable.

Add this condition to your https://github.com/aria-tools/ARIA-tools-docs/blob/aria_sds/JupyterDocs/ariaMintpy_install/ariaMintpy_env.yml yml file: libgdal==3.0.4

dbekaert commented 4 years ago

Thanks for looking into this Sim. Perhaps for now lets us forced 3.0.4 <= ligbgdal version for ariatools, till fixed by gdal community.

On Tue, Aug 18, 2020 at 8:28 PM sssangha notifications@github.com wrote:

@ehavazli https://github.com/ehavazli figured it out! I was able to replicate the issue and figured out it was associated with the most recent release of gdal.

I have confirmed that libgdal revision 3.0.4 is stable.

Add this condition to your https://github.com/aria-tools/ARIA-tools-docs/blob/aria_sds/JupyterDocs/ariaMintpy_install/ariaMintpy_env.yml yml file: libgdal==3.0.4

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/aria-tools/ARIA-tools/issues/210#issuecomment-675830193, or unsubscribe https://github.com/notifications/unsubscribe-auth/AESZPSJVO6SUYM45SJ63JGLSBNBGHANCNFSM4PX6L3OA .

sssangha commented 4 years ago

@ehavazli after communicating with the GDAL team (https://github.com/OSGeo/gdal/issues/2867), I realized we just need to make a minor change to ARIA-tools instead of just sticking with an older, static build of gdalWarp.

Files cannot be directly overwritten by gdalWarp anymore, so temporary files are created and deleted after warping. This fixes the issue you raised.

So please refer to PR #212 to test this out.

ehavazli commented 4 years ago

The problem is fixed with the new PR

Screen Shot 2020-08-19 at 2 00 37 PM