search

Applied-GeoSolutions / gips

Geospatial Image Processing System
GNU General Public License v3.0
17 stars 5 forks source link

S2 not falling back to LT means for atmospheric correction #494

Open bhbraswell opened 6 years ago

bhbraswell commented 6 years ago

This is different from #490 I think.

And forgive the lengthy TS below but I believe that this case should produce a resulting product that is atmospherically corrected based on the composite long term means. The S2 data are available, but the AOD are not.

It calculates ref-toa then fails with trying to read too large an array from AOD.

root@46ca0ff723da:/gips# gips_process sentinel2 -p ref -d 2018-06-02 -s gips/test/NHseacoast.shp -v4 --fetch --overwrite
GIPS Data Processing (v0.11.0-dev)
Retrieving inventory for site NHseacoast-0
For L1C, requested dates, 2018-06-02 - 2018-06-02, are not in the valid range of 2015-01-01 - 2018-06-01.
For L1C, requested dates, 2018-06-02 - 2018-06-02, are not in the valid range of 2015-01-01 - 2018-06-01.
For L1C, requested dates, 2018-06-02 - 2018-06-02, are not in the valid range of 2015-01-01 - 2018-06-01.
For L1C, requested dates, 2018-06-02 - 2018-06-02, are not in the valid range of 2015-01-01 - 2018-06-01.
Processing [ref] on 0 dates (0 files)
Processing completed in 0:00:00.000164
root@46ca0ff723da:/gips# gips_process sentinel2 -p ref -d 2018-05-31 -s gips/test/NHseacoast.shp -v4 --fetch --overwrite
GIPS Data Processing (v0.11.0-dev)
Retrieving inventory for site NHseacoast-0
/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
Fetching S2A_MSIL1C_20180531T155021_N0206_R054_T18TYP_20180531T193310.zip
Authentication selected: Basic realm="OData service"

     0K ........ ........ ........ .....   100% 2.25M=13s2018-06-04 19:44:21 URL:https://scihub.copernicus.eu/dhus/odata/v1/Products('f3ff7009-18da-476d-8f48-6d42b2b060e8')/$value [30473273/30473273] -> "/archive/sentinel2/stage/tmpzSFI91/S2A_MSIL1C_20180531T155021_N0206_R054_T18TYP_20180531T193310.zip" [1]
S2A_MSIL1C_20180531T155021_N0206_R054_T18TYP_20180531T193310.zip -> /archive/sentinel2/tiles/18TYP/2018151/S2A_MSIL1C_20180531T155021_N0206_R054_T18TYP_20180531T193310.zip
1 files (1 links) from /archive/sentinel2/stage/S2A_MSIL1C_20180531T155021_N0206_R054_T18TYP_20180531T193310.zip added to archive in 0:00:00.043839
Extracting files from /archive/sentinel2/tiles/18TYP/2018151/S2A_MSIL1C_20180531T155021_N0206_R054_T18TYP_20180531T193310.zip to /tmp/tmpgX1UBU/extractZXyvCJ
Extracting S2A_MSIL1C_20180531T155021_N0206_R054_T18TYP_20180531T193310.SAFE/GRANULE/L1C_T18TYP_A015352_20180531T155122/MTD_TL.xml
Moving files from /tmp/tmpgX1UBU/extractZXyvCJ to /tmp/tmpgX1UBU
Asset cloud cover is 0.0 %, meets pclouds threshold of 100 %
Processing [ref] on 1 dates (1 files)
0:00:00.000023:  Starting processing for this temporal-spatial unit
0:00:00.005486:  Start VRT for ref-toa image
0...10...20...30...40...50...60...70...80...90...100 - done.
0:00:01.137292:  Finished VRT for ref-toa image
0:00:01.137398:  Starting reversion to TOA radiance.
0:00:05.299434:  TOA radiance reversion factor for BLUE (band 1): 0.556712268288
0:00:05.300942:  TOA radiance reversion factor for GREEN (band 2): 0.517941377357
0:00:05.301111:  TOA radiance reversion factor for RED (band 3): 0.4295421552
0:00:05.301189:  TOA radiance reversion factor for REDEDGE1 (band 4): 0.404708104165
0:00:05.301246:  TOA radiance reversion factor for REDEDGE2 (band 5): 0.365780970634
0:00:05.301296:  TOA radiance reversion factor for REDEDGE3 (band 6): 0.330120727825
0:00:05.301343:  TOA radiance reversion factor for NIR (band 7): 0.295903598482
0:00:05.301456:  TOA radiance reversion factor for REDEDGE4 (band 8): 0.271384873421
0:00:05.301733:  TOA radiance reversion factor for SWIR1 (band 9): 0.0697665819448
0:00:05.301810:  TOA radiance reversion factor for SWIR2 (band 10): 0.0242176029594
0:00:05.302110:  Computing atmospheric corrections for surface reflectance
Generating atmospheric correction object.
Running atmospheric model (6S)
Retrieving inventory for site tiles
For MOD08, requested dates, 2018-05-31 - 2018-05-31, are not in the valid range of 2000-02-18 - 2018-05-28.
No files found; nothing to archive.
Unable to load aod values:
Traceback (most recent call last):
  File "/gips/gips/utils.py", line 586, in cli_error_handler
    yield
  File "/gips/gips/data/aod/aod.py", line 354, in get_aod
    img = inv[date].tiles[cls.Asset.Repository._the_tile].open('aod')
  File "/gips/gips/inventory/__init__.py", line 49, in __getitem__
    return self.data[date]
KeyError: datetime.date(2018, 5, 31)

runtime error
Unable to read point from /archive/aod/composites/lta.tif:
Traceback (most recent call last):
  File "/gips/gips/utils.py", line 586, in cli_error_handler
    yield
  File "/gips/gips/data/aod/aod.py", line 320, in _read_point
    vals = img[0].Read(roi).squeeze()
  File "/usr/local/lib/python2.7/dist-packages/gippy/algorithms.py", line 3836, in Read
    return _algorithms.GeoRaster_Read(self, *args)
RuntimeError: error reading /archive/aod/composites/lta.tif, band 1: Access window out of range in RasterIO().  Requested
(466,44) of size 3x3 on raster of 360x180.

Error:
Traceback (most recent call last):
  File "/gips/gips/utils.py", line 586, in cli_error_handler
    yield
  File "/gips/gips/inventory/__init__.py", line 332, in process
    self.data[date].process(*args, **kwargs)
  File "/gips/gips/tiles.py", line 73, in process
    [t.process(*args, products=self.products.products, **kwargs) for t in self.tiles.values()]
  File "/gips/gips/data/core.py", line 1306, in wrapper
    return wrapped_method(self, *args, **kwargs)
  File "/gips/gips/data/sentinel2/sentinel2.py", line 1253, in process
    self.ref_geoimage(asset_type, sensor)
  File "/gips/gips/data/sentinel2/sentinel2.py", line 1035, in ref_geoimage
    atm6s = self.assets[asset_type].generate_atmo_corrector()
  File "/gips/gips/data/sentinel2/sentinel2.py", line 658, in generate_atmo_corrector
    self._atmo_corrector = atmosphere.SIXS(visbands, wvlens, geo, dt, sensor=self.sensor)
  File "/gips/gips/atmosphere.py", line 126, in __init__
    geometry['lat'], geometry['lon'], date_time.date()
  File "/gips/gips/data/aod/aod.py", line 407, in get_aod
    raise Exception("Could not retrieve AOD")
Exception: Could not retrieve AOD

Processing completed in 0:00:05.532085
Fatal: 3 error(s) occurred:
Unable to load aod values:
Traceback (most recent call last):
  File "/gips/gips/utils.py", line 586, in cli_error_handler
    yield
  File "/gips/gips/data/aod/aod.py", line 354, in get_aod
    img = inv[date].tiles[cls.Asset.Repository._the_tile].open('aod')
  File "/gips/gips/inventory/__init__.py", line 49, in __getitem__
    return self.data[date]
KeyError: datetime.date(2018, 5, 31)

Unable to read point from /archive/aod/composites/lta.tif:
Traceback (most recent call last):
  File "/gips/gips/utils.py", line 586, in cli_error_handler
    yield
  File "/gips/gips/data/aod/aod.py", line 320, in _read_point
    vals = img[0].Read(roi).squeeze()
  File "/usr/local/lib/python2.7/dist-packages/gippy/algorithms.py", line 3836, in Read
    return _algorithms.GeoRaster_Read(self, *args)
RuntimeError: error reading /archive/aod/composites/lta.tif, band 1: Access window out of range in RasterIO().  Requested
(466,44) of size 3x3 on raster of 360x180.

Error:
Traceback (most recent call last):
  File "/gips/gips/utils.py", line 586, in cli_error_handler
    yield
  File "/gips/gips/inventory/__init__.py", line 332, in process
    self.data[date].process(*args, **kwargs)
  File "/gips/gips/tiles.py", line 73, in process
    [t.process(*args, products=self.products.products, **kwargs) for t in self.tiles.values()]
  File "/gips/gips/data/core.py", line 1306, in wrapper
    return wrapped_method(self, *args, **kwargs)
  File "/gips/gips/data/sentinel2/sentinel2.py", line 1253, in process
    self.ref_geoimage(asset_type, sensor)
  File "/gips/gips/data/sentinel2/sentinel2.py", line 1035, in ref_geoimage
    atm6s = self.assets[asset_type].generate_atmo_corrector()
  File "/gips/gips/data/sentinel2/sentinel2.py", line 658, in generate_atmo_corrector
    self._atmo_corrector = atmosphere.SIXS(visbands, wvlens, geo, dt, sensor=self.sensor)
  File "/gips/gips/atmosphere.py", line 126, in __init__
    geometry['lat'], geometry['lon'], date_time.date()
  File "/gips/gips/data/aod/aod.py", line 407, in get_aod
    raise Exception("Could not retrieve AOD")
Exception: Could not retrieve AOD

root@46ca0ff723da:/gips# gdalinfo /archive/aod/composites/lta.tif
Driver: GTiff/GeoTIFF
Files: /archive/aod/composites/lta.tif
Size is 360, 180
Coordinate System is:
GEOGCS["Unknown datum based upon the Clarke 1866 ellipsoid",
    DATUM["Not_specified_based_on_Clarke_1866_spheroid",
        SPHEROID["Clarke 1866",6378206.4,294.9786982138982,
            AUTHORITY["EPSG","7008"]]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433]]
Origin = (-180.000000000000000,90.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (-180.0000000,  90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"N)
Lower Left  (-180.0000000, -90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"S)
Upper Right ( 180.0000000,  90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"N)
Lower Right ( 180.0000000, -90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"S)
Center      (   0.0000000,   0.0000000) (  0d 0' 0.01"E,  0d 0' 0.01"N)
Band 1 Block=360x2 Type=Float32, ColorInterp=Gray
  Min=-0.026 Max=2.148 
  Minimum=-0.026, Maximum=2.148, Mean=0.097, StdDev=0.064
  NoData Value=-32768
  Metadata:
    STATISTICS_MAXIMUM=2.1480000019073
    STATISTICS_MEAN=0.09720040762772
    STATISTICS_MINIMUM=-0.025500001385808
    STATISTICS_STDDEV=0.064231771279441
Band 2 Block=360x2 Type=Float32, ColorInterp=Undefined
  Min=0.000 Max=82.069 
  Minimum=0.000, Maximum=82.069, Mean=0.248, StdDev=1.232
  NoData Value=-32768
  Metadata:
    STATISTICS_MAXIMUM=82.0693359375
    STATISTICS_MEAN=0.24828596542023
    STATISTICS_MINIMUM=1.0000001111621e-06
    STATISTICS_STDDEV=1.2318142438689
bhbraswell commented 6 years ago

@ircwaves if this looks like a real thing then you can assign it to me if you want.

ircwaves commented 5 years ago

The error messages here are lost in the noise but they are all about asking for AOD data before its latency window has passed. I believe we agreed that latency metadata should be ignored for fetch purposes, but maybe used to inform the error messages.

For MOD08, requested dates, 2018-05-31 - 2018-05-31, are not in the valid range of 2000-02-18 - 2018-05-28.

If you want to get into that, it seems like a reasonable change that might help NRT processing streams. The question becomes to bail on processing, or to use LTA(D) composites and march on? Seems like you'd want submit reprocessing jobs for each scene that was corrected with an LTA(D) once the AOD data was available.

If you don't want to get into that, I'd say this can be closed.