search

GFDRR / thinkhazard

ThinkHazard!
http://thinkhazard.org/
GNU General Public License v3.0
33 stars 19 forks source link

Please Re-run harvest and process #569

Closed stufraser1 closed 8 years ago

stufraser1 commented 8 years ago

CF regional data is being uploaded now (At Friday 18:35, half way through the datasets) SHARE data is complete on Geonode. FL-IND-SSBN and FL-EUROPE-ALFIERI have masks that should now line up, and have decimal values for permanent water.

pgiraud commented 8 years ago

Hi @stufraser1, We unfortunately still encounter problems with the FL-EUROPE-ALFIERI dataset. The mask doesn't use the exact same grid (ie. resolution, origin, size don't perfectly match). gdalinfo on the raster files give the following results:

$ gdalinfo /tmp/hazardsets/maxdepthseu_100y.tif 
Driver: GTiff/GeoTIFF
Files: /tmp/hazardsets/maxdepthseu_100y.tif
Size is 66290, 27372
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (-35.065387185674908,72.848052260302623)
Pixel Size = (0.001529595101217,-0.001529595101217)
Metadata:
  AREA_OR_POINT=Area
  DataType=Generic
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  ( -35.0653872,  72.8480523) ( 35d 3'55.39"W, 72d50'52.99"N)
Lower Left  ( -35.0653872,  30.9799751) ( 35d 3'55.39"W, 30d58'47.91"N)
Upper Right (  66.3314721,  72.8480523) ( 66d19'53.30"E, 72d50'52.99"N)
Lower Right (  66.3314721,  30.9799751) ( 66d19'53.30"E, 30d58'47.91"N)
Center      (  15.6330424,  51.9140137) ( 15d37'58.95"E, 51d54'50.45"N)
Band 1 Block=128x128 Type=Float32, ColorInterp=Gray
  NoData Value=-3.4028234663852886e+38

$ gdalinfo /tmp/hazardsets/europe_mask_n8Qrjtw.tif 
Driver: GTiff/GeoTIFF
Files: /tmp/hazardsets/europe_mask_n8Qrjtw.tif
Size is 66290, 27372
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (-35.065387185674908,72.848052260319079)
Pixel Size = (0.001529595101218,-0.001529595101218)
Metadata:
  AREA_OR_POINT=Area
  DataType=Generic
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  ( -35.0653872,  72.8480523) ( 35d 3'55.39"W, 72d50'52.99"N)
Lower Left  ( -35.0653872,  30.9799751) ( 35d 3'55.39"W, 30d58'47.91"N)
Upper Right (  66.3314721,  72.8480523) ( 66d19'53.30"E, 72d50'52.99"N)
Lower Right (  66.3314721,  30.9799751) ( 66d19'53.30"E, 30d58'47.91"N)
Center      (  15.6330424,  51.9140137) ( 15d37'58.95"E, 51d54'50.45"N)
Band 1 Block=128x128 Type=Float32, ColorInterp=Gray
  NoData Value=0

I know that the difference is really small, but it looks like the processing script takes a lot more resources when there's an offset. In the current case, the pixel size differs. We currently get a MemoryError.

Can Andy try to provide a new raster file that matches?

FYI, n #567 we added a test to verify that the matching is correct. Hazard sets will not be accepted if the raster grids don't match. Thus they won't be processed at all.

pgiraud commented 8 years ago

Note: same applies to FL-IND-SSBN.

stufraser1 commented 8 years ago

I've contacted Andy to try another approach. The differences are so small, but obviously crucial.

pgiraud commented 8 years ago

Everything works correctly now. Closing.

stufraser1 commented 8 years ago

@pgiraud which server is this implemented on - I tried int.thinkhazard.org and thinkhazard.org but both show mask errors in the 'complete' column of 'hazardsets' admin page

pgiraud commented 8 years ago

@stufraser1 It's on the production server. However, we just figured out that things are messing up between the two databases. We're trying to fix this as soon as possible.