search

GFDRR / thinkhazard

ThinkHazard!
http://thinkhazard.org/
GNU General Public License v3.0
33 stars 19 forks source link

processing errors #754

Closed fvanderbiest closed 7 years ago

fvanderbiest commented 7 years ago

On our dev server, make process yields

INFO     EQ-CUB-GEM
INFO       Cleaning previous outputs
INFO       Opening raster files
INFO       Iterating over 443 administrative divisions
INFO       Closing raster files
ERROR    Traceback (most recent call last):
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 78, in do_execute
    self.process_hazardset(id[0])
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 145, in process_hazardset
    outputs = self.create_outputs(hazardset)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 224, in create_outputs
    shape)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 333, in notpreprocessed_hazardlevel
    data = reader.read(1, window=window, masked=True)
  File "rasterio/_io.pyx", line 752, in rasterio._io.RasterReader.read (rasterio/_io.c:9256)
  File "rasterio/_base.pyx", line 719, in rasterio._base.eval_window (rasterio/_base.c:15813)
  File "rasterio/_base.pyx", line 741, in rasterio._base.eval_window (rasterio/_base.c:15407)
ValueError: invalid window: row range (30, 21)

INFO     EQ-SAR-GEM
INFO       Cleaning previous outputs
INFO       Opening raster files
INFO       Iterating over 8364 administrative divisions
INFO       Closing raster files
ERROR    Traceback (most recent call last):
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 78, in do_execute
    self.process_hazardset(id[0])
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 145, in process_hazardset
    outputs = self.create_outputs(hazardset)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 224, in create_outputs
    shape)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 333, in notpreprocessed_hazardlevel
    data = reader.read(1, window=window, masked=True)
  File "rasterio/_io.pyx", line 752, in rasterio._io.RasterReader.read (rasterio/_io.c:9256)
  File "rasterio/_base.pyx", line 719, in rasterio._base.eval_window (rasterio/_base.c:15813)
  File "rasterio/_base.pyx", line 741, in rasterio._base.eval_window (rasterio/_base.c:15407)
ValueError: invalid window: row range (464, 458)

INFO     EQ-US-GEM
INFO       Cleaning previous outputs
INFO       Opening raster files
INFO       Iterating over 3106 administrative divisions
INFO       Closing raster files
ERROR    Traceback (most recent call last):
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 78, in do_execute
    self.process_hazardset(id[0])
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 145, in process_hazardset
    outputs = self.create_outputs(hazardset)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 224, in create_outputs
    shape)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 333, in notpreprocessed_hazardlevel
    data = reader.read(1, window=window, masked=True)
  File "rasterio/_io.pyx", line 752, in rasterio._io.RasterReader.read (rasterio/_io.c:9256)
  File "rasterio/_base.pyx", line 719, in rasterio._base.eval_window (rasterio/_base.c:15813)
  File "rasterio/_base.pyx", line 741, in rasterio._base.eval_window (rasterio/_base.c:15407)
ValueError: invalid window: row range (94, 90)

INFO     EQ-SOA-GEM
INFO       Cleaning previous outputs
INFO       Opening raster files
INFO       Iterating over 8365 administrative divisions
INFO       Closing raster files
ERROR    Traceback (most recent call last):
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 78, in do_execute
    self.process_hazardset(id[0])
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 145, in process_hazardset
    outputs = self.create_outputs(hazardset)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 224, in create_outputs
    shape)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 333, in notpreprocessed_hazardlevel
    data = reader.read(1, window=window, masked=True)
  File "rasterio/_io.pyx", line 752, in rasterio._io.RasterReader.read (rasterio/_io.c:9256)
  File "rasterio/_base.pyx", line 719, in rasterio._base.eval_window (rasterio/_base.c:15813)
  File "rasterio/_base.pyx", line 741, in rasterio._base.eval_window (rasterio/_base.c:15407)
ValueError: invalid window: row range (486, 480)

INFO     EQ-SSA-GEM
INFO       Cleaning previous outputs
INFO       Opening raster files
INFO       Iterating over 1220 administrative divisions
INFO       Closing raster files
ERROR    Traceback (most recent call last):
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 78, in do_execute
    self.process_hazardset(id[0])
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 145, in process_hazardset
    outputs = self.create_outputs(hazardset)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 224, in create_outputs
    shape)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 333, in notpreprocessed_hazardlevel
    data = reader.read(1, window=window, masked=True)
  File "rasterio/_io.pyx", line 752, in rasterio._io.RasterReader.read (rasterio/_io.c:9256)
  File "rasterio/_base.pyx", line 719, in rasterio._base.eval_window (rasterio/_base.c:15813)
  File "rasterio/_base.pyx", line 741, in rasterio._base.eval_window (rasterio/_base.c:15407)
ValueError: invalid window: row range (218, 217)

INFO     FL-TLS-WB
INFO       Cleaning previous outputs
INFO       Opening raster files
INFO       Iterating over 25 administrative divisions
INFO       Closing raster files
ERROR    Traceback (most recent call last):
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 78, in do_execute
    self.process_hazardset(id[0])
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 145, in process_hazardset
    outputs = self.create_outputs(hazardset)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 224, in create_outputs
    shape)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 333, in notpreprocessed_hazardlevel
    data = reader.read(1, window=window, masked=True)
  File "rasterio/_io.pyx", line 811, in rasterio._io.RasterReader.read (rasterio/_io.c:10482)
MemoryError

INFO     EQ-TWN-GEM
INFO       Cleaning previous outputs
INFO       Opening raster files
INFO       Iterating over 2 administrative divisions
INFO       Closing raster files
ERROR    Traceback (most recent call last):
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 78, in do_execute
    self.process_hazardset(id[0])
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 145, in process_hazardset
    outputs = self.create_outputs(hazardset)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 224, in create_outputs
    shape)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 333, in notpreprocessed_hazardlevel
    data = reader.read(1, window=window, masked=True)
  File "rasterio/_io.pyx", line 752, in rasterio._io.RasterReader.read (rasterio/_io.c:9256)
  File "rasterio/_base.pyx", line 719, in rasterio._base.eval_window (rasterio/_base.c:15813)
  File "rasterio/_base.pyx", line 741, in rasterio._base.eval_window (rasterio/_base.c:15407)
ValueError: invalid window: row range (81, 79)

INFO     EQ-LEA-GEM
INFO       Cleaning previous outputs
INFO       Opening raster files
INFO       Iterating over 187 administrative divisions
INFO       Closing raster files
ERROR    Traceback (most recent call last):
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 78, in do_execute
    self.process_hazardset(id[0])
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 145, in process_hazardset
    outputs = self.create_outputs(hazardset)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 224, in create_outputs
    shape)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 333, in notpreprocessed_hazardlevel
    data = reader.read(1, window=window, masked=True)
  File "rasterio/_io.pyx", line 752, in rasterio._io.RasterReader.read (rasterio/_io.c:9256)
  File "rasterio/_base.pyx", line 719, in rasterio._base.eval_window (rasterio/_base.c:15813)
  File "rasterio/_base.pyx", line 741, in rasterio._base.eval_window (rasterio/_base.c:15407)
ValueError: invalid window: row range (100, 96)
stufraser1 commented 7 years ago

We had a problem with the projection of the first set of GEM rasters delivered. This error looks to be the same issue. The errors were corrected in new layers; the layers on old geonode were replaced with ones that we then successfully processed in TH v1.

The transfer of data from old to new geonode may have been done before the problematic files were replaced. Please copy the latest set of data from old geonode to new geonode, and try this again - the problem should be fixed

fvanderbiest commented 7 years ago

I'll try that, thanks !

arnaud-morvan commented 7 years ago

I've added, in complete task, a test on rasters. If bounds.bottom > bounds.top, hazardset is considered incomplete and this displayed in admin interface.

stufraser1 commented 7 years ago

@arnaud-morvan @fvanderbiest is this fixed now? the above layers seem to show as imported to TH at http://int.thinkhazard.org/admin/hazardsets