search

natcap / pygeoprocessing

Geoprocessing operations for Python
https://pygeoprocessing.readthedocs.io/en/latest/
Other
78 stars 8 forks source link

`convolve_2d` raises unhelpful error on worker timeout #360

Open emlys opened 10 months ago

emlys commented 10 months ago

From this forum thread: https://community.naturalcapitalproject.org/t/errors-and-very-long-period-of-running-urban-cooling-model/4298

Traceback (most recent call last):
  File "natcap\invest\utils.py", line 164, in prepare_workspace
  File "invest\cli.py", line 470, in main
  File "natcap\invest\urban_cooling_model.py", line 651, in execute
  File "taskgraph\Task.py", line 674, in add_task
  File "taskgraph\Task.py", line 1093, in _call
  File "natcap\invest\urban_cooling_model.py", line 1500, in convolve_2d_by_exponential
  File "pygeoprocessing\geoprocessing.py", line 3035, in convolve_2d
  File "queue.py", line 179, in get
_queue.Empty

This could happen if no results are available to get from the queue within the default timeout of 60 seconds. We should provide an error message explaining what timed out.

phargogh commented 10 months ago

Another missing piece from the InVEST side of things is what's a reasonable timeout for super large rasters. I think we cap it to some fixed number of seconds, regardless of the data size. Would it be worth making the timeout a function of the source raster size and the kernel size?

Related: https://github.com/natcap/pygeoprocessing/issues/251