search

WFP-VAM / prism-app

PRISM is an interactive map-based dashboard that simplifies the integration of geospatial data on hazards, along with information on socioeconomic vulnerability
MIT License
45 stars 32 forks source link

Exposure analysis request consumes near 4gb of memory on open data cube request #423

Closed wadhwamatic closed 2 months ago

wadhwamatic commented 2 years ago

The current exposure analysis, which requests high resolution population raster data from WFP's data cube instance, highlights a problem we have with WCS requests used by PRISM. The population raster dataset from worldpop is 100 meter resolution, and we're requesting an uncompressed file with national coverage which, in the case of Mozambique, is 3.74.

As an example, a request for exposure to a tropical storm in Mozambique generates this request:

curl 'https://prism-api.ovio.org/stats' \ -H 'authority: prism-api.ovio.org' \ -H 'cache-control: max-age=0' \ -H 'sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="99", "Google Chrome";v="99"' \ -H 'accept: application/json' \ -H 'content-type: application/json' \ -H 'sec-ch-ua-mobile: ?0' \ -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.83 Safari/537.36' \ -H 'sec-ch-ua-platform: "macOS"' \ -H 'origin: https://new-ows-moz.surge.sh' \ -H 'sec-fetch-site: cross-site' \ -H 'sec-fetch-mode: cors' \ -H 'sec-fetch-dest: empty' \ -H 'referer: http://localhost:3000/' \ -H 'accept-language: en-US,en;q=0.9,id;q=0.8' \ --data-raw '{"geotiff_url":"https://odc.ovio.org/?service=WCS&request=GetCoverage&version=2.0.0&coverageId=wp_pop_cicunadj&subset=Long(30.215605, 40.84062576300005)&subset=Lat(-26.867040757999973, -10.472198489999926)","zones_url":"https://prism-admin-boundaries.s3.us-east-2.amazonaws.com/moz_bnd_adm2_WFP.json","group_by":"source_id","wfs_params":{"url":"https://geonode.wfp.org/geoserver/ows","layer_name":"moz_gdacs_buffers","time":"2021-02-22","key":"label"}}' \ --compressed

In a test instance of our Data Cube, this fails with a 500 error:

500 Internal Server Error: The file you requested is not available - https://ows.earthobservation.vam.wfp.org/?service=WCS&request=GetCoverage&version=2.0.0&coverageId=wp_pop_cicunadj&subset=Long(30.215605, 40.84062576300005)&subset=Lat(-26.867040757999973, -10.472198489999926)"

And this is the error raised on the data cube:

Unexpected server error: Unable to allocate 3.74 GiB for an array with shape (1, 39349, 25501) and data type uint32

We had a similar issue surface when we tried to develop a download feature for raster files. See this PR https://github.com/WFP-VAM/prism-frontend/pull/208 and @valpesendorfer 's comment

Of note, we'll have a STAC implementation in the near-term and should consider moving away from WCS requests and use STAC instead. Details on STAC are here and example of downloading a raster through a STAC request is here

wadhwamatic commented 2 months ago

The WCS request was replaced by a STAC request