search

WikiWatershed / model-my-watershed

The web application front end for Model My Watershed.
https://modelmywatershed.org
Apache License 2.0
57 stars 31 forks source link

Switch Front-end Area Guard to AoI instead of Bounding Box #2237

Open rajadain opened 7 years ago

rajadain commented 7 years ago

The front-end area guard is currently limiting on the AoI's bounding box instead of the AoI's actual area, because previously that was the limiting factor. Since as of the Collections API we are not pulling all the tiles in the bounding box, this guard is no longer accurate and removing it will allow us to work with more interesting shapes.

kellyi commented 7 years ago

2286 will adjust the guard, but when testing it out with TX-11 Congressional District (72k sq km) I encountered some geoprocessing errors which seemed to have stopped the service.

I'm getting the geoprocessing project set up in my worker vm so I can introspect. I think shapes of that size may be hitting some kind of memory limit, but it's not clear from the upstart logs.

kellyi commented 7 years ago

I'm getting the geoprocessing project set up in my worker vm so I can introspect. I think shapes of that size may be hitting some kind of memory limit, but it's not clear from the upstart logs.

Did this & the service stopped without restarting or issuing any additional logging information. https://github.com/WikiWatershed/mmw-geoprocessing/issues/63 is there to add & adjust logging.

kellyi commented 7 years ago

Spoke with @mmcfarland and I think we may not want to drop the guard & make the other required adjustments without first figuring out what are some good performance benchmarks.

Dropping the bounding box & upping the guard would conceivably allow the app to analyze larger shapes, but it may be more meaningful for it so analyze a greater number of smaller shapes simultaneously.

If I'm reading https://water.usgs.gov/GIS/wbd_huc8.pdf correctly the largest HUC8s seems to be 12k sq km, which is well below the 75km guard we've currently got set on the bounding box.

FWIW staging seems to be able to handle congressional districts at or above 30k sq km; these may not be totally meaningful shapes, however. I imagine RWD may be able to create larger shapes; I don't know whether they'd routinely have a bounding box above 75k sq km.

Staging & production worker machines have way more memory than my local, so it seems like if it's an out of memory error it's likely to happen earlier on dev machines than on workers.

However, we should try to ensure that any out of memory or similar errors for the geoprocessing service yield an actual error message and cause the health check endpoint to fail, too. I was able to get a response from the ping endpoint even while the /run endpoint wasn't sending anything back. I can make a new issue for this either in MMW or mmw-geoprocessing.

kellyi commented 7 years ago

I'm putting this back in the queue for now so we can figure out whether and how we want to handle it.