WikiWatershed / mmw-geoprocessing

A Spark Job Server job for Model My Watershed geoprocessing.
Apache License 2.0
6 stars 6 forks source link

Increase Timeout to 2 minutes #98

Closed rajadain closed 2 years ago

rajadain commented 2 years ago

Overview

The longer the timeouts are, the more the likelihood of our resources being caught up and overloaded. To prevent that, we pick a short timeout. But, if the timeout is too short, then the expected requests will not complete.

Previously, we had set the default timeout to 1 minute in 26771b9 as part of #87. This was to allow the subbasin query for the Schuylkill HUC-8, which has 61 HUC-12s inside it, to complete. When running the geoprocessing service natively, using ./scripts/server, this would take ~13s:

61 HUC12 MultiOperation Subbasin, run 1 -> 18.611194 s
61 HUC12 MultiOperation Subbasin, run 2 -> 11.976093 s
61 HUC12 MultiOperation Subbasin, run 3 -> 14.061069 s
61 HUC12 MultiOperation Subbasin, run 4 -> 13.024482 s
61 HUC12 MultiOperation Subbasin, run 5 -> 12.124294 s

61 HUC12 MultiOperation Subbasin average -> 13.959426400000002 s

When running inside the Worker VM, with fewer resources than in native environments, it would take a little longer, ~20s:

61 HUC12 MultiOperation Subbasin, run 1 -> 21.185288 s
61 HUC12 MultiOperation Subbasin, run 2 -> 19.080413 s
61 HUC12 MultiOperation Subbasin, run 3 -> 19.835294 s
61 HUC12 MultiOperation Subbasin, run 4 -> 19.566461 s
61 HUC12 MultiOperation Subbasin, run 5 -> 19.327254 s

61 HUC12 MultiOperation Subbasin average -> 19.798942 s

With the usage of NHD Hi Res streams as of https://github.com/WikiWatershed/model-my-watershed/pull/3432, the expected times have increased significantly, beyond the 1 minute timeout.

By upping the limit to 2 minutes, we ensure that the Schuylkill HUC-8's subbasin query completes successfully.

Connects https://github.com/WikiWatershed/model-my-watershed/issues/3446

Demo

Schuylkill HUC-8 Subbasin with NHD Hi Res Streams, running natively:

61 HUC12 MultiOperation Subbasin, run 1 -> 68.708514 s
61 HUC12 MultiOperation Subbasin, run 2 -> 68.344673 s
61 HUC12 MultiOperation Subbasin, run 3 -> 66.588282 s
61 HUC12 MultiOperation Subbasin, run 4 -> 68.047426 s
61 HUC12 MultiOperation Subbasin, run 5 -> 67.541642 s

61 HUC12 MultiOperation Subbasin average -> 67.84610740000001 s

Schuylkill HUC-8 Subbasin with NHD Hi Res Streams, running in the Worker VM:

61 HUC12 MultiOperation Subbasin, run 1 -> 98.325671 s
61 HUC12 MultiOperation Subbasin, run 2 -> 93.014152 s
61 HUC12 MultiOperation Subbasin, run 3 -> 88.35564 s
61 HUC12 MultiOperation Subbasin, run 4 -> 91.579747 s
61 HUC12 MultiOperation Subbasin, run 5 -> 98.874168 s

61 HUC12 MultiOperation Subbasin average -> 94.0298756 s

Testing Instructions

rajadain commented 2 years ago

In the interest of time I'm going to merge this in directly.