search

openskope / skopeui

SKOPE user interface and visualization app
https://www.openskope.org
MIT License
0 stars 4 forks source link

support larger area queries #195

Open alee opened 1 year ago

alee commented 1 year ago

currently there is a limit to the geographical size https://www.openskope.org/skope-users-guide#study-area-size-limits due to timeouts #42

this is more to do with the number of pixels that are needed from the datacube to process things than the actual geographical square km area, a long thin rectangle would also timeout.

come up with a plan to support larger area queries

this would probably require asynchronous processing / notification changes to https://github.com/openskope/skope-api as well as this repo

related to #186 and https://github.com/openskope/skope-api/issues/39

kintigh commented 1 year ago

Allen. I have a distinct memory that sometime in the past, SKOPE was able to run much larger datasets than it can now. As I recall, this was associated with a major rewrite that someone not you (Calvin) was doing. This makes me think that perhaps there is some critical piece of code that got rewritten in a much less efficient manner. I don't think that it was just a longer timeout.

Documentation dated 11/15/2021 (from the wayback machine https://web.archive.org/web/20211201222306/https://www.openskope.org/skope-users-guide) says

For high spatial resolution datasets (such as PaleoCAR), the application will be more responsive for smaller study areas. And, for those datasets, some areas may be too large for SKOPE to process interactively. At present, for PaleoCAR, a maximum area of 5,000 to 20,000km2 (depending on server load) can be computed before the app times out.

This would have been based on some real experiments on my part some time around then. But at some point after than when I did the experiments for the current version, the capacity went down a lot. I do remember that even farther back whebn we were talking about the timeouts you tried changing them but there were some real complications in doing that.

I don't know if this helps. Keith

Current documentation language based on experiments about April 2022

SKOPE’s computation load mainly depends on the size of the smallest rectangle (oriented E-W/N-S) that will enclose the selected area. At present, for PaleoCAR, a maximum area of that rectangle before the app times out is about 5700km2 (e.g. ,a square about 75km on a side or a 38km radius circle). For lower resolution datasets, such as the Living Blended Drought Atlas, there is no practical limit.

alee commented 1 year ago

Thanks for the extra context Keith! This is definitely helpful, we'll take a deep dive into the backend performance as well as we look into longer-term solutions for the timeout issue.