search

EspressoSystems / hotshot-query-service

Generic query service for HotShot applications
https://espressosystems.github.io/hotshot-query-service/
GNU General Public License v3.0
5 stars 1 forks source link

Rate limit the proactive fetching task #603

Closed jbearer closed 3 months ago

jbearer commented 3 months ago

We now have rate limiting at the low-level fetcher, which is an excellent failsafe. However, by the time we get to that point, we have already consumed resources: memory for the list of waiters on the semaphore, memory for the task doing the fetch, memory for the fetch request itself, and memory for the list of callbacks on each item which is fetched for more than one purpose.

In most cases this is fine, because we just don't spawn that many active fetches at once. However, for proactive fetching, we may request thousands of active fetches at the same time, and even though we are now limiting the network traffic they generate, we are not limiting the amount of memory they consume.

This change adds a simple await instead of dropping/detaching the active fetch futures so that the proactive fetching task cannot proceed faster than the fetches can be completed. Note that we still get parallelism up to the chunk size.

imabdulbasit commented 3 months ago

ah please bump version

jbearer commented 3 months ago

Offhand I'm guessing the test timeout is from the NoStorage tests, which depend heavily on proactive fetching. We already turn the proactive fetching intervals way down for those tests. We probably also have to turn down the new delay to get them to pass.

imabdulbasit commented 3 months ago

Offhand I'm guessing the test timeout is from the NoStorage tests, which depend heavily on proactive fetching. We already turn the proactive fetching intervals way down for those tests. We probably also have to turn down the new delay to get them to pass.

https://github.com/EspressoSystems/hotshot-query-service/pull/603/commits/d9977b6099e6b29b79f44f2011644dd94602b015