Cache Warmup only issues requests sequentially

openeduhub / metalookup

Provide metadata about domains w.r.t accessibility, licencing, adds, etc.

GNU General Public License v3.0

5 stars 0 forks source link

Cache Warmup only issues requests sequentially #119

Closed MRuecklCC closed 2 years ago

MRuecklCC commented 2 years ago

To speed up cache warmup it would be nice to issue multiple requests to the /extract endpoint concurrently. The concurrency should be a configuration parameter of the application. More CPU/Ram will allow for higher concurrency without affecting other requests to much.

MRuecklCC commented 2 years ago

Concurrency can now be configured with the CACHE_WARMUP_CONCURRENCY environment variable. See also the metalookup/cache/warmup.py file for additional documentation.