iiasa / ixmp4

A data warehouse for high-powered scenario analysis in the domain of integrated assessment of climate change and energy systems modeling
https://docs.ece.iiasa.ac.at/ixmp4
MIT License
10 stars 6 forks source link

Asyncronous Client #76

Closed meksor closed 5 months ago

meksor commented 5 months ago

Makes some of the client code asynchronous. This enables the concurrent sending of requests for enumerations. Asynchronicity is /not/ exposed to the user via async functions. Adds some settings variables to control client behaviour, like retries, how long to wait before retrying and when the new ApiEncumbered exception is thrown.

client_max_concurrent_requests: int = Field(2, lt=4)
client_max_request_retries: int = Field(3)
client_backoff_factor: int = Field(5)
client_timeout: int = Field(30)

Uses asyncio.Semaphore to control the maximum concurrency.

meksor commented 5 months ago

I can't really add tests for that, no. There is in general 2 ways the new exception is triggered and they are both dependant on the webserver configuration and how beefy the setup is. For example, ixmp4 would never send the response code 429 because it is not able to tell if the entire system is overloaded. Only the webserver/proxy (in our case nginx) in front of ixmp4 or the process manager/ orchestration thingy (in our case docker/kubernetes) could know this and react to it by returning 429. Also tracking IP's and rate limiting by IP is something best done in the webserver. I can imagine writing system tests that include ixmp4 and check if the exception is thrown under certain circumstances, but there is some way to go until then.