✨ Limit inflight requests to the api-server

[bisgaard-itis]

What do these changes do?

• Introduce a new traefik middleware which limits the total number of in-flight requests to the api-server. This means there is an absolute limit to the number of requests the api-server will be handling at any given time (see https://doc.traefik.io/traefik/middlewares/http/inflightreq/).
• If the number of allowed in-flight requests to the api-server is exceeded traefik returns a 429
• Related ops MR: https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/651

Related issue/s

• 5952

How to test

run local deployment with API_SERVER_INFLIGHTREQ_AMOUNT=1 in .env. Then run

seq 10 | parallel -j 10 "curl -X 'GET'   'http://<ip address>.nip.io:<api-server-port>/v0/solvers'   -H 'accept: application/json'   -H 'Authorization: Basic <auth>' -v > output{#}.txt"

with <api-server-port>=the port exposing the api-server through traefik and <auth>=valid authorization. Observe the output*.txt files and see that only a single request received a 2xx return code and the others received a 429.

Dev-ops checklist

• [x] No ENV changes or I properly updated ENV (read the instruction)

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 87.9%. Comparing base (cafbf96) to head (0922c6b). Report is 310 commits behind head on master.

Additional details and impacted files

[](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6007?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) ```diff @@ Coverage Diff @@ ## master #6007 +/- ## ========================================= + Coverage 84.5% 87.9% +3.3% ========================================= Files 10 1419 +1409 Lines 214 58212 +57998 Branches 25 1395 +1370 ========================================= + Hits 181 51169 +50988 - Misses 23 6750 +6727 - Partials 10 293 +283 ``` | [Flag](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6007/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | Coverage Δ | | |---|---|---| | [integrationtests](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6007/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `64.7% <ø> (?)` | | | [unittests](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6007/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `85.9% <ø> (+1.3%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#carryforward-flags-in-the-pull-request-comment) to find out more. [see 1384 files with indirect coverage changes](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6007/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation)

[YuryHrytsuk]

Thanks!

This limit will only take place for requests coming via public network. For the internal (within docker swarm cluster) communication it will not apply.

[bisgaard-itis]

Q: are you adding a sleep+retry in the client for this code?

there is already the retry strategy in place with exponential backoff. That should also be triggered by a 429

[bisgaard-itis]

very nice! Q: what is your strategy to determine this value (e.g. why did you start in 25 simultaneous requests?)

When running Werner's "load test" I see in our grafana dashboard that the api-server is at times handling around 50 requests simultaniously. That seems to add quite a performance overhead, so 25 is chosen to try and avoid that heavy load. Not sure this is optimal of course. So we will have to tune it as we go

[bisgaard-itis]

@YuryHrytsuk and @mrnicegyu11 can I get you guys (codeowners) to accept this one?