add queueing and multi process support

bananaml / potassium

An HTTP serving framework by Banana

Apache License 2.0

97 stars 9 forks source link

add queueing and multi process support #48

Closed Peddle closed 9 months ago

Peddle commented 9 months ago

Adds ability to queue multiple requests for better local dev experience
adds request id to all logs
adds ability to configure number of processes

erik-dunteman commented 9 months ago

Doing some QA.

I am just observing behavior, not saying we should fix anything just yet.

erik-dunteman commented 9 months ago

If I run single worker, three background tasks w/ 5s sleep, they all queue (nice) but the 200 responses all log under the first request id.

erik-dunteman commented 9 months ago

Worker ID super helpful in logs. The order they consume from the queue makes sense. 3 bg calls to 2 workers, first 2 calls claimed immediately, third claimed by the worker that took # 1

erik-dunteman commented 9 months ago

Going to try with 100k processes, wish me luck

erik-dunteman commented 9 months ago

Got up to 32 workers (BERT) running locally, laptop struggled but seemingly no bugs.

I'd suggest

cap quantity of workers so users don't footgun and Banana doesn't get loaded up with OOM-bombs
showing some proof of a highly concurrent slam test passing

erik-dunteman commented 9 months ago

Tentative LGTM from me given above is addressed