la-famiglia-jst2324 / parma-analytics

ParmaAI analytics repository providing data processing and inference.
3 stars 0 forks source link

fix: Use singleton for database engine object for better connection pool #96

Closed egekocabas closed 9 months ago

egekocabas commented 9 months ago

Motivation

We were getting the database connection pool error while we were bursting one endpoint that uses the database connection.

image (6)

I searched about this error, and it seems that creating an engine for every time (every time get_engine() or get_session() is called) consumes all the connection pools.

Using a singleton pattern for the engine object and letting it manage its connection pool fixed the problem locally. I managed to send requests (uses the get_session()) to the /feed-raw-data endpoint with 5000 requests in a loop without a connection pool problem.

image (7)

Changes

Checklist

github-actions[bot] commented 9 months ago

β˜‚οΈ Python Coverage

current status: βœ…

Overall Coverage

Lines Covered Coverage Threshold Status
2046 1500 73% 0% 🟒

New Files

No new covered files...

Modified Files

File Coverage Status
parma_analytics/db/prod/engine.py 100% 🟒
TOTAL 100% 🟒

updated for commit: 4f44750 by action🐍