search

ciur / papermerge

Open Source Document Management System for Digital Archives (Scanned Documents)
https://papermerge.com
Apache License 2.0
2.41k stars 257 forks source link

Django ORM leaves DB connections open #575

Closed ciur closed 6 months ago

ciur commented 6 months ago

Django ORM leaves open database connections in IDLE state for following REST API endpoints:

- GET /api/users/me
- GET /api/nodes/<id>
- GET /api/thumbnails/<id>
- GET /api/documents/<id>
- GET /api/pages/<id>/jpg
- GET /api/pages/<id>/svg

As result connections pile-up and when reaching specific limit, e.g. 100, 50, 25, depending on DB config, the app stops working as DB refuses all further connections. Same behavior was observed for PostgreSQL 16.1 and MariaDB 11.2.

Expected behavior: DB connections should be released when not used; or number of DB connections should remain stable when connection pooling is employed.

Info:

ciur commented 6 months ago

First PR which fixes couple of critical endpoints: https://github.com/papermerge/papermerge-core/pull/292

ciur commented 6 months ago

PR 2: https://github.com/papermerge/papermerge-core/pull/294 PR 3: https://github.com/papermerge/papermerge-core/pull/295

The solution is to replace Django ORM with SQLAlchemy. Of course ORM replacement it not matter of couple of weeks. This ticket is for fixing the most critical parts so that app is usable in production.

This fix will be shipped in 3.0.1 release.