Djongo/PyMongo keeps creating new DB connections for Django REST framework API requests

[martin-marko]

I'm experiencing an issue with djongo when used with Django REST framework where it initiates a new connection with my MongoDB at every request except for the very first one after starting the webserver. This means the first request to my Django API is fast (~80 ms) but all the following requests are significantly slower (~800 ms) because they all initiate a new connection.

When profiling the requests I can see that pymongo\pool.py:950(_create_connection) and pymongo\pool.py:1263(connect) are present in each request except for the first one, which I assume means the first request uses the pre-initiated DB connection, but all the others have to create a new connection, hence why it's so slow.

I'm also seeing huge spikes of connections in my MongoDB dashboard, there should be only 1 connection, but when I'm testing this it suddenly gets up to 50+ connections, and even when I shut down the webserver, the connections only drop by one and plenty of zombie connections remain that only die after some timeout.

Here are the profiling results for the first and second request.

For the installation and configuration, I've followed the djongo GitHub instructions, so it goes like this:

DATABASES = {
    "default": {
        "ENGINE": "djongo",
        "NAME": "django",
        "CLIENT": {
            "host": f"mongodb+srv://{MONGODB_PATH}",
        }
    }
}

(MONGODB_PATH = f"{MONGODB_DJANGO_USER}:{MONGODB_DJANGO_PASS}@{MONGODB_CLUSTER}" from environment variables)

The API endpoint I'm testing with goes like this:

@api_view(["GET"])
@authentication_classes([TokenAuthentication])
@permission_classes([IsAuthenticated])
def test(request):
    if request.method == "GET":
        data = {"user": str(request.user)}
        return JsonResponse(data)

Versions:

Django==3.2.7
djangorestframework==3.12.4
djongo==1.3.6
pymongo==3.12.0 (MongoDB 4.4.8 Enterprise)

P.S. It's much faster when I'm running the webserver closer to the DB, but I'm running them far apart from each other for debugging on purpose, so it's easier to spot such issues.

[martin-marko]

Misclick, closed by accident

[martin-marko]

New findings to this, if I'm logged into Django admin while profiling this, I'm getting consistently fast results and no signs of reconnecting to the DB... why? I'm not using SessionAuthentication, I'm authenticating via tokens still, no matter if I'm logged in to admin or not. Why/how does it make a difference?

[mosi-kha]

@martin-marko you should set CONN_MAX_AGE: None

[martin-marko]

@martin-marko you should set CONN_MAX_AGE: None

I have just tried, but it doesn't seem to change anything, djongo is still closing and creating new connections on every request (can confirm both in logs and in profiling)

2021-11-14 15:32:29,317 - DEBUG - djongo.base - MongoClient connection closed
2021-11-14 15:32:31,545 - DEBUG - djongo.base - Existing MongoClient connection closed
2021-11-14 15:32:31,546 - DEBUG - djongo.base - New Database connection

[mosi-kha]

we had this issue, running performance and loading test to figure out this issue. if you read the below codes see it's not about djongo to create/close connections https://github.com/nesdis/djongo/blob/master/djongo/base.py https://github.com/django/django/blob/main/django/db/backends/base/base.py#L198 https://github.com/django/django/blob/main/django/db/__init__.py#L36

[martin-marko]

See https://github.com/nesdis/djongo/issues/571#issuecomment-950265651, that looks pretty djongo related to me?

[martin-marko]

So how am I supposed to fix it?

[praveenexaf]

Anyone found a fix for this?