Mongoose opening too many Mongodb connections

[vitthalzephyr]

Do you want to request a feature or report a bug? Reporting a bug

What is the current behavior? I am using mongoose in my application to create connections. Mongoose is configured for pool size of 10 with socketTimeout of 10000ms. There are six such instances of my application that I am running so at max I would expect 60 such active connections. I can't figure out exactly why 400+ connections are being made.

If the current behavior is a bug, please provide the steps to reproduce. Here is the code I am using to connect to mongoose:

mongoose.connect(process.env.INRMONGODB, 
    {useNewUrlParser: true, useUnifiedTopology: true, useFindAndModify: false, useCreateIndex: true, poolSize: 10, socketTimeoutMS: 10000, }, 
    (err) => {
    if (err) {
        console.log(err, "MONGODB CONNECTION ERROR");
        return false;
    }
    console.log("Mongoose Is Connected");
    }
);

What is the expected behavior? Ideally I would not expect more than 60 connections being opened. What are the versions of Node.js, Mongoose and MongoDB you are using? Note that "latest" is not a version. Node.js - 14.16.1 Mongoose - 5.12.3 Mongodb - 4.4.8

[vitthalzephyr]

OR is it that my session-store (connect-mongodb-session v2.4.1) causing an issue? below is the code for the same:

app.use(session({
    secret: process.env.SESSION_SECRET,
    saveUninitialized: false,
    resave: false,
    rolling: false,
    name: process.env.COOKIENAME,
    cookie: sessionCookieOptions,
    store: new MongoStore({
        uri: process.env.INRMONGODB, ttl: 2000, autoRemove: 'interval', autoRemoveInterval: 20, // In minutes
        hash: {
            salt: process.env.SALTSECRET,
            algorithm: 'sha1'
        },
        connectionOptions: {useNewUrlParser: true, useUnifiedTopology: true, poolSize: 10, socketTimeoutMS: 10000, }
    })
}));

[IslandRhythms]

My best guess with the current information provided is that you're saving 60 connections everytime you run your project. So you would need to clear out your previous sessions if you wanted to keep it at 60 or below.

Do you have any other mongodb tools connecting? You should at most have 20

[vitthalzephyr]

No Sir. All the six processes I mentioned are express servers and I have seen that upon bringing all of them down, the number of connections reduces to zero. But when they are running, the number of active connections keeps on fluctuating and sometimes reaches as high as 500. My MongoDB is hosted on Atlas cluster in M0 tier and I am only using Atlas for monitoring

[burn2delete]

I also experience this issue, with mongoose v5.13, also using Atlas M0

[vitthalzephyr]

I also experience this issue, with mongoose v5.13, also using Atlas M0

How did you fix this? I am eagerly waiting to see a solution. Should I downgrade mongoose? To what version?

[burn2delete]

I have not found a solution yet, we currently see 400+ connections opened with a single instance of our app.

[vkarpov15]

Are you sure that connect-mongodb-session and mongoose are the only libs you have that connect to MongoDB, and you only create one connect-mongodb-session store and call mongoose.connect() once per app?

Try running npm list | grep "mongo" and paste the output here.

Also, is the 400 connections for an individual server, or for your entire Atlas replica set? If it is across the replica set, that wouldn't be surprising: 10 connections from Mongoose, plus 2 from the MongoDB Node driver for connection monitoring. 10 from connect-mongodb-session, plus 2 from the MongoDB Node driver for connection monitoring. That's 24 connections per replica set server, per app. Multiply that by 6 app servers and 3 replica set servers, 24 6 3 = 432.

Based on the below script printing around 10000, I can confirm that Mongoose 5.12.3 does respect poolSize correctly. Although the MongoDB logs do show 3 connections, not 1, because the MongoDB Node driver opens 2 extra connections for internal monitoring

'use strict';

const mongoose = require('mongoose');

const { Schema } = mongoose;

run().catch(err => console.log(err));

async function run() {
  await mongoose.connect('mongodb://localhost:27017/test', {
    useNewUrlParser: true,
    useUnifiedTopology: true,
    poolSize: 2
  });

  const Test = mongoose.model('Test', Schema({ name: String }));
  await Test.updateOne({ name: 'test' }, { name: 'test' }, { upsert: true });

  const start = Date.now();
  let res = [];
  for (let i = 0; i < 10; ++i) {
    res.push(Test.find({ $where: 'sleep(1000) || true' }));
  }
  await Promise.all(res);
  console.log('Done in', Date.now() - start);
}

[vitthalzephyr]

1. Earlier, I had left the poolSize to default (which is 5) and yet I had the same problem - number of connections going high. Then I read somewhere it is better to specify poolSize so I added poolSize=10.
2. If 432 is the standard number of connections expected for my setup to maintain, then why the number of connections fluctuate between as low as 90's to as high as 480-500?
3. I will surely run your script and try to see if I can get more help.
4. Below is the output for all mongo package lists as requested by you: yarn list | grep mongo ├─ @types/mongodb@3.6.20 ├─ connect-mongodb-session@2.4.1 │ └─ mongodb@3.6.x │ ├─ mongoose-float@^1.0.4 │ ├─ mongoose@^5.12.3 │ ├─ connect-mongodb-session@^2.4.1 ├─ mongodb@3.6.11 ├─ mongoose-float@1.0.4 ├─ mongoose-legacy-pluralize@1.0.2 ├─ mongoose@5.13.8 │ ├─ @types/mongodb@^3.5.27 │ ├─ mongodb@3.6.11 │ ├─ mongoose-legacy-pluralize@1.0.2 │ ├─ connect-mongodb-session@^2.4.1

[vkarpov15]

2) The MongoDB node driver treats poolSize as a maximum pool size rather than an exact pool size. During periods of reduced activity, the MongoDB node driver sockets may time out due to inactivity. You can control this by using minPoolSize and maxPoolSize if this is causing problems for you, or you can increase socketTimeoutMS.

4) It looks like you have multiple copies of connect-mongodb-session, why is that?

[vitthalzephyr]

2. I have changed pool related parameters let me observe it for a few days and come back if issue still persist.
3. That is more of yarn-monorepo issue. connect-mongodb-session is used only once by each process.

[vitthalzephyr]

While I was experimenting throughout so far and have tried various possibilities, let me say that I haven't been able to solve this problem yet. Not sure but the last thing I want to try is move away from mongoose and do with driver queries.

[vkarpov15]

@vitthalzephyr what are the minPoolSize / maxPoolSize settings you're running with for connect-mongodb-session and mongoose?

Keep in mind that Mongoose doesn't do any connection management on its own, so if your current setup is causing a problematic number of connections, switching to the MongoDB driver won't help you.

[vitthalzephyr]

Unfortunately connect-mongodb-session doesn't support these parameters. For mongoose I am only specifying poolSize as 2 currently. Anyways I don't have too much traffic(max 100 sessions) and no long running queries (though I haven't checked the performance, but the http response is almost instantaneous to the user. So thinking no need for min/max poolSize. connect-mongodb-session is also potentially one culprit for this issue and I am also thinking of replacing the same as well. Hope to see a solution soon....

[vitthalzephyr]

Hi, hopefully I have a solution. My mongoose connection part was globally defined inside one module (daos.js) along with the mongoose models I need and then every other application module that requires database access will simply import the requisite mongoose model from there (i.e. daos.js). I changed my code to establish connection upon first database query like:

let cached = global.mongoose;

if(!cached) cached = global.mongoose = {conn: null, promise: null};

const daos = {
    connect: async function(){
        if(cached.conn) return cached.conn;
        if(!cached.promise){
            let options = {useNewUrlParser: true, useUnifiedTopology: true, useFindAndModify: false, useCreateIndex: true, 
                poolSize: 2, socketTimeoutMS: 10000, };
            cached.promise = mongoose.connect(process.env.INRMONGODB, options)
                .then((mongoose) => {
                    console.log('MONGODB CONNECTION SUCESSFUL');
                    return mongoose;
                }).catch((error)=>{
                    console.log(error, "MONGODB CONNECTION ERROR");
                });
        }
        cached.conn = await cached.promise;
        return cached.conn;
    },
    getBondMaster: async function(){
        await this.connect();
        return bondMasterModel = require('./bondMaster').bondMasterModel;
    },

And now I am observing my number of connections less than 20 for 3 of my servers. I didn't realize this probably because even my earlier code would say "Mongodb Connection Successful" only one time during the life of the process, but likely that it was opening connections per request.

Thanks to @vkarpov15 for the help.