TypeError: Cannot read properties of null (reading 'points') using mongoose

[Kinuseka]

Getting an error issue with ratelimiter, this error happens once if the IP connects for the first time and any subsequent connection the rate-limiter will work correctly.

TypeError: Cannot read properties of null (reading 'points')
    at RateLimiterMongo._getRateLimiterRes (\user\node_modules\rate-limiter-flexible\lib\RateLimiterMongo.js:124:33)
    at RateLimiterMongo._afterConsume (\user\node_modules\rate-limiter-flexible\lib\RateLimiterStoreAbstract.js:51:22)
    at \user\node_modules\rate-limiter-flexible\lib\RateLimiterStoreAbstract.js:263:16
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

I am using mongoose 6.5.3 under serverless instance, (do note that this issue also occurs on shared instances.)

[Kinuseka]

code setup:

const ratelimit_db = mongoose.createConnection(`mongodb+srv://${config.DB_USER}:${config.DB_PASSWORD}@${config.ENDPOINT}/ratelimiter?retryWrites=true&w=majority`, options={useNewUrlParser: true});
var ratelimiter = new RateLimiterMongo({
    storeClient: ratelimit_db,
    points: 10,
    duration: 15 * 60 //15 minutes
});
async function ratelimitPage(req,res,next) {
    ratelimiter.consume(req.ip, 2).then((ratelimitResponse)=>{
        res.locals.ratelimited = false;
        next();
    })
    .catch((ratelimitResponse)=>{
        console.log(ratelimitResponse);
        res.locals.ratelimited = true;
        res.locals.ratelimit = ratelimitResponse.msBeforeNext;
        next();
    })
}

[animir]

@Kinuseka Hi, please take a look at this Wiki page https://github.com/animir/node-rate-limiter-flexible/wiki/Mongo#note-about-buffering-in-mongoose. It is about Buffering and connection. Also, check this part https://github.com/animir/node-rate-limiter-flexible/wiki/Mongo#connection-to-mongo-and-errors.

[Kinuseka]

This seems pretty difficult situation to deal with, I have tried multiple methods

1

var mongo_ratelimit = mongoose.createConnection(`mongodb+srv://${config.DB_USER}:${config.DB_PASSWORD}@${config.ENDPOINT}/ratelimiter?retryWrites=true&w=majority`, options={useNewUrlParser: true});

mongo_ratelimit.set('bufferCommands', false);
mongo_ratelimit.set('autoCreate', false);

still results in the same error

2

var mongo_ratelimit = mongoose.createConnection(`mongodb+srv://${config.DB_USER}:${config.DB_PASSWORD}@${config.ENDPOINT}/ratelimiter?retryWrites=true&w=majority`, options={useNewUrlParser: true, bufferCommands: false, autoCreate: false});

or

var mongo_ratelimit = async ()=>{
    return await mongoose.createConnection(`mongodb+srv://${config.DB_USER}:${config.DB_PASSWORD}@${config.ENDPOINT}/ratelimiter?retryWrites=true&w=majority`, options={useNewUrlParser: true, bufferCommands: false, autoCreate: false});
} 

Results in an error:

MongooseError: Cannot call `rlflx.createIndex()` before initial connection is complete if `bufferCommands = false`. Make sure you `await mongoose.connect()` if you have `bufferCommands = false`.
    at NativeCollection.<computed> [as createIndex] (user\node_modules\mongoose\lib\drivers\node-mongodb-native\collection.js:219:15)
    at RateLimiterMongo._initCollection (user\node_modules\rate-limiter-flexible\lib\RateLimiterMongo.js:108:16)
    at user\node_modules\rate-limiter-flexible\lib\RateLimiterMongo.js:54:16
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

3

according to this it seems that disabling buffering is not recommended. I also tried the unpopular solution

mongoose.createConnection(`mongodb+srv://${config.DB_USER}:${config.DB_PASSWORD}@${config.ENDPOINT}/ratelimiter?retryWrites=true&w=majority`, options={useNewUrlParser: true, bufferCommands: false, autoCreate: false, bufferMaxEntries:0});
} 

Which just results in:

MongoParseError: option buffermaxentries is not supported

okay

If anything, these feels like a bandaid solution than a permanent fix. using insuranceLimiter with RateLimiterMemory will remediate the issue (still not a permanent fix though).

[animir]

@Kinuseka Could you start your server after the connection is established? Like in this answer https://stackoverflow.com/a/42186818/4444520.

[Kinuseka]

Found the root cause, mongoDB in the examples works flawlessly due to .connect() actually creates multiple connection pools and manages the database on those multiple connections. The downside is that we can only use it once. To mitigate this issue we would have to use createConnection().

Unlike .connect(), createConnection() only creates 1 connection, so we would lose out on the automatic connection pool management from the .connect() therefore we would need to manage it ourselves.

Although this solution above works, it is a stretch especially when you already have your workflow arranged.

my solution to this problem is to simply wait for the database to connect before creating a RateLimitMongo instance

//mongo.js
var mongo_db_rt = mongoose.createConnection(`mongodb+srv://${config.DB_USER}:${config.DB_PASSWORD}@${config.ENDPOINT}/ratelimiter?retryWrites=true&w=majority`, options);

async function DB_wait(db) {
    function sleep(ms) {
        return new Promise((resolve) => {
            setTimeout(resolve, ms);
        });
    }
    /*
    0: disconnected
    1: connected
    2: connecting
    3: disconnecting
    */
    var state = { 0: "Disconnected", 1: "Connected", 2: "Connecting", 3: "Disconnecting" };
    while (db.readyState !== 1) {
        console.log(`Waiting for connection on db: ${db.name} | State: ${state[db.readyState]}`);
        await sleep(1000);
    }
    console.log(`Connection established with: ${db.name} | State: ${state[db.readyState]}`);
    return db;
}
var mongo_ratelimit = DB_wait(mongo_db_rt); // this assigns the variable into an unresolved promise
module.exports = {mongo_ratelimit};

const {mongo_ratelimit} = require('./database/mongo');

var ratelimiter = async ()=>{
    await mongo_ratelimit //since this is a promise, we wait for it to become resolved
    return new RateLimiterMongo({
        storeClient: mongo_ratelimit,
        points: 10,
        duration: 10 * 60 //10 minutes
        });
}
async function ratelimitPage(req,res,next) {
    (await ratelimiter()).consume(req.ip, 2).then((ratelimitResponse)=>{
        next();
    }
    ...
}

[Kinuseka]

I have read past issues, and this mistake happens quite pretty often. I feel like it is worth noting the differences between .connect and .createConnection where the latter requires connection initialization

[o-ali]

jumping on this question because im seeing a similar issue and was wondering if it had to do with buffering.

I see error Cannot read properties of null (reading 'value') when the first call is made, and every call after succeeds as long as the entry in the database hasnt been deleted for that key.

Is this a bug with the rate limiter that isnt handling null from the value properly on the first call, it should expect that the first call will not have any entry in the database.

edit: adding context

TypeError: Cannot read properties of null (reading 'value')
    at RateLimiterMongo._getRateLimiterRes (node_modules\rate-limiter-flexible\lib\RateLimiterMongo.js:118:23)
    at RateLimiterMongo._afterConsume (node_modules\rate-limiter-flexible\lib\RateLimiterStoreAbstract.js:51:22)
    at node_modules\rate-limiter-flexible\lib\RateLimiterStoreAbstract.js:205:16
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

using rate limiter with the following setup

        const opts: IRateLimiterMongoOptions = {
            storeClient: db, // await mongoose.connect(uri).then((res) => res.connection)
            points: 3, 
            duration: 5, 
            tableName: 'rate-limiter',
        };
        rateLimiter = new RateLimiterMongo(opts);

    "mongoose": "^8.0.1",
    "mongodb": "^6.2.0",

@animir for vis

[issam-seghir]

I have encountered the same error mentioned by @o-ali . My rate limiter middleware looks like this:

const mongoConn = mongoose.connection;

const options = new RateLimiterMongo({
    storeClient: mongoConn,
    dbName: ENV.DATABASE_NAME,
    keyPrefix: "middleware",
    points: 2, // 10 requests
    duration: 1, // per 1 second by IP
    tableName: "rate_limits", // Name of the collection to use for storing rate limit data
});
const rateLimiterMiddleware = async (req, res, next) => {
    try {
        const rateLimiterRes = await options.consume(req.ip); // Consume 1 point for each request
        log.debug("RateLimit-Limit Response .....");
        console.log(rateLimiterRes);
        res.setHeader("Retry-After", rateLimiterRes.msBeforeNext / 1000);
        res.setHeader("X-RateLimit-Limit", options.points);
        res.setHeader("X-RateLimit-Remaining", rateLimiterRes.remainingPoints);
        res.setHeader("X-RateLimit-Reset", new Date(Date.now() + rateLimiterRes.msBeforeNext).toISOString());

        next();
    // eslint-disable-next-line unicorn/catch-error-name
    } catch (rateLimiterRes) {
        if (rateLimiterRes instanceof RateLimiterRes) {
            log.warning("RateLimit-Limit Error .....");
            console.log(rateLimiterRes);

            res.setHeader("Retry-After", rateLimiterRes.msBeforeNext / 1000);
            res.setHeader("X-RateLimit-Limit", options.points);
            res.setHeader("X-RateLimit-Remaining", rateLimiterRes.remainingPoints);
            res.setHeader("X-RateLimit-Reset", new Date(Date.now() + rateLimiterRes.msBeforeNext).toISOString());

            log.error("rate-limiter-flexible : ", "Too Many Requests");
            res.status(429).send("Too Many Requests");
        } else {
            // Handle other types of errors
            console.error(rateLimiterRes);
            res.status(500).send("Internal Server Error");
        }
    }
};

module.exports = rateLimiterMiddleware;

[o-ali]

my issue was fixed in 4.0.1, see: https://github.com/animir/node-rate-limiter-flexible/issues/251.