Hambergo / 9kmmrbot

MIT License
104 stars 29 forks source link

Exceeding memory limit on mongo aggregations #30

Closed Hambergo closed 5 years ago

Hambergo commented 5 years ago

Currently there are 6 aggregations that can lead to an MongoError: Exceeded memory limit for $group, but didn't allow external sort. Pass allowDiskUse:true to opt in.

index.js 112: let games = await mongoDb.collection('games').aggregate([{ $group: { _id: '$createdAt', matches: { $addToSet: '$$ROOT' } } }, { $sort: { _id: -1 } }, { $limit: 2 }]).toArray()

lib\commands\Account.js 15: let rps = mongoDb().collection('rps').aggregate([{ $group: { _id: '$createdAt', rps: { $addToSet: '$$ROOT' } } }, { $sort: { _id: -1 } }, { $limit: 1 }]).toArray()

lib\commands\NotablePlayers.js 51: let rps = mongoDb().collection('rps').aggregate([{ $group: { _id: '$createdAt', rps: { $addToSet: '$$ROOT' } } }, { $sort: { _id: -1 } }, { $limit: 1 }]).toArray()

lib\util.js 26: let games = mongoDb().collection('games').aggregate([{ $group: { _id: '$createdAt', matches: { $addToSet: '$$ROOT' } } }, { $sort: { _id: -1 } }, { $skip: index }, { $limit: 1 }]).toArray()

lib\util.js 49: let rps = mongoDb().collection('rps').aggregate([{ $group: { _id: '$createdAt', rps: { $addToSet: '$$ROOT' } } }, { $sort: { _id: -1 } }, { $skip: index }, { $limit: 1 }]).toArray()

lib\util.js 87: let rps = mongoDb().collection('rps').aggregate([{ $group: { _id: '$createdAt', rps: { $addToSet: '$$ROOT' } } }, { $sort: { _id: -1 } }, { $skip: index }, { $limit: 1 }]).toArray()

Need to add as a first stage to the aggregation pipeline a check that the documents createdAt field is from the last 15 or so minutes (the bot allows to retrieve info about games 10 minutes into the past, plus a bit of a safe zone)

Also need to check if it's still needed to pass allowDiskUse:true like the error shows after limiting the $group stage to documents from the past 15 minutes