Closed simllll closed 3 years ago
JSON.stringify(mongooseModelResult[0].someProperty) returns e.g. { a: 1 }, but Object.values(mongooseModelResult[0].someProperty) returns something complete different? This reminds me of a memory issue in good old c? This issue is gone if I use .lean() for the query... still doesn't seem right. The related has this structure:
Object.values would return the values in the key value pair so instead of returning {a:1}
it will just return 1.
Please provide a script that demonstrates these issues you have
I can comment on this. The cursor doesn't clean up after itself, so if you have millions of data points, it will eventually use up too much:
Here is a simplified version of my script:
const fs = require('fs')
const mongoose = require('mongoose')
const log = require('single-line-log').stdout
const PlacePointModel = require('./schema/PlacePoint.js').PlacePointModel
let total = 0
async function BUILD () {
await mongoose.connect('mongodb://127.0.0.1:27017/data', {
useNewUrlParser: true,
useUnifiedTopology: true,
useFindAndModify: false,
useCreateIndex: true
})
const db = mongoose.connection
const pointCursor = PlacePointModel.find().populate('relations').cursor()
for (let point = await pointCursor.next(); point != null; point = await pointCursor.next()) {
log(++total)
// pull
const { ... } = point
// manipulate
...
// store
fs.appendFileSync('./data.json', JSON.stringify(feature) + '\n')
}
db.close()
}
BUILD()
version of node: 12.22.3 mongoose: 5.13.2
JSON.stringify(mongooseModelResult[0].someProperty) returns e.g. { a: 1 }, but Object.values(mongooseModelResult[0].someProperty) returns something complete different? This reminds me of a memory issue in good old c? This issue is gone if I use .lean() for the query... still doesn't seem right. The related has this structure:
Object.values would return the values in the key value pair so instead of returning
{a:1}
it will just return 1.Please provide a script that demonstrates these issues you have
Sorry, I meant Object.entries, but the main culprit here is that it reported values/keys/content which is not part of the original object.
Simplified version of a scirpt that produces this errornous output:
import 'reflect-metadata';
import { DBUserModel } from '@hokify/user-data';
import { DBMailBounceModel } from '@hokify/messaging-servicefeedback';
import { ObjectId } from 'mongodb';
import { MongooseConnection } from '../src';
(async () => {
await MongooseConnection.connect();
// fetch a user object
const selUser = await DBUserModel.findOne({
_id: new ObjectId('60e61c0dd8c15500190e031b')
}).exec();
// fetc mail bounces
await DBMailBounceModel.findOne({ email: selUser.general.email })
.exec()
.then(bounce => {
selUser.mailBounce = bounce;
});
console.log('selUser.mailBounce', selUser.mailBounce);
console.log('selUser.privacy', selUser.privacy);
console.log('JSON.stringify(selUser.privacy)', JSON.stringify(selUser.privacy));
console.log('Object.keys(selUser.privacy)', Object.keys(selUser.privacy));
console.log(
'Object.entries(selUser.privacy)',
Object.entries(selUser.privacy).map(
([key, _value]) => `privacy[${key}]` + `: ${selUser.privacy[key] ? 'Exists' : ' FAIL'}`
)
);
})();
and the output of this scirpt:
selUser.mailBounce {
amount: 1,
_id: 60ebe31dd4b4c9cf128712cb,
email: 'ewald_dresbach@yahoo.it',
__v: 0,
createdAt: 2021-07-12T06:37:17.224Z,
lastError: 'Permanent, General / failed:smtp; 552 1 Requested mail action aborted, mailbox not found',
updatedAt: 2021-07-12T06:37:17.224Z
}
selUser.privacy { user_register: 2021-07-07T21:26:37.934Z }
JSON.stringify(selUser.privacy) {"user_register":"2021-07-07T21:26:37.934Z"}
Object.keys(selUser.privacy) [
'$init',
'mailBounce',
'user_register',
'user_application',
'user_active_sourcing',
'company_register',
'company_posting',
'company_application',
'company_club',
'company_digital_content'
]
Object.entries(selUser.privacy) [
'privacy[$init]: Exists',
'privacy[mailBounce]: Exists',
'privacy[user_register]: Exists',
'privacy[user_application]: FAIL',
'privacy[user_active_sourcing]: FAIL',
'privacy[company_register]: FAIL',
'privacy[company_posting]: FAIL',
'privacy[company_application]: FAIL',
'privacy[company_club]: FAIL',
'privacy[company_digital_content]: FAIL'
]
look at selUser.privacy and JSON.stringify => all good but look at Object.keys or Object.values => it returns weird other stuff (e.g. it includes mailBoucne?!). The mongoose model defintion part of the user object for privacy looks like this:
privacy: {
user_register: Date,
user_application: Date,
user_active_sourcing: Date,
company_register: Date,
company_posting: Date,
company_application: Date,
company_club: Date,
company_digital_content: Date
},
there is no such thing as mailBounce.. furthermore if I add .lean() to the user query, all works correctly.
Version checks:
so it seems the bug got introduced in 5.12.13.
not quite sure if this is bug is directly related to the e.g. non freeing cursors issues? is this somehow related? @CraigglesO can you verify if the cursor issue also got introduced in 5.12.13?
Please include your models @simllll
@CraigglesO Please define what feature is suppose to be along with the {...} = point
as well as your model
Please include your models @simllll
const userSchema = new Schema({
general: {
email: { type: String, lowercase: true, trim: true },
},
privacy: {
user_register: Date,
user_application: Date,
user_active_sourcing: Date,
company_register: Date,
company_posting: Date,
company_application: Date,
company_club: Date,
company_digital_content: Date
}
});
const DBUserModel = model('UserMongooseBug', userSchema);
await DBUserModel.create({
general: {
email: 'test'
},
privacy: {
user_register: new Date()
}
});
const mailBounceSchema = new Schema(
{
email: {
type: String,
unique: true,
required: true,
dropDups: true
},
amount: { type: Number, default: 0 },
lastError: Schema.Types.Mixed
},
{
timestamps: true
}
);
const DBMailBounceModel = model('MailBounceMongooseBug', mailBounceSchema);
await DBUserModel.create({
email: 'test',
amount: 1,
lastError: {
bla: true
}
});
const mongoose = require('mongoose');
const userSchema = new mongoose.Schema({
general: {
email: { type: String, lowercase: true, trim: true },
},
privacy: {
user_register: Date,
user_application: Date,
user_active_sourcing: Date,
company_register: Date,
company_posting: Date,
company_application: Date,
company_club: Date,
company_digital_content: Date
}
});
const DBUserModel = mongoose.model('UserMongooseBug', userSchema);
const mailBounceSchema = new mongoose.Schema(
{
email: {
type: String,
unique: true,
required: true,
dropDups: true
},
amount: { type: Number, default: 0 },
lastError: mongoose.Schema.Types.Mixed
},
{
timestamps: true
}
);
const DBMailBounceModel = mongoose.model('MailBounceMongooseBug', mailBounceSchema);
(async () => {
await mongoose.connect('mongodb://localhost:27017/test', { useNewUrlParser: true, useUnifiedTopology: true });
await mongoose.connection.dropDatabase();
let general = await DBUserModel.create({
general: {
email: 'test'
},
privacy: {
user_register: new Date()
}
});
let entry = await DBUserModel.create({
email: 'test',
amount: 1,
lastError: {
bla: true
}
});
// fetch a user object
const selUser = await DBUserModel.findOne({
_id: general._id
}).exec();
// fetc mail bounces
await DBMailBounceModel.findOne({ email: selUser.general.email })
.exec()
.then(bounce => {
selUser.mailBounce = bounce;
});
console.log('selUser.mailBounce', selUser.mailBounce);
console.log('selUser.privacy', selUser.privacy);
console.log('JSON.stringify(selUser.privacy)', JSON.stringify(selUser.privacy));
console.log('Object.keys(selUser.privacy)', Object.keys(selUser.privacy));
console.log(
'Object.entries(selUser.privacy)',
Object.entries(selUser.privacy).map(
([key, _value]) => `privacy[${key}]` + `: ${selUser.privacy[key] ? 'Exists' : ' FAIL'}`
)
);
})();
Is this intended that you have added the can't reproduce label @IslandRhythms ? If i run your script with mongoose ^5.13.3:
JSON.stringify(selUser.privacy) {"user_register":"2021-07-20T10:24:27.938Z"}
Object.keys(selUser.privacy) [
'$init',
'mailBounce',
'user_register',
'user_application',
'user_active_sourcing',
'company_register',
'company_posting',
'company_application',
'company_club',
'company_digital_content'
]
Object.entries(selUser.privacy) [
'privacy[$init]: Exists',
'privacy[mailBounce]: FAIL',
'privacy[user_register]: Exists',
'privacy[user_application]: FAIL',
'privacy[user_active_sourcing]: FAIL',
'privacy[company_register]: FAIL',
'privacy[company_posting]: FAIL',
'privacy[company_application]: FAIL',
'privacy[company_club]: FAIL',
'privacy[company_digital_content]: FAIL'
]
I run with ts-node test.ts
, mailBounce popps up inside privacy?
node v16.4.0
mongo v4.4.1 (via mongos)
My bad, missed the version number
So actually looking at it, the mailbounce key does exist but it is null
selUser.mailBounce null
selUser.privacy { user_register: 2021-07-20T17:17:25.403Z }
JSON.stringify(selUser.privacy) {"user_register":"2021-07-20T17:17:25.403Z"}
Object.keys(selUser.privacy) [
'$init',
'mailBounce',
'user_register',
'user_application',
'user_active_sourcing',
'company_register',
'company_posting',
'company_application',
'company_club',
'company_digital_content'
]
Object.entries(selUser.privacy) [
'privacy[$init]: Exists',
'privacy[mailBounce]: FAIL',
'privacy[user_register]: Exists',
'privacy[user_application]: FAIL',
'privacy[user_active_sourcing]: FAIL',
'privacy[company_register]: FAIL',
'privacy[company_posting]: FAIL',
'privacy[company_application]: FAIL',
'privacy[company_club]: FAIL',
'privacy[company_digital_content]: FAIL'
]
The fact that our outputs don't line up usually means there is something wrong on your end.
this is selUser {
general: { email: 'test' },
privacy: { user_register: 2021-07-20T17:22:20.694Z },
_id: 60f7064c2921a2451c6d2a23,
__v: 0
}
Exactly, but this is wrong ;-) mailbounce should exists on "selUser", but not under privacy. it is set to the "selUser.mailBoucne" not to "selUser.privacy.mailbounce" look at objects.keys of selUser.privacy or object.entires of seluser.privacy.. it both shows "mailBounce" (which is not part of "privacy" in any way). If you output it via json.stringify, output is fine.
@simllll are you setting a batchSize
on your cursors? Your stack trace makes it look like there's a huge batchSize
.
Thanks for pointing out the privacy.mailBounce
issue, we fixed that in 62b7b9c
Yeah, right.. the script that breaks has a batchSize set to 150000. Do you think setting it smaller would solve this issue?
Yep I think the call stack size exceeded error is due to #9366, which made populate work with batchSize
rather than populating one at a time. Specifically, it looks like we need to make sure to clear the call stack here to avoid putting 150k calls on the stack.
I think a better option would be to cap the batch size we use for populate()
to 1000 or so. Populating 150k documents at a time would likely not work very well.
Intersesting, but in this case I'm not populating anything. Only going through the documents as they are (I only can't use lean because I need some information about the model), also in this case we are talking about very small documents where a high batchSize actually has a very good performance impact.
That said, I'm not 100% sure which query is causing this, but in general we are not using populate (there where only some queries left that used mongoose's populate, that we changed in the meantime too ..), as we have our own implementatoin / cache layer to do population,.. which makes my guess stronger that this is not directly related to this 3rd issue. But it's the cause of the 2nd one pretty sure :)
@simllll in v5.13.6 we'll clear the stack every 1000 docs, so this stack overflow shouldn't happen anymore. Let me know if this issue persists.
Do you want to request a feature or report a bug? bug
What is the current behavior? there are some other issues out there with maximum call stack size exceeded, but I#m unsure if they are related to this one:
If the current behavior is a bug, please provide the steps to reproduce. I cannot really reproduce it, as it seems it only happens from time to time, but it crashes the whole node process after that. I see following strange behaviours:
with one data point set:
after the mongoose result, I add some more data to the object, in this case I added to the root object "user.mailBounces = {...}". when I use Object.values of user.privacy I get also the mailBounces values returned?
I put all these 3 issues into one ticket, as I hardly believe this is all related with virtuals somehow?
What is the expected behavior? no errors, I just recently upgraded from "mongoose": "5.10.19", to "mongoose": "^5.13.2", ..there were no issues with 5.10.19.
What are the versions of Node.js, Mongoose and MongoDB you are using? Note that "latest" is not a version. mongoose: 5.13.12 node: v16.4.0 mongodb: 4.4.1