Automattic / mongoose

MongoDB object modeling designed to work in an asynchronous environment.
https://mongoosejs.com
MIT License
26.96k stars 3.84k forks source link

Mongoose 6.2.3 breaks behaviour used since 5.12.2 #11445

Closed SK-FComputer closed 2 years ago

SK-FComputer commented 2 years ago

Do you want to request a feature or report a bug? bug

What is the current behavior?

If the current behavior is a bug, please provide the steps to reproduce. After updating to typegoose 9.7.0 and mongoose 6.2.3 (Works fine in 9.6.0 and 6.2.0) i get the error:

Error [MongoNotConnectedError]: MongoClient must be connected to perform this operation
    at getTopology (D:\project\node_modules\mongoose\node_modules\mongodb\src\utils.ts:451:9)
    at Db.createCollection (D:\project\node_modules\mongoose\node_modules\mongodb\src\db.ts:261:18)
    at NativeConnection.createCollection (D:\project\node_modules\mongoose\lib\connection.js:404:11)
    at NativeConnection.Connection.onOpen (D:\project\node_modules\mongoose\lib\connection.js:635:10)
    at NativeConnection.wireup (D:\project\node_modules\mongoose\lib\drivers\node-mongodb-native\connection.js:105:13)
    at Object.onceWrapper (events.js:519:28)
    at NativeConnection.emit (events.js:400:28)
    at NativeConnection.emit (domain.js:470:12)
    at NativeConnection.set (D:\project\node_modules\mongoose\lib\connection.js:126:12)
    at NativeConnection.set (D:\project\node_modules\mongoose\lib\connection.js:119:23)

I have a master process that starts workers with the library https://threads.js.org/. Before updating i had to create a new connection (DB class) In the child worker, such that i could use the typegoose model functions.

Code Example

DB (other DB connections here too, but truncated for example):

import { connect } from 'mongoose';
import { config } from './config';

export default class DB {
    constructor(public entities: any[] = []) {
        config;
    }

    async init(
        createMongoDB: boolean = true,
        debug: boolean = false
    ) {
        try {
            if (createMongoDB ?? true) {
                await connect(process.env.DB_MONGO_CONN);
                console.log('MongoDB Connected');
            }
        } catch (error) {
            console.error(error);
            throw error;
        }
    }
}

Master process:

const worker = await spawn(
            new Worker(workerPath, {
                resourceLimits: {
                    maxOldGenerationSizeMb: 32_768,
                },
            }),
            { timeout: 30_000 }
        );
const workerResult = await worker(data);

Child worker

import { connection } from 'mongoose';
import { expose } from 'threads/worker';
import DB from '../../database';
import { ITables } from '../../models';
import { ProductModel } from '../../odm';

async function processData(data: ITables['LAGKART']['data']) {
    try {
        await new DB().init(true);
        const updates = [];
        for (const val of data) {
            if (
                val.SYS_CHANGE_OPERATION === 'I' ||
                val.SYS_CHANGE_OPERATION === 'U'
            ) {
                updates.push({
                    updateOne: {
                        filter: { lxbenummer: val.LXBENUMMER },
                        update: val.data,
                        upsert: true,
                    },
                });
            } else {
                updates.push({
                    deleteOne: {
                        filter: { lxbenummer: val.LXBENUMMER },
                    },
                });
            }
        }
        const update = await ProductModel.bulkWrite(updates);

        if (update.result.writeConcernErrors.length) {
            throw update.getWriteConcernError();
        } else if (update.result.writeErrors.length) {
            throw update.getWriteErrors();
        }
    } catch (error) {
        throw error;
    } finally {
        try {
            await connection.close();
        } catch (error) {
            console.error('Connection already closed!');
        }
    }
}

expose(processData);

ProductModel

import { getModelForClass, prop } from '@typegoose/typegoose';
import { connection, Types } from 'mongoose';
class Product {
    _id: Types.ObjectId;

    @prop({ required: true, select: false })
    dataset: string;
}

const productDb = connection.useDb('dbName', { useCache: true });

const ProductModel = getModelForClass(Product, {
    existingConnection: productDb,
});

export { Product, ProductModel };

tsconfig.json

{
    "compilerOptions": {
        "baseUrl": "./src",
        "outDir": "./dist",
        "allowJs": false,
        "sourceMap": true,
        "moduleResolution": "node",
        "removeComments": true,
        "target": "ES2020",
        "module": "CommonJS",
        "experimentalDecorators": true,
        "emitDecoratorMetadata": true,
        "noImplicitAny": true,
        "esModuleInterop": true
    },
    "include": ["./src/**/*"],
    "exclude": ["node_modules", "**/node_modules/*"]
}

What is the expected behavior? I expect it to not throw an error. Worked since typegoose 8.0.0-beta.2 and mongoose 5.12.2. Only with latest versions this breaks,

I suspect it has to do with the import "order" since: After trial and error i can see that using dynamic import instead of import { ProductModel } from '../../odm'; i don't get this error message, but this hasn't been a problem since start of project. Last version of typegoose + mongoose breaks this behaviour and i can't seem to find out why.

What are the versions of Node.js, Mongoose and MongoDB you are using? Note that "latest" is not a version.

SK-FComputer commented 2 years ago

Probably occurs because of this

So i guess as @hasezoey pointed out here (https://github.com/typegoose/typegoose/issues/672#issuecomment-1049030419), there has been an update to useDb (https://github.com/Automattic/mongoose/commit/ad01d5e4ca6d53df156b29662b9cfbc9638507e5).

Why this breaks old behaviour i don't understand, and my current only solution is using dynamic imports. This is not feasible in all situations..

SK-FComputer commented 2 years ago

IS NOT A TYPEGOOSE ISSUE, tested with typegoose@9.7.0 and mongoose 6.2.1 and confirmed to work. Only upgrade of mongoose breaks this behaviour.

Minimal reproduceable example: https://github.com/SK-FComputer/mongoose-useDb-error

In src/worker.ts if DB() init is NOT used:

MongoDB Connected
Master update:  {
  title: 'Master',
  productID: 1,
  _id: new ObjectId("62175c1a6139f021165f602e"),
  __v: 0
}
Error [MongooseError]: Operation `products.insertOne()` buffering timed out after 10000ms
    at Timeout.<anonymous> (C:\Users\User\Desktop\mongoose-error\node_modules\mongoose\lib\drivers\node-mongodb-native\collection.js:151:23)
    at listOnTimeout (internal/timers.js:557:17)
    at processTimers (internal/timers.js:500:7)
Closing MongoDB connection master..

In src/worker.ts if DB() init is used:

MongoDB Connected
Master update:  {
  title: 'Master',
  productID: 1,
  _id: new ObjectId("62175804f97d81c206ce2ab3"),
  __v: 0
}
MongoNotConnectedError: MongoClient must be connected to perform this operation
    at getTopology (C:\Users\User\Desktop\mongoose-error\node_modules\mongodb\src\utils.ts:451:9)
    at Db.createCollection (C:\Users\User\Desktop\mongoose-error\node_modules\mongodb\src\db.ts:261:18)
    at NativeConnection.createCollection (C:\Users\User\Desktop\mongoose-error\node_modules\mongoose\lib\connection.js:404:11)
    at NativeConnection.Connection.onOpen (C:\Users\User\Desktop\mongoose-error\node_modules\mongoose\lib\connection.js:635:10)
    at NativeConnection.wireup (C:\Users\User\Desktop\mongoose-error\node_modules\mongoose\lib\drivers\node-mongodb-native\connection.js:105:13)
    at Object.onceWrapper (events.js:519:28)
    at NativeConnection.emit (events.js:400:28)
    at NativeConnection.emit (domain.js:470:12)
    at NativeConnection.set (C:\Users\User\Desktop\mongoose-error\node_modules\mongoose\lib\connection.js:126:12)
    at NativeConnection.set (C:\Users\User\Desktop\mongoose-error\node_modules\mongoose\lib\connection.js:119:23)
IslandRhythms commented 2 years ago

What file do I need to run? sync-data or worker?

SK-FComputer commented 2 years ago

@IslandRhythms sync-data.ts, should see output that it can write from master but not from worker Also removing the DB().init() in worker, won't use connection from master.

vkarpov15 commented 2 years ago

Fix will be in v6.2.7. We haven't been able to come up with a test case that reproduces this issue in our test suite, but I confirmed that 5bdd836 fixes the issue in your repo :+1:

SK-FComputer commented 2 years ago

Thank you very much! @vkarpov15