newWatcher never finish on Azure Cosmos DB for MongoDB

[guilhem]

Calling await MongoChangeStreamWatcher.newWatcher(CONNECTIONSTRING) never finish.

After debugging, it looks like await this.streamReady(watcher.changeStream) is falling in loop.

[casbin-bot]

@nodece @Zxilly @Shivansh-yadav13

[github-actions[bot]]

:tada: This issue has been resolved in version 1.1.0 :tada:

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot :package::rocket:

[Sefriol]

Interesting. Is there anything special with Azure Cosmos DB @guilhem? I have never seen this happen on MongoDB atlas or on a database hosted on Kubernetes cluster.

EDIT: It would be nice to know the database version as well. I have mainly worked on MongoDB 5 and 6.

[guilhem]

Interesting. Is there anything special with Azure Cosmos DB @guilhem? I have never seen this happen on MongoDB atlas or on a database hosted on Kubernetes cluster.

Yes, sadly, CosmosDB for MongoDB is quite different about change stream. I'm not an expert, but this page is giving some information: https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/change-streams

CosmosDB is emulating a 4.4 MongoDB protocol.

[Sefriol]

Hmm, yeah. That could be the problem. I originally wrote this in 4.8, but kinda quickly changed to 5 since it added more robust change stream support. I was originally planning on adding the support for individual changes that could be linked to insert, update and delete events, but decided to make this simple watcher first since delete events were not enabled by default.

[Sefriol]

@guilhem do you have chance to debug whether stream.cursor.startAtOperationTime exists in CosmoDB? In general we would ofc like to get a robust way to indicate whether stream is ready or not, but I do not think that exists yet.

One other option suggested would be to use resumeTokenChanged.

EDIT: There seems to be an init event as well, but I do not remember if I tried that already

[guilhem]

FTR, our watcher work with this configuration:

const watcher: Watcher = await MongoChangeStreamWatcher.newWatcher(
  AZURE_COSMOS_CONNECTIONSTRING,
  {
    dbName: COSMOS_CASBIN_DB_NAME,
    streamOptions: { fullDocument: "updateLookup" },
    callbackStreamCloseEvent: false,
    waitStreamReady: false,
    streamAgregator: [
      {
        $match: { operationType: { $in: ["insert", "update", "replace"] } },
      },
      { $project: { _id: 1, fullDocument: 1, ns: 1, documentKey: 1 } },
    ],
  }
);