Open BerndHardrock opened 3 years ago
hi @rwynn I'm working on the issue together with @BerndHardrock
In the meantime I was able to narrow down the issue. I've used ES javascript library to watch for changes and I'm getting the same error with the following $project config:
db.collection(collectionName).watch([
{
"$match": {
"operationType": {
"$in": ["insert", "update", "replace"]
}
}
},
{
"$project": {
"documentKey": false
}
}
], {"resumeAfter": resumeToken, "fullDocument": "updateLookup"});
I then followed an example of the watch from Azure CosmosDB docs and added the "ns": 1
to the $project config and now the error Change stream must be followed by a match and then a project stage
is gone:
"$project": {
"documentKey": false,
"ns": 1
}
Not sure what exactly "ns": 1
does but do you think you could implement this change? I'm unfortunately no golang specialist so can't properly create a PR for that.
Also, as CosmosDB doesn't support delete - can we disable the delete watch events through configuration?
Thanks a lot
Hi @nadworny @BerndHardrock ,
I've never tried monstache with CosmosDB. It seems that because of these limitations it would be a challenge.
Normally, you can send empty pipeline stages to the Watch API, but seems CosmosDB does not allow that. Monstache does allow you to customize the pipeline so we can work around the limitation with monstache configuration...
sample from Cosmos docs applied...
[[pipeline]]
script = """
module.exports = function(ns, changeStream) {
if (changeStream) {
return [
{ $match: { "operationType": { $in: ["insert", "update", "replace"] } } },
{ $project: { "_id": 1, "fullDocument": 1, "ns": 1, "documentKey": 1 } }
];
}
}
"""
However, I don't see a workaround for
The operationType and updateDescription properties are not yet supported in the output document.
Monstache uses gtm to listen to the change stream and operationType
is something that gtm uses.
We are trying to synchronise ElasticSearch with Azure CosmosDB through monstache.
Versions used:
Error:
Toml config:
In the Azure CosmosDB documentation we found the following regarding change streams: https://docs.microsoft.com/en-us/azure/cosmos-db/mongodb-change-streams?tabs=javascript
Can you confirm this is a CosmosDB issue and not a configuration issue from our side?