Support Mongo API with CosmosDB trigger

[asos-martinsmith]

MongoDB API does not support change feed but does support change streams.

Conceptually it seems that it ought to be possible to provide a change stream triggered function that operates much the same as the change feed triggered function (calls "GetChangeStreamTokens" and then stores a document per partition analogous to the leases stored by the change feed processor and updates this with the resume token as each batch is processed).

Is it on the roadmap to do this?

[bhagyshricompany]

It is possible to use MongoDB's change streams to trigger Azure Functions, but it would require some custom development work to implement. Currently, Azure Functions does not have built-in support for MongoDB change streams.

I am not aware of any specific roadmap for adding support for MongoDB change streams to Azure Functions. However, it is possible for developers to create their own custom triggers and bindings to integrate MongoDB change streams with Azure Functions.

You can use MongoDB driver to open a change stream on your MongoDB collection and then use the MongoDB change stream events to trigger your Azure function. You will need to handle the resume token and partitioning yourself, but this should be possible.

Alternatively, you can use other services such as Azure Cosmos DB that support change feed out of the box, which may be a better option if your use case requires minimal setup and minimal custom development.

[asos-martinsmith]

To be clear I am already using Cosmos DB - just the Mongo API flavour of it.

I'm not proposing that Microsoft write a generic MongoDB change streams processor - just one that works with their own product.

I imagine "GetChangeStreamTokens" is likely a proprietary command anyway as it is coupled to the Cosmos concept of partitions,

[asos-martinsmith]

And it is true that we can write our own but pushing this work for all customers to do individually vs providing parity with customers using the "No SQL" API seems somewhat sub optimal in terms of duplication of effort and forcing all customers to first learn about function app extension development in order to do something that NoSQL API customers get "out of the box"

[bhagyshricompany]

As you mention correct providing this functionality out of the box would make it more accessible for customers who may not have experience with function app extension development. @fabiocav please comment and validate on this

[asos-martinsmith]

Type of use case for this would be for keeping another copy of collection in synch partitioned differently (materialized view type of functionality) - or performing a full collection migration in an efficient way (spreading the load across all physical partitions so all use their RU budget).

The requirement for full collection migration crops up fairly often with Mongo API because often other product features are lacking - e.g. analytical store is incompatible with continuous backup so the only way of getting both simultaneously is maintaining two copies of the collection.

And also it is not possible to enable analytical store on restored backups so the only way of getting a restored backup into analytical store is to copy the whole thing to a new collection.

[bhagyshricompany]

@asos-martinsmith any more input you need pls update

[ghost]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

[asos-martinsmith]

@asos-martinsmith any more input you need pls update

Sorry I have no idea why you have tagged me. I have made a request and explained why I need it so the ball is with Microsoft

[bhagyshricompany]

for this feature will update in future as discussed

[ghost]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

[asos-martinsmith]

Please re-open this.

I have no idea who @bhagyshricompany is - but it doesn't look like they are affiliated with Microsoft so I'm not sure why they are getting involved in this and tagging as "Needs: Author Feedback" without any request for any feedback

[sasiteluk-8451]

@bhagyshricompany Can we re-open this issue, we need change feed support for MongoDB API

[asos-martinsmith]

@bhagyshricompany Can we re-open this issue, we need change feed support for MongoDB API

For anyone else interested in this issue I have just come across this repo. https://github.com/Azure/azure-functions-cosmosdb-extension#triggers

So looks like work is (maybe) being done on this - even though no activity for months and seems to currently be missing a lot of functionality that the no SQL API change triggers have (such as starting from a specific time).

Personally I would much rather see it integrated into this solution though so it can benefit from things like retry policies and other such code without having discrepancies in what is implemented.