CosmosDB change feed integration

[mariomeyrelles]

Usecase scenario

In order to leverage serverless event-oriented programming scenarios, it would be interesting to connect the change feed to the event grid.

[tomkerkhove]

So basically you want events for all Cosmos DB information in the change feed, is that correct summary?

[markjbrown]

@mariomeyrelles. I'd like to understand what use-cases and expectations you have with exposing Cosmos data plane events (change feed) to Event Grid.

For data plane operations firing Event Grid notifications seems to duplicate what customers already get with ChangeFeed itself. So it's utility beyond what customers already get with Change Feed is a hard sell internally to us. Change Feed already enables event-oriented scenarios so what does Event Grid provide that makes that better?

Another problematic consideration here is where you have high volume of data plane events in Cosmos. There are two issues here.

First, how would you handle a scenario where you had multiple consumers of ChangeFeed, event grid and another consumer. Assume the second consumer changed the data, this may have an impact for anyone consuming the Event Grid. This will improve once we implement a true oplog because ChangeFeed will point at an oplog lsn which will contain all changes, but for now this may be problematic.

Second, how do you handle high volume and scale out for EventGrid? Cosmos ChangeFeed does scale out over partition key ranges. Not sure how that would work with EventGrid. Any use of Event Grid provisioning would need to be tied to RU/s in Cosmos and I'm not clear how we would do that in providing a manged solution to customers.

Definitely appreciate your feedback. Thanks.

[tomkerkhove]

For data plane operations firing Event Grid notifications seems to duplicate what customers already get with ChangeFeed itself. So it's utility beyond what customers already get with Change Feed is a hard sell internally to us. Change Feed already enables event-oriented scenarios so what does Event Grid provide that makes that better?

For what it's worth, the difference is very simple - Change feed is a log-based pull-based model where we do the checkpointing. With Event Grid, we just subscribe for new events (and filter on certain criteria) and get triggered when we have to.

Event Grid doesn't allow to go back in time, which is fair for those workloads.

TLDR; Same data, different workloads & scenarios.

Second, how do you handle high volume and scale out for EventGrid? Cosmos ChangeFeed does scale out over partition key ranges. Not sure how that would work with EventGrid. Any use of Event Grid provisioning would need to be tied to RU/s in Cosmos and I'm not clear how we would do that in providing a manged solution to customers.

It does not perse need to be linked to RU/s of Cosmos, what if we just want to react to a new record being added, ie. a new customer? We might want to send an email based on that data or trigger another process.

Scaling of that section can be done by services like Azure Functions or so which are not linked to the Cosmos account itself.

[markjbrown]

and filter on certain criteria can you give me an example of what criteria?

It does not perse need to be linked to RU/s of Cosmos Actually it will need to be tied to Cosmos as we would need to account for the compute necessary to push the event on our end. After thinking about this, I believe the best way to do this is to simply add to the cost of a write so instead of a 1KB item costing 5 RU it would cost 6 RU, etc.

However, this still leaves open the question of scale on Event Grid side. If I push 50K messages per second from Cosmos does Event Grid just happily handle that or do I need to provision for that?

[SeanFeldman]

However, this still leaves open the question of scale on Event Grid side. If I push 50K messages per second from Cosmos does Event Grid just happily handle that or do I need to provision for that?

That's the beauty not EventGrid. As a publisher, you shouldn't be worried. It's a subscriber concern. And subscribers can choose how to handle the load. They could handle it real time by scaling out/up their processing or push those EG events into some messaging service (Event Hubs, Storage queue or Service Bus).

[markjbrown]

ok, good to know.

What about criteria is there to know about subscribing to events?

[SeanFeldman]

What about criteria is there to know about subscribing to events?

It depends on the scenarios, but there are multiple ways. Subscriptions can filter events based on the metadata in various ways. Some of it will depend on what metadata will be exposed by the particular service. More info here: https://docs.microsoft.com/en-us/azure/event-grid/event-filtering

[mariomeyrelles]

Hello all,

I proposed this integration because I have been working on a serverless and event-driven architecture on Azure. On my proof-of-concepts, CosmosDB is used as our event store. And the Change Feed is extremely useful because each new document is published as a Command or Event.

I already can do Azure Functions integration using the Change Feed binding. The idea is to avoid the creating of a new function and add coding to a problem that potentially can be solved without code.

Let me illustrate my use case with a use business use case from a fictitious example:

1. User A buys Credits
2. The system validates and issues a command: IncreaseCreditsCommand.
3. The system somehow reacts to this command and generates a new Event: CreditsIncreasedEvent
4. We add this event to our Events Collection on CosmosDB.

So far so good. But in order to let multiple consumers react to this event independently, actually we can use existing Azure Functions or Web Jobs to:

1. Manually send the event to each listener directly (potentially creating a different function for each kind of consumer) or
2. Put the message on Azure Service Bus / Event Hubs or
3. Send the message to Event Grid

Option 3 is better for reactive programming and not, massive messaging. In this specific use case, as we are using CosmosDB as an event source, it's acceptable to initially only consider documents added to the collection. Edits and deletes shouldn't happen in this kind of architecture.

Simplying option 3 (sending feed directly to Event Grid) can lead to more development productivity and can overcome a current issue on Change Feed - we can't filter the feed to only react to commands we are interested. And of course, we would gain the ability to easily add and remove event listeners.

[markjbrown]

@mariomeyrelles thank you for the additional details.

I need to point out that the mechanism for us to publish to Event Grid itself would be driven by ChangeFeed. It's important to call this out because the way you are describing how this should work would require us to modify how ChangeFeed works to provide the functionality you're asking for on two capabilities:

1. Being able to filter on the type of operation (in this case, inserts only). This is currently not supported but the ability to distinguish CRUD operations is on our road map when ChangeFeed becomes more op-log like. That said, we would not enable the filtering on our end but customers could on the subscribing side. We would still publish every CRUD operation (and charge for it, see below).
2. One message per CRUD operation. This might be more problematic as ChangeFeed does polling on the leader replica within each partition. But to even enable EventGrid publishing we'd need to host ChangeFeed ourselves so can probably work around this and just handle the polling internally and publish each CRUD operation if multiple items are in the ChangeFeed.

I do have some more comments/questions around all this.

What amount of latency is acceptable for this sort of thing? Milliseconds, Seconds, Minutes?

We would need to host some compute to read ChangeFeed and publish to EventGrid which has cost implications that we would need to account for (charge for). Since our currency is RU/s we would likely charge additional RU/s to publish to Event Grid, regardless of whether anyone read from it. Is that a deal-breaker or otherwise a problem? My guess is we would add somewhere between 1-3 RU/s but won't know for sure until we figure out COGS to publish events.

Not sure how Subject Filtering would work. We are not blob storage. The Subject would likely be CosmosAccount/database/container/partitionkeyrange. I would not expect there to be any way of including properties from the data itself in Subject Filtering.

On a related Note, the data attribute in the message would not include the document or item but partition key and id value to it which the reader would use a point read on Cosmos to get the actual data. The reason is because an item can be up to 2MB in size.

Would love to get feedback or questions about any of this.

Thanks!

[mariomeyrelles]

@markjbrown Good morning,

Let me try to answer:

1) I understand that currently we can`t change how Change Feed operates. I was suggesting to consider only inserts because I thought that it would be easier. It's not an issue to receive all data from Change Feed. No problem on this point.

2) Yes, it is really important to have one message per item inserted item on the Cosmos' Collection. The business value of each event is very important. The idea of Event Grid acting as a host of a Change Feed client seems adequate to me as a customer.

3) Latency: actually, using function, we start to see events coming in less than 30 seconds. But I do admit that for many use cases, we can accept minutes of latency.

4) Regarding RU/s cost, I don`t this as a problem as long as the number of RU/s remains low as in your example above.

5) Subject filtering is very important. If we publish the message in an specific format (for example, Event Grid Schema) on the Cosmos collection, can this help on filtering? I would be willing to impose an Event format on my systems if this helps. Actually I use the Event name as the partitition key - for simple cases, partition name filtering can suffice.

6) To very honest, an event in an event-oriented architecture shouldn't be big. And we shouldn't need to go to Cosmos to get the details of the event, since it breaks the whole philosophy of being reactive on the cloud. The event published should contain the maximum information needed to be processed locally. See the example below of a typical event. I understand that are limitations on Event Grid (64kb / 1MB) and Cosmos (2MB). I`m not sure what to do here. Is there some way to impose a document size restriction on a given collection?

Kind Regards, Mário

{
    "Reason": "Value higher than $7500",
    "RequestId": "2c3fdc08-5001-4fd8-ba85-4678f44f7371",
    "EventDateTimeUtc": "2019-12-20T18:05:22.1934497Z",
    "AggregateId": "637124619076512373",
    "AggregateName": "CreditsTransfer",
    "Payload": {
        "id": "637124619076512373",
        "Sender": {
            "Id": "937",
            "FullName": "Neida XXXXXXX",
            "DocumentNumber": "XXX.230.905-XX"
        },
        "Receiver": {
            "Id": "522",
            "FullName": "XXXXX Olerdatter",
            "DocumentNumber": "XXX.865.462-XX"
        },
        "Amount": 8814.39,
        "TransferPurpose": "Donation",
        "DateScheduledUtc": "2019-12-21T19:30:19.5700354Z",
        "DateCreatedUtc": "2019-12-20T18:05:07.6512172Z"
    },
    "EventName": "CreditsTransferSentToManualValidationEvent",
    "id": "b60be146-3aeb-49b7-b67f-b46833893e97",
    "_rid": "WcQuAMJ+vLhMFgAAAAAAAA==",
    "_self": "dbs/WcQuAA==/colls/WcQuAMJ+vLg=/docs/WcQuAMJ+vLhMFgAAAAAAAA==/",
    "_etag": "\"2e00bbe1-0000-0300-0000-5dfd0d620000\"",
    "_attachments": "attachments/",
    "_ts": 1576865122
}

[markjbrown]

I see #5 and #6 where there is likely to be friction on this.

Consider a blob storage example. EventGrid here pushes an event that provides meta data on the new blob object and provides the URI to the new or updated blob but not the blob itself. You'd still need to do a storage transaction to fetch the blob.

Data plane for Cosmos is the same. Events to Cosmos DB would/could provide the partition key and id to the data but I cannot see any scenario where we would be able to publish the data itself.

With regards to payload size, there is no way to limit this.

With regards to putting anything "custom" in "data" property, I don't see any examples where customers can create custom schema's and have a resource provider publish events that conform to a per customer custom schema. If I'm wrong about this please let me know. I'd like to take a look. I can say that if this is possible it is going to be fairly expensive for us to process this and publish but we can get to that bridge if/when we need to.

btw, I really appreciate everyone providing comments on this. I need this level of detail to even have discussions internally on how or if we do this so keep it coming.

Thanks!

[mariomeyrelles]

Good morning again,

I trully undestand your side and it makes sense. The "magic" of reactivity we already can achieve using the combination of Cosmos -> Functions -> Event Grid -> WebHook is very handy and we can build cool reactive applications today. The webhook skeleton can be written like this:

        [HttpPost]
        public async Task<IActionResult> Post()
        {
            using (var reader = new StreamReader(Request.Body, Encoding.UTF8))
            {
                var jsonContent = await reader.ReadToEndAsync();

                // Check the event type.
                // Return the validation code if it's 
                // a subscription validation request. 
                if (EventTypeSubcriptionValidation)
                {
                    return await HandleValidation(jsonContent);
                }
                else if (EventTypeNotification)
                {
                    return await HandleGridEvents(jsonContent);
                }

                return BadRequest();                
            }
        }

Using a reactive approach we can:

1) Remove completely CosmosDB dependencies, creating simpler (micro)services. 2) Avoid extra cost querying Cosmos again and thus, avoiding concerns about how should we write a good query. 3) Simplify the controllers, avoid dependency injection of external dependencies 4) The reaction can be written in any language.

Thinking a little bit outside the box, in the general case, to be able to implement this kind of functionality, we would probably need a new feature on Cosmos: something like typed collections where we would restrict the data format and size using some kind of schema/schema validation during document creation on Cosmos. Having something like this, we then would be able to make assumptions on the data format of each document so we would apply filtering on Event Grid itself. In my use case, it would be ok to force my data to be on a given schema you provide. And for me it would ok to have a fixed schema like Event Grid / Cloud Events schema.

Kind Regards, Mário

[tomkerkhove]

Not sure how Subject Filtering would work. We are not blob storage. The Subject would likely be CosmosAccount/database/container/partitionkeyrange. I would not expect there to be any way of including properties from the data itself in Subject Filtering.

For a lot of scenarios this subject filtering would already be great if you ask me, we already know what is changed, if we need more granular filtering we can take it from there which leaves us already with a very narrow dataset rather than drinking from the firehose

Yes, it is really important to have one message per item inserted item on the Cosmos' Collection. The business value of each event is very important. The idea of Event Grid acting as a host of a Change Feed client seems adequate to me as a customer.

I'm agreeing with @mariomeyrelles here.

Data plane for Cosmos is the same. Events to Cosmos DB would/could provide the partition key and id to the data but I cannot see any scenario where we would be able to publish the data itself.

For me this is fine to be honest, certainly better than not having anything. The reason is that we know that the "Customer" document has been added or updated so we can look it up if we have to. It's not ideal, but you can't shove all the data in there as those would 1) overload the message size 2) reduce performance 3) overkill.

Sometimes it's just enough to know that something happened.

We would need to host some compute to read ChangeFeed and publish to EventGrid which has cost implications that we would need to account for (charge for). Since our currency is RU/s we would likely charge additional RU/s to publish to Event Grid, regardless of whether anyone read from it. Is that a deal-breaker or otherwise a problem? My guess is we would add somewhere between 1-3 RU/s but won't know for sure until we figure out COGS to publish events.

Now let's take a step back.

With change feed, a log-based model is chosen which basically gives you a fully ordered history, which is persisted, of what happened and when in the Cosmos DB account. Log-based models are using a pull-based approach where the consumer has to keep on checking if there is new change events in the feed based on the previous location cursor.

This works fine for a lot of scenarios, but not every scenario because : 1) it's pull based which is intensive and requires long-running/scheduled processes that dispatch work 2) you have state since you need to keep track of the cursor and sync it across multiple runtimes

With Event Grid, it's basically the same with what changefeed leverages but the consumption model is different: 1) Push-based model - We just subscribe and grid will trigger us, allowing us to use event-driven tech 2) No state to manage 3) No persisted data, but for some scenarios that's fair.

But in the end; both provide same data, only the way is just different.

How I see it, I'd see it like this:

Heck, you could even use Event Grid to feed your changelog:

Obviously this is higly simplified but I hope it just helps prove the point that other than the data that's in there, they are fully seperated.

[markjbrown]

Thanks @tomkerkhove for the detail and clarification.

With regards to whether this is a push or pull model, we still need to account for the compute required to publish the event. This is why I'm suggesting there would be an RU charge to turn this on.

I can't say for sure we would do this but I need to allow for the possibility and want to measure whether this is a deal breaker or not.

[tomkerkhove]

That's a fair point but how would that work with regards to change feed then? What if I want Grid and no change feed? Do I still need to pay more?

If I may be frank, I'd expect events to always be there and free for charge. This feels like a platform thing to me, not a feature. You should also think about this as a potential differentiater with respect to your competition.

[markjbrown]

@mariomeyrelles

Everything you suggest there sounds good but to be clear though all of that would be out of scope for what we would build to enable EventGrid support in Cosmos.

The straightest route to getting EventGrid would be a feature set that reflects and respects how the service is designed and works today. Anything that would require substantial changes would be a non-starter. It would also need to have the smallest operation COGS because customers do not want to pay a lot of money for this.

Thanks to you and everyone else for your thoughts and suggestions. Keep them coming and please feel free to add others here to comment.

[markjbrown]

@tomkerkhove change feed is already free. You only pay if you use it (read from it) so if you had events and not change feed you would not pay twice.

With regards to Event Grid. The way this would likely work is customer would first enable it on our end per collection. Once enabled, we'd publish events and add some RU to each non-read operation. To be clear, I'm not saying this may never be free. It's possible, but given that Change Feed requires some RU to consume to see if there are new items and then additional RU to then read the data off of it there is little to no chance we would first build a feature that would have negative revenue impact for us. I just need to be completely clear on this point. The only way that would ever happen is if the customer demand for the feature was so overwhelming that it creates net new revenue opportunities for us.

Thinking about this as a differentiation with our competition is certainly something I would like to think is in play here but I do not see customers choosing Cosmos vs Dynamo vs Atlas because of this. I would need to see evidence of this and I'm not seeing it yet.

When it comes to me making a case for EventGrid internally the thing that will matter most is 1) what is the revenue potential (i.e. how many customers are willing to use it and at what price) and 2) what is the cost to build and operate. As a product manager anywhere at Microsoft, this is what we need to bring to the table when we work out what we put on our road map and commit to build. Anything you can do to help me get clarity on these two pieces helps me build this case.

Thanks!

[tomkerkhove]

The only way that would ever happen is if the customer demand for the feature was so overwhelming that it creates net new revenue opportunities for us.

Sometimes they don't know what they are missing until they've been able to use it and see its true power - So if it's free or not, that's fine for me!

Like you say, this can certainly change over time as well.

Thinking about this as a differentiation with our competition is certainly something I would like to think is in play here but I do not see customers choosing Cosmos vs Dynamo vs Atlas because of this. I would need to see evidence of this and I'm not seeing it yet.

Same here, they might not be doing that because it's not there yet, but if it is I think this will certainly help you win customers.

Simplest example is - "Aha a new customer lead doc was created, let's send them a customized welcome email".

Thanks for considering @markjbrown!

[p-bojkowski]

To bring some context to this discussion, here are the proof-of-concepts from @mariomeyrelles that he mentioned:

https://dev.to/mariomeyrelles/serverless-event-driven-architecture-on-azure-a-worked-example-part-1-481e

https://dev.to/mariomeyrelles/serverless-event-driven-architecture-on-azure-a-worked-example-part-2-3im6

Nice work @mariomeyrelles 👍👍👍 Will you proof-of-concepts available soon on GitHub and what about part 3 of the series? :)

[markjbrown]

@p-bojkowski thanks for posting that blog post by @mariomeyrelles. That's a great set of blog posts there on CQRS and Cosmos.

Mario, I'm going to reach out to you directly. Have some ideas.

Thanks.

[markjbrown]

@mariomeyrelles. It looks like you've already left MS. Would still like to chat with you. DM me on twitter if you can, @markjbrown.

thanks.

[mariomeyrelles]

Hello @markjbrown

I am available on Twitter (@mariomeyrelles), LinkedIn and e-mail. Please feel free to contact me. I was away for some time and now I am back to work. I am actually working for a company in Dublin :)

Kind Regards, Mário

[mariomeyrelles]

@p-bojkowski I have the code ready, but I still need to write part 3 (working with Events) and part 4 (process managers using Durable Functions). I need to find the time... I got a new job and a new baby.