Event Sourced State has 2k events in it causing Out of Memory Error

[cross311]

Had an infinite add event to stream issue due to listening to cosmos change feed and not marking event in the state as handled, so it kept putting the same event back on the state. Eventually, it got to the point where it has 2.4K events on the event stream, which then when CC reads in causes an Out Of Memory Crash.

I am wondering if there should be some sort of warning / error that cookie cutter should throw when state gets to big, number of evens or total memory used for a stream. And the error reports the stream that caused the issue, else it is kinda hard to track down which stream read caused the out of memory?

Another possibility is to warn if source event making new event is exactly the same and the source even is from the event stream, to catch infinite event adds like this? Happens when the service listens to its changefeed and also listens to another Kafka topic for an event then puts that event directly on its own stream with no real changes. You can see in cosmos the source metadata is stream id for the current stream and sn is the event before it, over and over again.

[sklose]

Add metrics to State Provider:

• number of events read per stream
• snapshot size + event payload sizes

Potentially read + apply events in chunks to prevent OOM