Allow to project Durable Entities at a given point in time

[david-acm]

I know that durable entities, like the rest of the durable extension, use event sourcing behind the scenes to manage state and resume execution. As far as I know, one of the main advantages of event sourcing is the ability to reconstruct / project the state of a system at a specific point in time.

It would be great if one could use that underlying technology to use the ReadEntityStateAsync() method to project the state of an entity in a specific point in time, maybe a timestamp or the operation ID as an additional optional parameter.

I'm currently working with a situation where I need to know the state of an entity at different times; the only other solution I can come up while still using the Durable Extension is to actually create a new Durable Entity instance (version) instead of updating the state, and create another master entity to aggregate the "entity versions". This doesn't seem ideal, as the Durable Entity is already using events to keep track of its state.

Please let me know if this feature makes sense at all or if I'm missing some way to achieve what I'm looking for, or if there is a better alternative for this kind of use case.

[cgillum]

Hey @daviacm7, actually, Durable entities do not maintain a history like orchestrations do. They share all the same basic properties of orchestrations except this one, since it wasn’t actually necessary. It’s therefore not possible to get an earlier snapshot.

[david-acm]

@cgillum what do you think about the use case? Do you think it's something useful? Will we see sometime durable entities saving state like orchestrations do and then making posible to snapshot them?

[ConnorMcMahon]

I think it is theoretically possible to add some support for something like this. That being said, due to the way entities are currently implemented and the fact that they are not a first-class notion in the core Durable Task Framework, this would be a lot of work, so I would want to see more concrete evidence that this is a widely desired feature. In the meantime, let's keep this ticket open to see how much desire people have for a feature like this.

There are still some open questions in this proposal for how users would interact with these entities, such as would snapshots be explicitly taken, or after every operation, and how do customers interact with these snapshots?

@daviacm7, my recommendation for your project would be to build some ad-hoc snapshot support in your own entity function. It should be straightforward to create a TaskSnapshot operation on your entity that writes the current state to blob storage with a timestamp, and you would then need some API to query these snapshots, but it should be fairly straightforward.

[olitomlinson]

If you could force Entities to use blob storage for their state (rather than Table) then you could possibly take advantage of the Change Feed for Blob Storage which was recently introduced.

I'm pretty sure Entities already support blob storage if their serialized size is too big for Table Storage, there is just no mechanism to force blob always.

https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-change-feed?tabs=azure-portal