search

akkadotnet / Akka.Persistence.Redis

Redis storage for Akka.NET Persistence
Apache License 2.0
30 stars 19 forks source link

Akka.Persistence.Query: add support for CurrentPersistenceIds #158

Open Aaronontheweb opened 3 years ago

Aaronontheweb commented 3 years ago

Is your feature request related to a problem? Please describe. While adding some integration tests for our Akka.Cluster.Sharding persistence cleanup tool, I ran into a small problem on this PR: https://github.com/petabridge/Akka.Cluster.Sharding.RepairTool/pull/7 - Akka.Persistence.Redis' IReadJournal does not support ICurrentPersistenceIds().

I know we removed it as part of https://github.com/akkadotnet/Akka.Persistence.Redis/issues/126 because we thought it'd be too expensive to implement in clustered redis scenarios, but I wonder if that's true.

Describe the solution you'd like The "all events" and "events by tag" queries are out of the question - they're expensive because they can't effectively be cleaned up in a clustered environment without some kind of inter-node index and we're not database architects so we're not going to build one. Users can use a relational database for that.

But Akka.Persistence requires us to keep track of all used PersistentIds even after the entities have had all of their data purged - we're required to keep those records in perpetuity and therefore cleanup is not an issue.

So, I suppose we can try implementing the ICurrentPersistentIds and IPersistentIds queries in one of two ways:

  1. Scatter-gather to all of the nodes in the cluster each time we need to run one of those queries and just use the built-in key indicies we have now. That's O(n) where n = entity count.
  2. Create a special table that only really needs to get replicated to single entity and have it contain a hashset that has all of the entity ids and we can use a Redis channel to receive notifications when that entity is updated. I don't really like this idea because it's single point of failure.

Describe alternatives you've considered We could also go on not doing this, but since we're taking a dependency on ICurrentPersistentIds for helping Cluster.Sharding users cleanup I'd like to do our best to support it.

Aaronontheweb commented 3 years ago

Worth noting: the actual delete and purge routines inside https://github.com/petabridge/Akka.Cluster.Sharding.RepairTool will stick work fine with Redis. Just not the tools for querying the journal to find the shardRegions before deleting them.