Transactions across Actors

[leo12chandu]

When creating a new issue, please make sure the following information is part of your issue description. (if applicable). Thank You!

• Which Akka.Net version you are using 1.3.3
• On which platform you are using Akka.Net Windows
• A list of steps to reproduce the issue. Or an gist or github repo which can be easily used to reproduce your case. Is there a way to implement transactional actors? So, if say I have 2 actors one persisting data into database, and another one writing some data into some other storage. I need to make sure that either both are written successfully or both are rolled back. Something like MSDTC. Like create a Transaction object and send it as a message across the actors. Only if any of the actors failed, we rollback that transaction?

[Horusiath]

@leo12chandu In general transactions are limiting the availability of distributed applications - as you need to potentially lock resources across multiple machines - and are not the default go-to when building such systems. Currently I've heard about more optimized algorithms to transactions in distributed systems, but they are still pretty much in their experimental stages.

Usually a common way of working with cases like yours is a Saga pattern.

[jalchr]

@Horusiath Any sample "Saga" using Akka.net ? Doesn't this require "guaranteed message delivery" ?

[Horusiath]

@jalchr to achieve easy guaranteed message delivery, you can ensure that all actors live in the same process. For stronger cross-process guarantees, you may use redeliveries with deduplication/idempotency. Akka.Persistence (which is necessary for sagas anyway) comes with AtLeastOnceDelivery actors, which can handle redeliveries - you may also decide to build your own if you'll find that feature too complex. For idempotent part - since Saga is an actor designed to handle a single unit of work, it's a lot easier to handle duplicates.

I've seen examples of saga pattern on the JVM - it may be good idea to give a demo of such use case for .NET, or even create a plugin library with generic implementation.

[jalchr]

@Horusiath I understand the tactics. I trust it is "necessary" to have a working example in akka.net, since such system is non-trivial to implement. I would love to participate in such example, but I trust we need to define a "solid model" that takes best practices and optimal usage of akka.net tools. @Aaronontheweb once discussed this issue with me and mentioned that he has some thoughts regarding it

[ismaelhamed]

Some insist in proving this wrong, so we'll see how this play out for Orleans. IMO distributed transactions are the number one PITA in our systems running NSB, so I'm not a big fan.

[kfrajtak]

@ismaelhamed Saga is not a distributed transaction in the ACID sense:

The Saga pattern is a way to manage transactions and their compensations in a distributed environment. It is used to maintain data consistency across multiple microservices.

Saga/transaction can for example span days waiting for user input. The rollback is not done automatically but explicitly by compensating actions, see for example https://github.com/BillyAutrey/akka-saga-sample/blob/main/src/main/scala/com/example/TypedSagaActor.scala#L77.

[Aaronontheweb]

I'm going to separate two issues from each other:

• Guaranteed or "reliable" delivery of messages between actors and
• ACID-style transactions

Actor Transactions

Some context: I've been working on large-scale distributed .NET systems full-time for ~13 years, on Akka.NET for ~10 years, and I've worked with hundreds of companies across hundreds of domains since I started working full-time on Akka.NET back at the start of 2015.

In all of that time I have never once ran into a scenario where an ACID-style distributed transaction between actors was a practical solution to a problem, let alone a feasible solution.

I've looked at code bases that use technologies like Azure Durable Functions where it's common to "lock" multiple discrete actor-like functions together into a single unit of work. That's "necessary" in those environments because there's no control over locality - functions execute arbitrarily on multi-tenant VMs at the discretion of the platform. In Akka.NET you can just call Context.ActorOf and create local actors to execute the unit-of-work in-process, all directed by a single parent actor that owns the complete unit - that's a much simpler, reliable, and performant solution compared to distributed locks spanning multiple processes.

The Azure Durable Functions example is not even really "ACID" in those scenarios (i.e. can't automatically rollback successful functions in a failed orchestration) - and this brings me to my principle issue with conflating ACID transactions in a well-defined application like a RDBMS versus a general purpose application programming framework like Akka.NET, Orleans, or Azure Durable Functions: how on earth do you guarantee atomicity when the possibilities for what users can accomplish mid-transaction are unbounded?

For instance, if I can make an external Web API call or send a transactional email in the middle of an actor transaction - there's no real possibility for a "rollback" even if the actor runtime "guarantees" or automates it.

Users would have to carefully reason about their effects - actor state changes would need to isolated from the code that reacts to those very state-change in real-time. Even worse: they'd have to do this for all of the actors participating in the transaction.

Databases don't have this problem as their domains are tightly scoped - application programming frameworks, on the other hand, are un-scoped by definition.

For all of the complaints we hear about Akka.NET's learning curve, I think a real ACID-style distributed transactions system for Akka.NET would be extremely difficult to implement correctly for users and a tremendous footgun.

I won't get into the drawbacks of distributed transactions of actors for areas like performance, propensity for deadlocking, and the debuggability problems this will present for users: the tremendous complexity it imposes at design-time for users is a persuasive enough argument for not doing it.

Reliable Delivery of Messages between Actors

When a developer asks for "transactions" between actors, what I suspect they're really asking for is some guaranteed way of ensuring that messages are delivered to and successfully processed by their destination actor. That's a much simpler and more feasible task. We've had Akka.Persistence's AtLeastOnceDeliveryActor for years but that's honestly been a half-measure for solving this problem.

As of Akka.NET v1.5.7, we're going to introduce Akka.Delivery (https://github.com/akkadotnet/akka.net/pull/6720) which addresses this problem much more robustly through a series of actors that manage sequencing, re-delivery, and buffering of unprocessed messages on both sides of the wire. I'm still writing up the documentation on it https://github.com/akkadotnet/akka.net/pull/6757 - but once that's finished the feature will ship.

If you want to see a preview of how it works, we covered this in our May 2023 Akka.NET Community Standup: https://www.youtube.com/live/hJH0cLq4JA8?feature=share&t=1027 - starting at the 17 minute mark.