search

akkadotnet / akka.net

Canonical actor model implementation for .NET with local + distributed actors in C# and F#.
http://getakka.net
Other
4.71k stars 1.04k forks source link

Akka Persistence Throwing Unobserved Task Exception #7358

Open stedmanm opened 2 weeks ago

stedmanm commented 2 weeks ago

Version Information Version of Akka.NET? 1.5.28 Which Akka.NET Modules? Akka.Persistence

Describe the bug Bug appears to be in method CallThrough of class AtomicState. If WaitAsync(Timeout timeout) completes before the task implemented by SnapshotStore/Journal, and the task in question throws an exception, then this exception will become unobserved.

To Reproduce

  1. Snapshot loading takes longer than call timeout specified by Akka framework, and Snapshot call throws an exception.

Expected behavior Would expect Akka framework to observe exception thrown by SnapshotStore/Journal.

Actual behavior Causes unobserved task exception and crashes our service.

Screenshots image

Environment Windows Service and .NET 7

Aaronontheweb commented 2 weeks ago

@stedmanm thanks for reporting this! We'll look into it

Arkatufus commented 6 days ago

A fix to this issue will have a very high knock on effect.

To fix this effectively, we'd need to implement cancellation token to all journal/snapshot database related methods to effectively signal a cancellation to the underlying task.

That means implementing methods supporting cancellation tokens to ALL public async API methods that is also backward compatible, this will be a binary breaking change for all persistence plugins.

Arkatufus commented 6 days ago

This is an actual bug, here is a simple reproduction to illustrate this: https://share.linqpad.net/pbjjq82i.linq

An .AsyncWait() call does throw a TimeoutException when the execution time exceeds the timeout argument, but it does not stop the running task, making it a detached task running in the background.