NullReferenceException when creating PersistentActor with MemoryJournal and CallingThreadDispatcherConfigurator

[fdbeirao]

Akka.Net version: 1.3.12 Platform: Windows (.net core)

When creating a PersistentActor, while using the CallingThreadDispatcherConfigurator, we can observe a NullReferenceException, and the actor doesn't get created.

The minimal code to reproduce this issue is available here.

If we apply #3770, we can see that the exception is no longer a NullReferenceException, but instead a more "useful" exception:

[ERROR][2019-04-30 15:19:28][Thread 0001][akka://mysystem/system/akka.persistence.journal.inmem] Error while creating actor instance of type Akka.Persistence.Journal.MemoryJournal with 0 args: ()
Cause: [akka://mysystem/system/akka.persistence.journal.inmem#102829150]: Akka.Actor.ActorInitializationException: Exception during creation ---> System.TypeLoadException: Error while creating actor instance of type Akka.Persistence.Journal.MemoryJournal with 0 args: () ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidOperationException: ValueFactory attempted to access the Value property of this instance.
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at Akka.Persistence.PersistenceExtension.<>c__DisplayClass20_0.<AdaptersFor>b__0(Lazy`1 e)
   at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1 source, Func`2 predicate, Boolean& found)
   at Akka.Persistence.PersistenceExtension.AdaptersFor(IActorRef journalPluginActor)
   at Akka.Persistence.Journal.WriteJournalBase..ctor()
   at Akka.Persistence.Journal.AsyncWriteJournal..ctor()
   --- End of inner exception stack trace ---
   at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor)
   at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(Type type, Boolean nonPublic)
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.Activator.CreateInstance(Type type, Object[] args)
   at Akka.Actor.Props.ActivatorProducer.Produce()
   at Akka.Actor.Props.NewActor()
   --- End of inner exception stack trace ---
   at Akka.Actor.Props.NewActor()
   at Akka.Actor.ActorCell.CreateNewActorInstance()
   at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<NewActor>b__0()
   at Akka.Actor.ActorCell.UseThreadContext(Action action)
   at Akka.Actor.ActorCell.NewActor()
   at Akka.Actor.ActorCell.Create(Exception failure)
   --- End of inner exception stack trace ---
   at Akka.Actor.ActorCell.Create(Exception failure)
   at Akka.Actor.ActorCell.SysMsgInvokeAll(EarliestFirstSystemMessageList messages, Int32 currentState)
Press Enter to terminate the actor system
[INFO][2019-04-30 15:19:28][Thread 0007][akka://mysystem/system/akka.persistence.journal.inmem] Message ReplayMessages from akka://mysystem/user/$a to akka://mysystem/system/akka.persistence.journal.inmem was not delivered. 1 dead letters encountered.

[fdbeirao]

This seems to be a duplicate of #2026. Hopefully I have provided a simpler way of reproducing the error.

[Aaronontheweb]

Dupe of #2026

[Aaronontheweb]

@fdbeirao thanks for helping me validate #3770 though!

[ghost]

I think this issue should be re-opened, as can be seen from the stacktrace embedded in the issue text, after applying #3770 . The actual cause of both #2026 and this issue surfaces. When using the CallingThreadDispatcherConfigurator the thread safety provided by:

private readonly ConcurrentDictionary<string, Lazy<PluginHolder>> _pluginExtensionIds = new ConcurrentDictionary<string, Lazy<PluginHolder>>();

in Persistence.cs breaks down into an infinite recursion that Lazy guards itself against by throwing the InvalidOperationException. @fdbeirao and I were trying to come up with fixes for this issue but failed so far. What we tried:

• Replace Lazy<PluginHolder> with PluginHolder in the above declaration (and fix the resulting errors)
• In AdaptersFor and ConfigFor add e.IsValueCreated && in the condition of the FirstOrDefault, this results in the ArgumentException($"Unknown plugin actor {journalPluginActor}") surfacing on lots of places in client code.

[Aaronontheweb]

Can you create a reproduction test first and work backwards from that?

[ghost]

I've tried to add a failing test, see here, however suprisingly this test passes. When debugging it with break on all exceptions I can observe the InvalidOperationException in the test however this is caught in ActorCell.SysMsgInvokeAll. @Aaronontheweb Do you have any idea what causes the test to behave differently? I've ruled out target framework as an option for the difference. A variation of @fdbeirao s gist using the net452 framework still produces the exception.

[Aaronontheweb]

I've tried to add a failing test, see here, however suprisingly this test passes. When debugging it with break on all exceptions I can observe the InvalidOperationException in the test however this is caught in ActorCell.SysMsgInvokeAll

Tell-tale sign of a Heisenbug. I think the best place to start is to review the faulty code first and understand what's happening there.