System.ExecutionEngineException when retrieving wrong reliable collection type.

[OlegKarasik]

Updated 2019/03/04

Recently I was experimenting with reliable collections and found that the following sequence of events causes System.ExecutionEngineException exception.

Create a new stateful service and write the following in RunAsync:

var queue = this.StateManager.GetOrAddAsync<IReliableQueue<long>>("value-collection");
using (var transaction = this.StateManager.CreateTransaction())
{
    for (var i = 0; i < 10; i++)
    {
        await queue .EnqueueAsync(transaction, i);
    }

    await transaction.CommitAsync();
}

Now create a new version of the service and replace IReliableQueue<long> with IReliableConcurrentQueue<long>.

When you hit this.StateManager.GetOrAddAsync<IReliableConcurrentQueue<long>>("value-collection"); the System.ArgumentException with the following exception message is thrown:

Returned reliable object of type Microsoft.ServiceFabric.Data.Collections.DistributedQueue`1[System.Int64] cannot be casted to requested type Microsoft.ServiceFabric.Data.Collections.IReliableConcurrentQueue`1[System.Int64]

Then if this SystemArgumentException isn't handled via try { ... } catch { ... } block then the second exception is thrown:

System.ExecutionEngineException: 'Exception of type 'System.ExecutionEngineException' was thrown.'

This sequence causes application process to terminate unexpectedly. Here is an event from event log:

{
  "Timestamp": "2019-03-04T11:47:32.4300359+03:00",
  "ProviderName": "Microsoft-ServiceFabric",
  "Id": 23083,
  "Message": "EventName: ApplicationProcessExited Category: StateTransition ApplicationHostTerminated:   ApplicationId=fabric:/Application13,  ServiceName=fabric:/Application13/Web1,  ServicePackageName=Web1Pkg,  ServicePackageActivationId=58a4eac9-5a8f-4d1f-aee9-25d55a52d1ad,  IsExclusive=True,  CodePackageName=Code,  EntryPointType=Exe,  ExeName=Web1.exe,  ProcessId=30560,  HostId=017280df-c352-4236-aa5f-a292911ea001,  ExitCode=2148734499,  UnexpectedTermination=True,  StartTime=03/04/2019 11:42:42. ",
  "ProcessId": 14544,
  "Level": "Informational",
  "Keywords": "0x4000000000000001",
  "EventName": "Hosting",
  "ActivityID": null,
  "RelatedActivityID": null,
  "Payload": {
    "eventName": "ApplicationProcessExited",
    "category": "StateTransition",
    "eventInstanceId": "\"1db8e21e-9dc1-4c4e-9b72-a3b8a03e08e6\"",
    "applicationName": "fabric:/Application13",
    "ServiceName": "fabric:/Application13/Web1",
    "ServicePackageName": "Web1Pkg",
    "ServicePackageActivationId": "58a4eac9-5a8f-4d1f-aee9-25d55a52d1ad",
    "IsExclusive": true,
    "CodePackageName": "Code",
    "EntryPointType": 1,
    "ExeName": "Web1.exe",
    "ProcessId": 30560,
    "HostId": "017280df-c352-4236-aa5f-a292911ea001",
    "ExitCode": 2148734499,
    "UnexpectedTermination": true,
    "StartTime": "\"\/Date(1551688962127)\/\""
  }
}

The exception has no stack trace and is thrown after previous exception leaves RunAsync method.

Version of Service Fabric is : 6.4.617.9590

Nuget packages have: 6.4.617 and 3.3.617 versions.

[ashishnegi]

@OlegKarasik does the app terminate because of not catching ExecutionEngineException by your code ? What is your expected behavior in this case ?

[OlegKarasik]

@ashishnegi

@OlegKarasik does the app terminate because of not catching ExecutionEngineException by your code ?

Sorry for confusion (I have update the question: replicating here too):

When SystemArgumentException isn't handled via try { ... } catch { ... } block then the second exception is thrown:

System.ExecutionEngineException: 'Exception of type 'System.ExecutionEngineException' was thrown.'

The exception has no stack trace and is thrown after previous exception leaves RunAsync method.

What is your expected behavior in this case ?

I think the expected behavior is no System.ExecutionEngineException and normal service instance / host process termination.

[ashishnegi]

If you are able to reproduce it, then you can enable the checkbox Common Language Runtime Exceptions in VS. This will stop the debugger at the point exception is thrown. This will give us insights into what is happening.

[OlegKarasik]

@ashishnegi I have checked the Common Language Runtime Exception but this didn't change anything. When the exception is thrown I get a Visual Studio screen.

---

The application is in break mode

Your app has entered a break state, but there is no code to show because all threads were executing external code (typically system or framework code).

---

Exception Unhandled

System.ExecutionEngineException: 'Exception of type 'System.ExecutionEngineException' was thrown.'

---

There is no StackTrace available in StackTrace window neither it is available as part of the exception.

[ashishnegi]

@OlegKarasik May be try disable : "Enable Just my code"

[OlegKarasik]

@ashishnegi

I've disabled the Enable Just my code and now Visual Studio debugger stops with ExecutionEngineException inside ExecutionContext.cs.

System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 167
    at E:\A\_work\287\s\src\mscorlib\shared\System\Threading\ExecutionContext.cs(167)
System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task currentTaskSlot) Line 2440
    at E:\A\_work\287\s\src\mscorlib\src\System\Threading\Tasks\Task.cs(2440)
System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 588
    at E:\A\_work\287\s\src\mscorlib\src\System\Threading\ThreadPool.cs(588)

The exception object still has no StackTrace information.

[OlegKarasik]

Hi @ashishnegi and everyone who have encountered this issue.

Recently I have found out what was really happening.

When Service Fabric starts RunAsync method the following code is being executed:

I will demonstrate it on StatelessService example but very similar mechanism is used in StatefulService

// File: StatelessServiceInstanceAdapter.cs

async Task<string> IStatelessServiceInstance.OpenAsync(
    IStatelessServicePartition partition,
    CancellationToken cancellationToken)
{
    // ...
    this.executeRunAsyncTask = this.ScheduleRunAsync(this.runAsynCancellationTokenSource.Token);
    // ...
}

private Task ScheduleRunAsync(CancellationToken runAsyncCancellationToken)
{
    // ...
    return Task.Run(
        () => this.ExecuteRunAsync(runAsyncCancellationToken),
        CancellationToken.None);
}

private async Task ExecuteRunAsync(CancellationToken runAsyncCancellationToken)
{
    // ...
    try
    {
        await this.userServiceInstance.RunAsync(runAsyncCancellationToken);
    }
    catch (OperationCanceledException e)
    {
        // ...
    }
    catch (FabricException e)
    {
        // ...
    }
    catch (Exception e)
    {
        // ...
        this.serviceHelper.HandleRunAsyncUnexpectedException(this.servicePartition, e);
        return;
    }
}

// File: ServiceHelper.cs

internal void HandleRunAsyncUnexpectedException(IServicePartition partition, Exception ex)
{
    // ...
    Task.Run(() => Environment.FailFast(msg));
}

You can notice that handling of unexpected exception ends with a call to Environment.FailFast (because there it no way to restore application state and it is easier just to terminate and restart the process).

The thing here is that when you are terminating application using Environment.FailFast with active Visual Studio debugger the call to Environment.FailFast throws an System.ExecutionEngineException and automatically triggers the fatalExecutionEngineError managed debugging assistant (MDA).

I have created a pull request to add information about System.ExecutionEngineException and Environment.FailFast to documentation (which is now merged and soon should be live on docs.microsoft.com).

Just wanted to share that this is not an issue :)

REFERENCES

1. StatelessServiceInstanceAdapter.cs
2. ServiceHelper.cs