ToTypedTask<T> fails with NullReferenceException on ASP.NET Core 2.1

[siennathesane]

I'm working on developing an ASP.NET Core Web API to front-end an Orleans cluster for image management. I'm leveraging ASP.NET Core 2.1 because of the new ActionResult<T> feature - this feature allows a Controller to return either an ActionResult which wraps the return type or the raw return type. I have tested this issue with both Windows and Ubuntu 16.04 (WSL) versions of .NET Core, and the results are the same. I have no build time errors or dependency warnings. I'm not sure how much information is enough information, so I've pasted the relevant code from the ASP.NET Controller, the ASP.NET configuration, the Grain class, and the Silo configuration, in case any of those configs are relevant.

I can share more code/configs upon request. :)

The workflow:

1. Image gets uploaded to ASP.NET Controller.
2. Image is converted to Orleans.Concurrency.Immutable<byte[]>.
3. Image is sent to silo through grain client.
4. Grain client throws NullReferenceException.

With this REST controller:

[HttpPost]
[ProducesResponseType(201)]
[ProducesResponseType(400)]
[ProducesResponseType(401)]
public async Task<ActionResult<Image>> PostImage([Required] string owner, [Required] IFormFile imagePayload) {
    if (imagePayload == null || imagePayload.Length == 0) {
        return new BadRequestObjectResult(new Image());
    }
    Image localImageRef = new Image();
    ...    
    var grain = _orleansClient.GetGrain<IImage>(localImageRef.Id);

    // make a quick buffer to read the image and then send it as a raw payload. since it's an image file, no
    // data should change, hence Immutable<T> is used.
    MemoryStream ms = new MemoryStream();
    await imagePayload.CopyToAsync(ms);
    Immutable<byte[]> imageByteArray = new Immutable<byte[]>(ms.ToArray());

    return await grain.CreateImage(localImageRef, imageByteArray);
}

Running in a Kestrel server with this configuration

public void ConfigureServices(IServiceCollection services) {
            services.AddSingleton<IClusterClient>(CreateOrleansClient());
            ...
        }
        public void Configure(IApplicationBuilder app, IHostingEnvironment env) {
            if (env.IsDevelopment()) {
                app.UseDeveloperExceptionPage();
            }
            ...
            app.UseMvc();
        }
        private IClusterClient CreateOrleansClient() {
            IClientBuilder clientBuilder = new ClientBuilder()
                .UseLocalhostClustering()
                .Configure<ClusterOptions>(opts => {
                    opts.ClusterId = "dev";
                    opts.ServiceId = "ImageStorageTest";
                })
                .ConfigureLogging(logging => logging.AddConsole());
            IClusterClient clusterClient = clientBuilder.Build();
            clusterClient.Connect(async ex => {
                Console.WriteLine("Retrying...");
                await Task.Delay(3000);
                return true;
            }).Wait();
            return clusterClient;
        }

Calling into this grain:

public class ImageGrain : Grain<ImageState>, IImage {
        private readonly MinioClient _minio;
        public ImageGrain(IServiceProvider serviceProvider) {
            _minio = serviceProvider.GetService<MinioClient>();
        }

        public async Task<Image> CreateImage(Image localImageRef, Immutable<byte[]> imagePayload) {
            localImageRef.LastUpdated = DateTime.UtcNow;
            await _minio.PutObjectAsync(...);
            localImageRef.Location = await _minio.PresignedGetObjectAsync(...);
            ...
            await WriteStateAsync();
            return localImageRef;
        }
        ...

Which runs with this silo configuration:

private static async Task<ISiloHost> StartSilo() {
            ISiloHostBuilder siloBuilder = new SiloHostBuilder()
                .AddMemoryGrainStorage("ImageStorage", options => options.NumStorageGrains = 10)
                .UseLocalhostClustering()
                .UseDashboard(opts => { })
                .Configure<ClusterOptions>(opts => {
                    opts.ClusterId = "dev";
                    opts.ServiceId = "ImageStorageTest";
                })
                .Configure<EndpointOptions>(opts => { opts.AdvertisedIPAddress = IPAddress.Loopback; })
                .ConfigureServices(services => {
                    services.AddSingleton<IMinioClient>(provider =>
                        new MinioStorageClient(...));
                })
                .ConfigureApplicationParts(parts => {
                    parts.AddApplicationPart(typeof(ImageGrain).Assembly).WithReferences();
                })
                .ConfigureLogging(logging => {
                    logging.AddConsole();
                });
            ISiloHost host = siloBuilder.Build();
            await host.StartAsync();
            return host;
        }

When I try to upload an image file from my local machine to the localhost ASP.NET Core server like this:

curl -X POST "http://localhost:5000/api/images?owner=mxplusb" -H  "accept: application/json" -H  "Content-Type: multipart/form-data" -F "imagePayload=@IMG_1460.jpg;type=image/jpeg"

I get this exception:

info: Messaging.GatewayConnection/GatewayClientSender_gwy.tcp://127.0.0.1:30000/0[101018]
      Preparing to send large message Size=1500814 HeaderLength=130 BodyLength=1500676 #ArraySegments=5. Msg=Messaging.GatewayConnection/GatewayClientSender_gwy.tcp://127.0.0.1:30000/0
fail: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[1]
      An unhandled exception has occurred while executing the request.
System.NullReferenceException: Object reference not set to an instance of an object.
   at Orleans.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync4_0[T](Task`1 asyncTask) in D:\build\agent\_work\18\s\src\Orleans.Core\Async\TaskExtensions.cs:line 100
   at WarframeAPI.Controllers.ImageController.PostImage(String owner, IFormFile imagePayload) in P:\csharp\warframe\Backend\Controllers\ImageController.cs:line 80
   at lambda_method(Closure , Object )
   at Microsoft.Extensions.Internal.ObjectMethodExecutorAwaitable.Awaiter.GetResult()
   at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIIndexMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

Windows .NET Core SDK info:

> dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   2.1.401
 Commit:    91b1c13032

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17134
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.1.401\

Host (useful for support):
  Version: 2.1.3-servicing-26724-03
  Commit:  124038c13e

.NET Core SDKs installed:
  2.1.2 [C:\Program Files\dotnet\sdk]
  2.1.202 [C:\Program Files\dotnet\sdk]
  2.1.302 [C:\Program Files\dotnet\sdk]
  2.1.401 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.3-servicing-26724-03 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Ubuntu 16.04 (WSL) .NET SDK info:

% dotnet --info
.NET Command Line Tools (2.1.105)

Product Information:
 Version:            2.1.105
 Commit SHA-1 hash:  141cc8d976

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  16.04
 OS Platform: Linux
 RID:         ubuntu.16.04-x64
 Base Path:   /usr/share/dotnet/sdk/2.1.105/

Host (useful for support):
  Version: 2.1.2
  Commit:  811c3ce6c0

.NET Core SDKs installed:
  2.1.105 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.NETCore.App 2.0.7 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.2 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Project configuration:

<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>netcoreapp2.1</TargetFramework>
  </PropertyGroup>
  <ItemGroup>
    <Folder Include="wwwroot\" />
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.All" Version="2.1.0" />
    <PackageReference Include="Microsoft.Orleans.Client" Version="2.0.3" />
    <PackageReference Include="Microsoft.Orleans.OrleansCodeGenerator.Build" Version="2.0.5" />
    <PackageReference Include="Swashbuckle.AspNetCore" Version="3.0.0" />
  </ItemGroup>
  <ItemGroup>
    <DotNetCliToolReference Include="Microsoft.VisualStudio.Web.CodeGeneration.Tools" Version="2.0.3" />
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="..\GrainInterfaces\WarframeGrainInterfaces.csproj" />
  </ItemGroup>
</Project>

Edit: added package config.

[sergeybykov]

Have you tried not passing the Image argument to the grain method, and define it as

public async Task<Image> CreateImage(Immutable<byte[]> imagePayload)

instead? It's not clear to me why you need to pass it at all, and I suspect the issue might be with it.

[siennathesane]

I am passing along information related to the Image object so it could be marshalled with some known fields already set. Do you think it's because I'm trying to pass an object instantiated on the sender end but not the receiver end? That would make sense, I think. I'll refactor it to only pass messages/non-objects and report back on my findings.

[siennathesane]

@sergeybykov I changed the signature to this:

// controller
public async Task<ActionResult<Image>> PostImage([Required] string owner, [Required] IFormFile imagePayload) {
    if (imagePayload == null || imagePayload.Length == 0) {
        return new BadRequestObjectResult(new Image());
    }
    Guid imageId = Guid.NewGuid();
    var grain = _orleansClient.GetGrain<IImage>(Guid.NewGuid());
    // make a quick buffer to read the image and then send it as a raw payload. since it's an image file, no
    // data should change, hence Immutable<T> is used.
    MemoryStream ms = new MemoryStream();
    await imagePayload.CopyToAsync(ms);
    Immutable<byte[]> imageByteArray = new Immutable<byte[]>(ms.ToArray());
    return await grain.CreateImage(imageByteArray, imagePayload.FileName, owner, imageId);
}

// grain interface
public async Task<Image> CreateImage(Immutable<byte[]> imagePayload, string uploadedName, string imageOwner, Guid imageGuid)

It still throws the same error:

fail: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[1]
      An unhandled exception has occurred while executing the request.
System.NullReferenceException: Object reference not set to an instance of an object.
   at Orleans.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync4_0[T](Task`1 asyncTask) in D:\build\agent\_work\18\s\src\Orleans.Core\Async\TaskExtensions.cs:line 100
   at WarframeAPI.Controllers.ImageController.PostImage(String owner, IFormFile imagePayload) in P:\csharp\warframe\Backend\Controllers\ImageController.cs:line 76
   at lambda_method(Closure , Object )
   at Microsoft.Extensions.Internal.ObjectMethodExecutorAwaitable.Awaiter.GetResult()
   at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIIndexMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

[sergeybykov]

@ReubenBond Does this ring any bell? Seems as if Image isn't getting serialized or deserialized properly.

[siennathesane]

For reference, here is my Image class and it's dependency, ImageState.

public class Image
{
    public Guid Id;
    public string Location;
    public string UploadedName;
    public string UniqueName;
    public string Owner;
    public DateTime LastUpdated;
    public ImageState CurrentState;
}
public class ImageState
{
    public Guid ImageId;
    public byte CurrentState;
    public byte NextState;
    public Guid WorkflowToken;
    public DateTime LastUpdate;

    public enum PotentialImageStates
    {
        Unprocessed = 0x0,
        Waiting = 0x1,
        QueuedForProcessing = 0x2,
        Processing = 0x4,
        Processed = 0x8,
        ToBeRemoved = 0x10,
    }

    public enum ImageWorkflowSteps
    {
        AnalyseImage = 0x0,
        DescribeImage = 0x1,
        RecognizeText = 0x2,
        OpticalCharacterRecognition = 0x4,
        TagImage = 0x8,
        GetThumbnail = 0x10,
    }
}

[ReubenBond]

I'd also guess at a serialization issue. All types which are being sent over the wire should be marked as [Serializable]. Does the project that Image lives in have the code generation package installed?

Can you attach a debugger and step through the code on client and silo?

[siennathesane]

I followed the standard 4 project set up from the examples:

• Interfaces
• Grains
• Silo
• Web API

Looking through all the csproj files, everything but the Silo project has this package reference:

<PackageReference Include="Microsoft.Orleans.OrleansCodeGenerator.Build" Version="2.0.5" />

I just added the [Serializable] attribute to the Image and ImageState classes and tried again, but got the same results.

I'm not sure what I'm looking for in the debugger, but here is the stack trace:

ExceptionDispatchInfo.Throw() in System.Runtime.ExceptionServices, System.Private.CoreLib.dll
TaskAwaiter.ThrowForNonSuccess() in System.Runtime.CompilerServices, System.Private.CoreLib.dll
TaskAwaiter.HandleNonSuccessAndDebuggerNotification() in System.Runtime.CompilerServices, System.Private.CoreLib.dll
TaskAwaiter.ValidateEnd() in System.Runtime.CompilerServices, System.Private.CoreLib.dll
TaskAwaiter<object>.GetResult() in System.Runtime.CompilerServices, System.Private.CoreLib.dll
async OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync4_0<WarframeGrainInterfaces.Image>() in Orleans, Orleans.Core.dll
AsyncTaskMethodBuilder<Image>.AsyncStateMachineBox<OrleansTaskExtentions.<<ToTypedTask>g__ConvertAsync4_0>d<Image>>.<>c.<.cctor>b__9_0() in System.Runtime.CompilerServices, System.Private.CoreLib.dll
ExecutionContext.RunInternal() in System.Threading, System.Private.CoreLib.dll
AsyncTaskMethodBuilder<Image>.AsyncStateMachineBox<OrleansTaskExtentions.<<ToTypedTask>g__ConvertAsync4_0>d<Image>>.MoveNext() in System.Runtime.CompilerServices, System.Private.CoreLib.dll
AwaitTaskContinuation.RunOrScheduleAction() in System.Threading.Tasks, System.Private.CoreLib.dll
Task.RunContinuations() in System.Threading.Tasks, System.Private.CoreLib.dll
Task.FinishContinuations() in System.Threading.Tasks, System.Private.CoreLib.dll
Task.FinishStageThree() in System.Threading.Tasks, System.Private.CoreLib.dll
Task.FinishStageTwo() in System.Threading.Tasks, System.Private.CoreLib.dll
Task.FinishSlow() in System.Threading.Tasks, System.Private.CoreLib.dll
Task.Finish() in System.Threading.Tasks, System.Private.CoreLib.dll
Task<object>.TrySetException() in System.Threading.Tasks, System.Private.CoreLib.dll
TaskCompletionSource<object>.TrySetException() in System.Threading.Tasks, System.Private.CoreLib.dll
OrleansTaskExtentions.<>c__DisplayClass21_0<object>.<ConvertTaskViaTcs>b__0() in Orleans, Orleans.Core.dll
ContinuationTaskFromResultTask<object>.InnerInvoke() in System.Threading.Tasks, System.Private.CoreLib.dll
Task.<>c.<.cctor>b__278_1() in System.Threading.Tasks, System.Private.CoreLib.dll
ExecutionContext.RunInternal() in System.Threading, System.Private.CoreLib.dll
Task.ExecuteWithThreadLocal() in System.Threading.Tasks, System.Private.CoreLib.dll
Task.ExecuteEntryUnsafe() in System.Threading.Tasks, System.Private.CoreLib.dll
Task.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() in System.Threading.Tasks, System.Private.CoreLib.dll
ThreadPoolWorkQueue.Dispatch() in System.Threading, System.Private.CoreLib.dll
_ThreadPoolWaitCallback.PerformWaitCallback() in System.Threading, System.Private.CoreLib.dll
[Native to Managed Transition]

Here are the stacks that are running:

Here are the variables running in the stack when it gets to ToTypedTask<T>:

Here are the variables running in the stack when it hits the exception:

Let me know if there is something else you are looking for.

[siennathesane]

To follow up on this issue, it turns out I was using the wrong Dependency Injection method within the grain.

This is what I had:

public class ImageGrain : Grain<ImageState>, IImage {
        private readonly MinioClient _minio;
        public ImageGrain(IServiceProvider serviceProvider) {
            _minio = serviceProvider.GetService<MinioClient>();
        }

This is what I needed:

    public class ImageGrain : Grain<ImageState>, IImage {
        private IMinioClient _minio;
        public readonly string ImageBucket = "wf-uploaded-images";

        public ImageGrain(IMinioClient minioClient) {
            _minio = minioClient;
        }

Closing since it's my code, not Orleans.