Windows Server 2012 R2 - Kestrel Immediate Shutdown on Fluent Build Job Decode/Encode

[RossLieberman]

I built a small middleware handler similar to @kurtgronbech to resize images. The app is very basic, (besides program/startup class files) it only has a class for the middleware and the nuget packages for Imageflow. It's using the .Net Core 2.0 assemblies. It was developed on OSX with the OSX native runtimes using rc 1.0.0-rc8. And everything is working great there. Before deployment I installed the native runtime for Imageflow.NativeRuntime.win-x86. Everything still runs fine on the mac through Apache/Kestrel.

.Net Core 2.0 Hosting is installed on the server. When I went to load up an image I was getting annoying 502.5 errors. After several hours of trying to debug the app through IIS I noticed that Kestrel was continuously assigning different port numbers and I couldn't access the localhost port assigned directly through the browser. This was hard to track down because no errors were being thrown in the error logs or event viewer. All I could see was that Kestrel started successful and the request was made.

I ran Kestrel directly within a command prompt for my application and when tried to hit the middleware again I noticed it immediately shut down without any errors. I changed my middleware to just output a string "Hello World" and it was successful and Kestrel didn't crash (eliminating a bad .NET/IIS install).

public async Task Invoke(HttpContext httpContext)
{
        httpContext.Response.ContentType = "text/html";
        await httpContext.Response.WriteAsync("Hello World");
}

I then changed the code to output just the image directly without processing.

public async Task Invoke(HttpContext httpContext)
{
       var path = httpContext.Request.Path;
       var imageName = path.Value.TrimStart('/');
       var diskPath = _appSettings.GetImageFlowDirectories().TestDirectory;

       if (!File.Exists($"{diskPath}{imageName}"))
           imageName = "hwx.530.530.png";

       var extension = Path.GetExtension($"{diskPath}{imageName}");

       using (var fileStream = File.Open($"{diskPath}{imageName}", FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
       {
           httpContext.Response.ContentType = GetContentType(extension);                            
           await fileStream.CopyToAsync(httpContext.Response.Body);
       }
}

This also worked just fine and output the image. Eliminating the possibility that it was a bad image or a problem between IIS returning an image from Kestrel.

However once I put the code back in to Decode / Encode, Kestrel just crashes with no error message. I'd like to provide more details on the failure except Kestrel just shuts down and doesn't provide an error message or more information.

I've tried changing the x86/x64 & Sandybridge DLL's. I noticed when I used x86 DLL (and didn't target x86) I got an error message that it was using the wrong runtime API (but Kestrel didn't shut down). I also noticed that when I try to open an image that doesn't exist on the server, I get a File Open Error (and Kestrel doesn't shut down). This leads me to believe that the error is within ImageFlow (or ImageFlow.net). Let me know how I can provide more debugging information or if anyone else has encountered this and/or fixed it that would be great.

Here's my full code:

public class ImageHandler
{
    #region Properties

    private readonly RequestDelegate _next;
    private readonly IAppSettings _appSettings;

    #endregion

    #region Constructors

    public ImageHandler(RequestDelegate next, IAppSettings appSettings)
    {
        _next = next;
        _appSettings = appSettings;
    }

    #endregion

    public async Task Invoke(HttpContext httpContext)
    {
        var path = httpContext.Request.Path;
        var imageName = path.Value.TrimStart('/');
        var diskPath = _appSettings.GetImageFlowDirectories().TestDirectory;

        if (!File.Exists($"{diskPath}{imageName}"))
            imageName = "hwx.530.530.png";

        var extension = Path.GetExtension($"{diskPath}{imageName}");

        using (var fileStream = File.Open($"{diskPath}{imageName}", FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
        {
            using (var b = new FluentBuildJob())
            {
                var r = await b
                        .Decode(fileStream, false)
                        //.ConstrainWithin(100, 100) // commented for this example
                        .EncodeToBytes(GetCurrentFormat(extension))
                        .FinishAsync();

                var bytes = r.First.TryGetBytes();

                if (bytes != null)
                    using (var stream = new MemoryStream(bytes.Value.Array, bytes.Value.Offset, bytes.Value.Count))
                    {
                        httpContext.Response.ContentType = GetContentType(extension);                               
                        await stream.CopyToAsync(httpContext.Response.Body);
                    }
            }

        }

    }

    private IEncoderPreset GetCurrentFormat(string extension)
    {
        switch (extension)
        {
            case ".png":
                return new LibPngEncoder();
        }

        return new LibJpegTurboEncoder();
    }

    private string GetContentType(string extension)
    {
        switch (extension)
        {
            case ".png":
                return "image/png";
        }

        return "image/jpeg";
    }
}

[lilith]

Debugging is harder inside a host process (like Kestrel) that might be suppressing error output.

imageflow_tool.exe is excellent for reproducing issues (it will even zip up a reproduction package for you). We don't yet have that tooling for the .NET wrapper, unfortunately.

Can you share the image that this crash occurred with, and the processor details for the server in question?

Also, I want to confirm that this code works fine on OS X, but fails on 2012 R2 with the same input, correct?

[lilith]

Does this happen with rc6 or rc7?

[RossLieberman]

According to BGInfo, the server is a Quad 2.0 GHz QEMU Virtualversion (cpu64-rhel6) (intel). Sadly, I'm unable to share an image of the server, despite that it's 40gb, it's a production server that handles credit cards and is hosted by a 3rd party at a location where I don't have that level of access. I'll set this up on a local Windows 7 VM to see if this can be duplicated.

This code still works in OSX for the same input, I was surprised that it still did with 2 native runtimes referenced in nuget, when I publish the site, JetBrains Rider puts the runtimes in separate folders (for osx, wind-x64 & win7-x86) and the website won't run without imageflow.dll copied into the root of the site for the correct runtime.

Thanks for replying so quickly, I'll try rc6/rc7 on Monday. I'll also give imageflow_tool.exe a try on Monday on the production server. I spent the day deploying 8 sites, this was the only one that caused any issues but that's why it's still a release candidate and not a final release. I know you are focusing on the main libimageflow tool and the .Net wrapper was a collaboration by Sam, but I do look forward to when it comes a bit more featured in functionality like Image Resizer, which is one of the reasons why I backed on Kickstarter. Keep up the good work.

[lilith]

I think I've been able to reproduce this.

[RossLieberman]

I've had an opportunity to test your requests. The web site does work on my Windows 7 VM through Kestrel, however I tried both rc6 & rc7 on the production server and both crash Kestrel immediately. I tried the imageflow_tool.exe v1.0.0-rc8 x64 on the production server and it throws an error. Visual Studio tries to debug it, no output occurs, and I've tried adding the diagnose flag or response flag but I was unable to get it to work.

> imageflow_tool v0.1/ir4 --in TCSH-0380x_l.jpg --out thumb.jpg --command "width=100&height=100"

The above command crashes on the production server. No output occurs (to disk), and when Visual Studio opens to debug it's looking for the PDB file (which I don't have). I did get a generic message within Visual Studio (below) but no stack trace or line number without the PDB. I tried the same command on my local Windows 7 VM and it resized the image with a JSON output correctly.

Unhandled exception at 0x00007FF694D04590 in imageflow_tool.exe: 0xC000001D: Illegal Instruction.

If you need more information, let me know, I might need some details on how to get additional debug data.

[lilith]

I've fixed a bug in rc9 that might have been causing the original error. As far as the Illegal Instruction, it might literally be correct - QEMU might not support the sandybridge extensions. I'm going to look into adding a baseline x64 build for older processors.

[RossLieberman]

Thanks. I just tried RC9 and got the same results (for both x86/x64 and the Sandybridge version). This issue seems to be part of the base imageflow project. I think it would make sense to close this ticket and open either a QEMU or Baseline build for older processors ticket in the main imageflow project. Does it make more sense for just a generic x64/x86 build instead of it being "older processors" From what I read, QEMU is a processor emulator anyway so who knows what it might identify as.

Thanks

[lilith]

Does this build work? It's baseline x86_64: https://ci.appveyor.com/api/buildjobs/5946k4fmuujhu26j/artifacts/imageflow-nightly-master-846-v1.0.0-rc9-2-g8d701f6-win-x64.zip

[RossLieberman]

As a matter of fact it does fix the issue. Kestrel didn't crash and the image was resized!!! Thanks for taking care of this.

[lilith]

Definitely an instruction set problem.

On Oct 19, 2017 1:16 PM, "Ross Lieberman" notifications@github.com wrote:

Closed #2 https://github.com/imazen/imageflow-dotnet/issues/2.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/imazen/imageflow-dotnet/issues/2#event-1301755579, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGln-H3hZESDi1OPQa9jgJdBJzTjhe0ks5st6ADgaJpZM4P5i0a .