Different result when resizing in another thread

[Skleni]

We are using MagicScaler in a WPF application to resize images and encountered a strange problem:

Depending on whether the call to ProcessImage happens on the main thread or in a background thread*, the result image is slightly different. Visually they are equal, you cannot see any difference, but the file size is different by a few bytes (86 000 vs 85 728]. This wouldn't be a problem in itself, but what is really strange is that when we analyze the two images using the Cognitive Services from Microsoft, there is a face detected in the image generated in the main thread, but no face detected in the image generated in the background thread.

Do you have any explanation or idea what is going on here? I can provide the images if that helps.

* Actually, it's even more complicated: It depends on the thread the first call to ProcessImage the app makes happens on. All subsequent calls seem to somehow be executed on the same thread, no matter which thread calls it. Is that on purpose?

[saucecontrol]

That's an interesting one. I'd definitely like to see some example images that show that behavior with Azure Computer Vision if you don't mind.

I probably should have some documentation to cover potential threading issues with WPF/Winforms apps. MagicScaler uses the COM-based WIC codecs internally, and that's where any threading irregularities would arise. The first call to a WIC codec sets it up in either STA or MTA mode (see here for slightly more detail), and it will stay in that threading model for the lifetime of the process. Since the UI thread of all Windows apps must be [STAThread], you get STA when you invoke WIC from there first and MTA if you invoke from any background thread first or from console or web apps.

Because the codecs support running in either threading model, it's possible for them to behave differently in STA vs MTA, although I've never observed actual output differences. I'd recommend as general practice never to use MagicScaler on your main/UI thread, and that should solve your issues.

[saucecontrol]

I should also point out that when I say you shouldn't use MagicScaler on your UI thread, I mean any part of it. For example, if you use ImageFileInfo to get an image size on your UI thread and then call ProcessImage on a background thread, you've still locked WIC in STA mode. By keeping everything WIC-related off the UI thread, you'll keep it in MTA mode, and any WIC operations can happen on any thread or multiple threads at once.

[Skleni]

Thanks for the quick reply. We tried specifying both STA and MTA on the background thread, it didn't change anything. I agree that the best solution would be not to call it on the UI thread at all, but currently that's the only way we can get the faces to be recognized. Currently we're calling ProcessImage on the background thread, but on application startup we're creating a new ImageFileInfo for a dummy image on the UI thread. This is enough that the faces are recognized, but the processing seems to happen on the UI thread as well - at least it freezes.

This is the code we're using:

MagicImageProcessor.ProcessImage(original, preview, new ProcessImageSettings()
{
    Width = 640,
    Height = 640,
    ResizeMode = CropScaleMode.Max,
    MetadataNames = new[]
    {
        "/app1/ifd/exif/{ushort=36867}", // DateTimeOriginal
        "/app1/ifd/exif/{ushort=33437}", // FNumber
        "/app1/ifd/exif/{ushort=41985}", // CustomRendered
        "/app1/ifd/gps/{ushort=1}",      // LatitudeRef
        "/app1/ifd/gps/{ushort=2}",      // Latitude
        "/app1/ifd/gps/{ushort=3}",      // LongitudeRef
        "/app1/ifd/gps/{ushort=4}"       // Longitude
    }
});

original no face face

[saucecontrol]

Thanks for the repro. I had a quick look at the output files, and they're subtly visibly different, although the JPEG settings appear to be the same (chroma subsampling and quantization tables match). I'll set that up in my WPF test harness and run through it in the next couple of days to see if I can figure out what's happening.

[Skleni]

Thanks!

[saucecontrol]

I was able to reproduce both of your output samples from the original image, but the answer had nothing to do with threading model (at least on my machine).

I can reproduce the 'face' sample by setting:

MagicImageProcessor.EnablePlanarPipeline = false;
MagicImageProcessor.EnableSimd = false;

and the 'no face' sample with:

MagicImageProcessor.EnablePlanarPipeline = true;
MagicImageProcessor.EnableSimd = false;

That leaves me with some questions about your environment.

First, what version of .NET are you testing on and what kind of hardware? The SIMD pipeline should work on any supported runtime and any supported hardware, so I'm curious why it's not being activated in your environment. That setting doesn't impact the output quality dramatically, but the perf impact can be very big.

Second, what version of Windows are you testing on? When the Planar Pipeline is enabled, it will auto-activate whenever the JPEG codec supports it. On Windows 10, there's no difference in codec support regardless of threading model, but perhaps that wasn't true on older versions?

From what I can tell, the Planar Pipeline is responsible for the difference you see in the face recognition results. The planar processing model can cause a slight color shift in the output, and in this example, there's a very slight shift toward yellow in the skintones. It's not visible to my eyes unless I zoom in to extreme levels with the images side by side, so I'm surprised it makes a difference to the computer vision models, but that seems to be the cause. You should be able to get all your usage on a background thread while keeping the good behavior by setting MagicImageProcessor.EnablePlanarPipeline = false

[Skleni]

Yes, with MagicImageProcessor.EnablePlanarPipeline = false it does indeed work as expected, thank you!

The target framework is .NET 4.6.2.

The machine I tested on:

Intel(R) Core (TM) i7-6700HQ CPU @ 2.60GHz Windows 10 Pro Version 1803 Build 17134.590

But originally the problem was reported by a user on another machine (which I don't have the details for unfortunately).

[saucecontrol]

Thanks for the update. I'm glad that worked out.

In thinking about it a bit more, I realized your app must be running in 32-bit mode, which explains why you're getting results that show the SIMD pipeline is disabled (32-bit netfx still uses the legacy JIT instead of RyuJIT, so no SIMD support). The WIC codecs may behave differently in 32-bit also, so that may explain the threading model behaviors you saw.

I'll go ahead and close this since EnablePlanarPipeline fixed it for you, and I'll add some notes to the docs about that and the threading model stuff. You might want to consider disabling the "Prefer 32-bit" option on your app if you don't have any dependencies that are incompatible with running in x64 mode. That will improve performance as well as image accuracy.

[Skleni]

Thanks, I'll look into that!