saucecontrol / PhotoSauce

MagicScaler high-performance, high-quality image processing pipeline for .NET
http://photosauce.net/
MIT License
582 stars 49 forks source link

[Feature Request] Preserve source image colorspace and embedded ICC Profile #13

Closed andreas-eriksson closed 6 years ago

andreas-eriksson commented 6 years ago

Are there some settings that can be used to mimic the behavior of scaling an image with GDI+?

I have tried using the below settings but high resolution images still produce thumbnails that differ from GDI+.

var settings = new ProcessImageSettings
{
    Width = 400,
    Height = 400,
    ResizeMode = CropScaleMode.Max,
    JpegQuality = 0,
    HybridMode = HybridScaleMode.Turbo,
    SaveFormat = FileFormat.Jpeg,
    Sharpen = false
};

My goal is to produce thumbnails that look the same to my image processing algorithm as it is much faster to run the algorithm on smaller images. Scaling using GDI+ produces very similar results with or without scaling. MagicScaler seems to produce slightly different images especially if the source image is of very high resolution, e.g. 15204x4942 or 8688x5792.

I have also tried using HybridScaleMode.Off but that was actually worse.

saucecontrol commented 6 years ago

There are a couple of possible answers here. I'll start with the simple one.

The following settings will produce output nearly identical to GDI+, assuming you're using the highest-quality settings for DrawImage.

MagicImageProcessor.EnablePlanarPipeline = false;
var settings = new ProcessImageSettings
{
    Width = 400,
    Height = 400,
    ResizeMode = CropScaleMode.Max,
    HybridMode = HybridScaleMode.Off,
    BlendingMode = GammaMode.sRGB,
    Interpolation = InterpolationSettings.Cubic,
    SaveFormat = FileFormat.Jpeg,
    JpegQuality = 75,
    Sharpen = false
};

The HighQualityBicubic interpolation DrawImage is adaptive -- in an undocumented way -- so results will likely not be 100% identical, but that's as close as MagicScaler gets, and it still works about twice as fast and with a small fraction of the GDI+ memory usage.

And now for some more detail...

It seems you've discovered that both the interpolation and sharpening in MagicScaler are also adaptive. The adaptive logic is a heuristic that I developed in working with some fairly large image sets, but there can certainly be cases where it may pick less than ideal settings. I'm always looking to improve that if you have some sample images that highlight problems.

In general, the sharpening is increased as scaling ratio increases, so scaling from very large to very small will have pronounced sharpening. That is, of course, customizable if you can work out settings that you prefer (although again, if there's a change I can make to the heuristic, I'd be all for that).

Also, the automatic interpolation selection will choose a sampler with a smaller window when performing a very high ratio resize with hybrid scaling disabled. The defaults for interpolation and hybrid scaling are meant to work together to give good speed and quality, but there may be room for improvements there.

The sRGB blending mode used by GDI+ is almost always inferior to the default Linear mode used by MagicScaler. I included that setting above because it's what GDI+ uses, but it's less mathematically correct and less visually-pleasing than Linear blending.

And finally, the EnablePlanarPipeline setting disables an optimization in MagicScaler that handles YCbCr images differently than RGB images. If your input images are JPEG, that optimization is enabled by default, but since it's something that GDI+ doesn't do, I included it above. This is unlikely to make any visible difference in the output images, but it does make a significant improvement in performance. It's something else you can experiment with.

I hope that helps, and again, if you can provide samples that show problems or room for improvement, that would be most appreciated.

andreas-eriksson commented 6 years ago

Thank you for the very detailed response.

It turns out that we actually used the below settings for scaling with GDI+.

Bitmap outputBitmap = new Bitmap(newWidth, newHeight);
using (Graphics gr = Graphics.FromImage(outputBitmap))
{
    gr.SmoothingMode = SmoothingMode.HighSpeed;
    gr.InterpolationMode = InterpolationMode.Default;
    gr.PixelOffsetMode = PixelOffsetMode.HighSpeed;
    gr.DrawImage(sourceBitmap, 0, 0, newWidth, newHeight);
}

I understand from you blog posts that SmoothingMode has no effect and that InterpolationMode.Default is actually Linear.

The settings you proposed seems work better for images that are sRGB but there are still some images that seems to have a built in color profile that produces thumbnails that yields different results with our image algorithm compared to the original image. Could it be that GDI+ handles built in color profiles differently than MagicScaler? The images "work" if I re-save them to another color profile using GIMP. Example image: https://www.dropbox.com/s/6t8r84gazkuugxl/1H9A9341.jpg?dl=0 "Adobe RGB (1998)" Example image: https://www.dropbox.com/s/9uypzmdf831ns1b/6.jpg?dl=0 "Generic RGB Profile"

I have also tested by changing the GDI+ scaling to

Bitmap outputBitmap = new Bitmap(newWidth, newHeight);
using (Graphics gr = Graphics.FromImage(outputBitmap))
{    
    gr.InterpolationMode = InterpolationMode.HighQualityBicubic;
    gr.PixelOffsetMode = PixelOffsetMode.Half;
    gr.DrawImage(sourceBitmap, 0, 0, newWidth, newHeight);
}

and that seems to make no difference to our image algorithm, at least not for high resolution images.

P.S. A KeyNotFoundException is thrown when resizing gray scale jpeg images and using GammaMode.sRBG. Example image: https://www.dropbox.com/s/vthptxs8op4t557/7.jpg?dl=0

saucecontrol commented 6 years ago

Thanks for the extra info. That's helpful.

The ICC color profile handling in GDI+ is similar to MagicScaler's... if you enable it. That behavior is set when loading the image initially. Most of the Image.FromStream and Image.FromFile overloads have a Boolean parameter for useEmbeddedColorManagement. Like such:

public static Image FromStream(
    Stream stream,
    bool useEmbeddedColorManagement,
    bool validateImageData
)

If that parameter is set to true, GDI+ will read the embedded profile and perform a conversion to the sRGB colorspace when loading the image. Because GDI+ doesn't save color profiles on output, and because all of its internal processing assumes it's working in sRGB, that's really the only correct way to use it.

Put simply, when you process an image with a color profile embedded, there are 3 options: 1) Convert to a common colorspace (sRGB) on load and work with that colorspace going forward. This ensures consistent output that has max compatibility with any software reading the output. 2) Treat the image data as color-agnostic but preserve the color profile and save it in the output image. This is also valid as long as the algorithms in use don't make assumptions about the colorspace of the pixels, but it requires the software reading the output image be color profile-aware. It's a less compatible option. 3) Pretend the color profile doesn't exist and treat the image as whatever colorspace you want (usually sRGB). This option is just plain wrong, as it will result in color distortions, and that's specifically what the profile is there to prevent.

GDI+ doesn't support option 2, so you have to pick between 1 and 3, which is what that boolean parameter does. MagicScaler uses option 1 and doesn't give you a choice. So basically, with the sample images you linked, GDI+ and MagicScaler produce the same output colors, but only if you turn on color management in GDI+.

There are limitations to the way GDI+ handles profile conversion, however. Correct CMYK to RGB conversion requires a color profile, but GDI+ doesn't use the profile in that scenario, so its output colors are always incorrect when reading CMYK (or YCCK) images. MagicScaler does that correctly and automatically.

The error you got on the greyscale JPEG is simply a missing mapping in the MagicScaler color converter. Thanks for pointing that out. I'll get a fix out soon.

As for the rest of the compatibility/consistency between GDI+ and MagicScaler, the important thing is that if you disable the adaptive processing options in MagicScaler and then match its static settings with what you're using in GDI+ (i.e. no sharpening, no hybrid scaling, and a fixed interpolation mode), they'll produce very similar if not identical output. Where they differ, I'd argue that MagicScaler is more correct (e.g. in linear light processing), so it's probably not wise to try to match them exactly.

andreas-eriksson commented 6 years ago

Thanks, I think my problems where strictly related to color profiles and had very little to do with scaling.

I was temporarily thinking about asking if there is a way of disabling it in MagicScaler but I think it actually handles it in the correct way.

Do you have any idea of why .NET/GDI+ does not have useEmbeddedColorManagement enabled by default? It seems as kind of a hidden setting not many people would know about.

P.S. Thanks for an excellent image scaling library. I am really impressed with the performance.

andreas-eriksson commented 6 years ago

On second thought, would it be possible to add an option to ProcessImageSettings to disable the color profile conversion?

I am not sure if I can/are allowed to change the code of our image algorithm to handle color profiles.

saucecontrol commented 6 years ago

I wouldn't rule it out completely. If there were a compelling use case for ignoring color profiles, it would be an easy change to make. On the other hand, matching the output of an existing poorly-implemented piece of software isn't exactly a compelling use case.

To answer your question about why GDI+ would hide something as important as color management under an easy-to-miss parameter, it's kind of just par for the course. They hide things like the WrapMode in even more obscure places. And the poor documentation makes it difficult to pick a good interpolation algorithm or to know what PixelOffsetMode does, etc. And of course, doing color management is more expensive than not doing it. Most of their defaults sacrifice image quality for speed, even if it means doing something that's never the right thing to do.

Take these three images, for example. In whatever web browser you're using, they should look the same. They'll also look the same in Photoshop, GIMP, Windows Photo Viewer... basically any imaging software that doesn't suck. But try them in GDI+ with color management disabled and see what you get.

Or take this image and resize it with PixelOffsetMode.Default and ask yourself why anyone would ever choose that or why Microsoft made it a default setting.

Or try resizing this image with any of the low-quality interpolation modes in GDI+. You mentioned that you couldn't see a difference between InterpolationMode.Default and InterpolationMode.HighQualityBicubic. This image has the type of detail that turns into interference patterns (moiré) with poor interpolation. It probably also looks bad in your browser, because browsers generally have bad-quality image scaling.

I guess I'm having trouble understanding where MagicScaler fits in with your software if it's so dependent on a GDI+ implementation that uses bad settings. What do you gain from MagicScaler if you change all of its settings to give the worst possible quality?

One of my main design goals for MagicScaler was to make it easy to get things right for developers who don't know all the obscure details of how imaging software works. Color management, Exif Orientation, Interpolation algorithms, and Gamma Correction are not widely-understood topics among developers, but most libraries require a developer to know those things in order to get imaging right. And really, the more options I add to MagicScaler, the more potentially-confusing things there are for my users to worry about.

andreas-eriksson commented 6 years ago

Hi, our use case isn't visual image quality. We are instead investigating your scaling library in order to speed up the calculation of Microsoft PhotoDNA by resizing the images before the calculation is made.

Our problem is that we are using a library supplied by Microsoft and we can't change that code. The same library is used by many different organizations to calculate the PhotoDNA hash so I cannot really make any changes that could risk the hash being calculated differently.

See more here: https://www.youtube.com/watch?v=NORlSXfcWlo

Would it be possible to have a different option for jpegs where the color profile is transferred to the output image? That would leave it up to the consumer to display the resized image correctly and respecting the color profile. It would also make it possible to disregard the color profile if necessary.

P.S. Thank you for your very detailed explanations, it has improved my understanding of the problems with resizing.

saucecontrol commented 6 years ago

That's really interesting, thanks. It's definitely not a use case I had considered.

I think your suggestion of preserving the profile through processing is fair. I had considered that at one point, but there can be problems introduced because of incompatibilities in file formats. For example, if you resize a JPEG with a color profile and then save it as a BMP or GIF, you can't save the profile because those older formats don't support them. But since most modern formats do support them, that may be ok.

It seems to me that if the PhotoDNA process completely ignores color profiles, that would be an easy way to subvert it. If you looked at the three photos I posted with the different color profiles, the first one of those has a crazy profile that I created specifically to make it clear when software gets color management wrong. If you open it in MSPaint, for example, it's a yellow bowl full of blue raspberries, because I essentially swapped the red and blue channels in that profile. Since conversion to greyscale weights the color channels differently, the greyscale representations of those photos would be quite different, even though the correct interpretation of them is that they are the same image. If fooling PhotoDNA is that simple, it's not really doing its job. Along the same lines, a color profile can be used to dramatically alter the gamma curve of an image, which would have an even more profound impact on its greyscale representation if the color profile were ignored.

In that respect, the most correct way of dealing with PhotoDNA would be to always ensure that you are feeding it images in a normalized colorspace, so that its own deficiencies in color handling don't impact your results. If, on the other hand, you're matching with a database created by other users allowing PhotoDNA to interpret the images incorrectly, I can see why that would be an issue for you.

As a quick test, I uploaded my crazy blue raspberry image to Google's image search, and while it says it couldn't match the image, the first web page match it shows is for pexels.com, which is exactly where I got the original. Despite my having resized and cropped it and assigned a new color profile, Google could tell it was the same image. If PhotoDNA can't make that same match, that's a bug for sure.

andreas-eriksson commented 6 years ago

I will report it as a bug but I don't know if Microsoft will fix it anytime soon, Would it be possible for you to implement the preserving of the color profile in the near future? Or do you prefer me to take a crack at it and supply a pull request?

saucecontrol commented 6 years ago

I stewed on this a bit while I was on vacation last week. I've got a rough plan for some internal improvements to the way MagicScaler handles colorspace conversion, and I believe the option to skip conversion entirely fits in with that. I've had a TODO item for a long time related to embedding or tagging the sRGB profile in output images, and I'm thinking I could add an option to select one of the following treatments of color profiles.

For your use case, you'd pick the PreserveSource option. The default would remain ConvertAndStrip.

As I mentioned, I have some related internal improvements planned to coincide with that change, so it'll probably be a couple of weeks before I can get all the research and testing done to make sure I get it all right. If you have any suggestions or other test cases, they'd be most welcome.

andreas-eriksson commented 6 years ago

Thanks, I would be happy to help out with testing in any way I can.

saucecontrol commented 6 years ago

This features is present in v0.9.0. See https://github.com/saucecontrol/PhotoSauce/blob/master/doc/main.md#colorprofilemode