search

saucecontrol / PhotoSauce

MagicScaler high-performance, high-quality image processing pipeline for .NET
http://photosauce.net/
MIT License
569 stars 49 forks source link

HEIC: Unsupported feature: Unsupported color conversion #102

Open cvdsoftware opened 1 year ago

cvdsoftware commented 1 year ago

I'm trying to process a heic file, but I got the following exception (using latest version):

System.InvalidOperationException
  HResult=0x80131509
  Message=Unsupported feature: Unsupported color conversion
  Source=PhotoSauce.NativeCodecs.Libheif
  StackTrace:
   at PhotoSauce.NativeCodecs.Libheif.HeifResult.Check(heif_error err)
   at PhotoSauce.NativeCodecs.Libheif.HeifContainer.HeifPixelSource.decodeImage()
   at PhotoSauce.NativeCodecs.Libheif.HeifContainer.HeifPixelSource.CopyPixelsInternal(PixelArea& prc, Int32 cbStride, Int32 cbBufferSize, Byte* pbBuffer)
   at PhotoSauce.MagicScaler.Transforms.ConversionTransform.copyPixelsDirect(PixelArea& prc, Int32 cbStride, Byte* pbBuffer)
   at PhotoSauce.MagicScaler.PixelSource.CopyPixels(PixelArea& prc, Int32 cbStride, Int32 cbBufferSize, Byte* pbBuffer)
   at PhotoSauce.Interop.Wic.IWICBitmapSourceImpl.copyPixels(IWICBitmapSource* pinst, WICRect* prc, UInt32 cbStride, UInt32 cbBufferSize, Byte* pbBuffer)
   at TerraFX.Interop.Windows.IWICBitmapFrameEncode.WriteSource(IWICBitmapSource* pIBitmapSource, WICRect* prc)
   at PhotoSauce.MagicScaler.WicImageEncoder.writeSource(IWICBitmapFrameEncode* frame, PixelSource src, PixelArea area)
   at PhotoSauce.MagicScaler.WicImageEncoder.WriteFrame(IPixelSource source, IMetadataSource meta, Rectangle area)
   at PhotoSauce.MagicScaler.MagicImageProcessor.WriteOutput(PipelineContext ctx, Stream ostm)
   at PhotoSauce.MagicScaler.MagicImageProcessor.ProcessImage(String imgPath, String outPath, ProcessImageSettings settings)
   at xxx

File properties: image image image

Windows Photos can show the file.

Unfortunately I can't share the file, can I send it privately to you?

saucecontrol commented 1 year ago

Yep, you can email it to me directly.

I expect if the Windows decoder can handle it, libheif should be able to. There were changes to the way it does conversions in the last few versions, and I don't have a ton of test images.

saucecontrol commented 1 year ago

Received the sample image by email, thanks.

The error is coming back from libheif, although I don't see immediately why it's unable to do the conversion. Ultimately this issue comes down to a change that libheif made around v1.7.0 when they added support for AVIF. Rather than allowing you to retrieve the raw YCbCr planar data that comes from the underlying video decoder along with enough information to correctly convert it, they decided to do the conversion internally.

The result of that change was that if you ask for YCbCr from the decoder, it double converts YcbCr->RGB->YCbCr. That is, of course, much slower (https://github.com/strukturag/libheif/issues/472) and, up until v1.12.0, resulted in incorrect colors due to a math error in the RGB->YCbCr side of the conversion (https://github.com/strukturag/libheif/issues/313).

The Windows HEIF codec (msheif_store.dll) doesn't share the same behavior/bugs, but it's worth noting it doesn't decode the sample correctly either -- there's a bright green line at the top of the result, indicating some kind of issue with interpretation of the chroma data. It could be that the image itself is broken in some way, or it could be a separate bug in the WIC decoder.

To sum up, it's likely I could handle that image at least as well as Windows if I could get the YCbCr data out from libheif, but it's attempting to force a conversion to RGB that it can't do, so I'm stuck.

My longer term plan is replace libheif with my own container parser that invokes the video frame decoders (libde265 and dav1d) directly, but I'm putting priority on other areas for the time being. I'll mark this as an external issue for now.

saucecontrol commented 3 months ago

Just leaving a few notes here for myself after looking at this in more detail...

Looks like the issue here is actually that the image has multiple frames (YUV420 Main + Y RangeExtension). iOS interprets the second frame as an alpha mask and outputs an image with a transparent border, while the WIC decoder ignores the second frame entirely. libheif returns the "Unsupported color conversion" error whether the requested format is RGB or RGBA -- it's unclear whether there is some combination of parameters that will actually allow decode.

The apparent chroma artifacts at the image top would seem to be unrelated to the issue. They show up in both the WIC and iOS decoded frames, but the container and both frame bitstreams pass all validation tests I've run on them.