JPEG XL / JPEG XR / AVIF / HEIF / Webp Encoders and Decoders

[petsuter]

System.Windows.Media.Imaging supports only e.g. Gif, Icon, Jpeg, Png, Tiff, Wmp encoders and decoders but not e.g. JPEG XL, JPEG XR, AVIF, HEIF, Webp encoders and decoders.

[singhashish-wpf]

@petsuter Would you please explain about the impact of not having these?

[petsuter]

Not having these means .NET applications can not load or save modern image file formats and must use old image file formats instead that require larger file sizes, worse image quality, less features, and reject valid modern images and not being compatible with other programs that produce such modern image formats.

"The average WebP file size is 25%-34% smaller compared to JPEG file"

"AVIF files are much smaller than the corresponding JPEG or PNG files"

"JPEG XL offers significantly better image quality and compression ratios than legacy JPEG"

(Each link contains a lot more information about the impact.)

[lindexi]

In fact, the WPF will call the WIC to decode the image file.

[ThomasGoulet73]

Like @lindexi said, WPF uses WIC and I don't see WPF using something else or adding support for custom codecs not supported by WIC. But, AVIF, HEIF and Webp are already supported by using Windows extensions which adds other codecs in WIC. AVIF requires AV1 Video Extension, HEIF requires HEIF Image Extensions and Webp requires Webp Image Extensions. JPEG XL and JPEG XR do not seem to be supported yet.

[petsuter]

(Maybe https://github.com/mirillis/jpegxl-wic?)

Relying on WIC and providing only WIC supported formats sounds perfectly reasonable. But can WPF apps take advantage of these extensions? It seems WPF would have to provide subclasses of e. g. BitmapEncoder and pass the right WIC GUIDs to the internal constructor, right?

[miloush]

BitmapDecoder.Create does exactly that. If the file is none of the known decoders, you get an instance of UnknownBitmapDecoder back.

[petsuter]

I now confirmed that BitmapDecoder.Create can indeed load AVIF, HEIF, and Webp files, if the WIC extensions linked above are installed first.

There's also BitmapEncoder.Create, but that doesn't seem to work for me with the Guid from CLSID_WICHeifEncoder even with the HEIF Image Extensions installed. Any ideas?

Guid wicHeifGuid = new Guid(0x0dbecec1, 0x9eb3, 0x4860, 0x9c, 0x6f, 0xdd, 0xbe, 0x86, 0x63, 0x45, 0x75);
var encoder = BitmapEncoder.Create(wicHeifGuid);
encoder.Frames.Add(bitmapFrame);
encoder.Save(File.Create("test.heif"));

System.NotSupportedException
  HResult=0x80131515
  Message=No imaging component suitable to complete this operation was found.
  Source=PresentationCore
  StackTrace:
   at System.Windows.Media.Imaging.BitmapEncoder.EnsureUnmanagedEncoder()
   at System.Windows.Media.Imaging.BitmapEncoder.Save(Stream stream)

[miloush]

BitmapEncoder.Create effectively calls IWICImagingFactory::CreateEncoder which according to https://docs.microsoft.com/en-us/windows/win32/api/wincodec/nf-wincodec-iwicimagingfactory-createencoder expects a container GUID rather than an encoder GUID.

That said, it still does not seem to be enough to be able to encode HEIF. I am seeing random suggestions on the internet that HEVC is required (see also HEIF Format Overview).

[petsuter]

For HEIF:

        Guid wicHeifContainerGuid = new Guid(0xe1e62521, 0x6787, 0x405b, 0xa3, 0x39, 0x50, 0x07, 0x15, 0xb5, 0x76, 0x3f);
        var encoder = BitmapEncoder.Create(wicHeifContainerGuid);
        encoder.Frames.Add(bitmapFrame);
        encoder.Save(stream);

(without buying HEVC) leads to:

System.NullReferenceException
  HResult=0x80004003
  Message=Object reference not set to an instance of an object.
  Source=PresentationCore
  StackTrace:
   at System.Windows.Media.StreamAsIStream.Write(Byte[] buffer, UInt32 cb, UInt32& cbWritten)
   at System.Windows.Media.StreamAsIStream.Write(StreamDescriptor& pSD, Byte[] buffer, UInt32 cb, UInt32& cbWritten)
   at MS.Win32.PresentationCore.UnsafeNativeMethods.WICBitmapEncoder.Commit(SafeMILHandle THIS_PTR)
   at System.Windows.Media.Imaging.BitmapEncoder.Save(Stream stream)

and for Webp:

        Guid wicWebpContainerGuid = new Guid(0xe094b0e2, 0x67f2, 0x45b3, 0xb0, 0xea, 0x11, 0x53, 0x37, 0xca, 0x7c, 0xf3);
        var encoder = BitmapEncoder.Create(wicWebpContainerGuid);
        encoder.Frames.Add(bitmapFrame);
        encoder.Save(stream);

still leads to:

System.NotSupportedException
  HResult=0x80131515
  Message=No imaging component suitable to complete this operation was found.
  Source=PresentationCore
  StackTrace:
   at System.Windows.Media.Imaging.BitmapEncoder.EnsureUnmanagedEncoder()
   at System.Windows.Media.Imaging.BitmapEncoder.Save(Stream stream)

and the same with Guid wicWebpContainerGuid = new Guid(0x1f122879, 0xeba0, 0x4670, 0x98, 0xc5, 0xcf, 0x29, 0xf3, 0xb9, 0x87, 0x11); from here.

And for AVIF I don't even know what to try.

[miloush]

"No imaging component suitable to complete this operation was found." is pretty clear that this format is not available/installed. As far as I know there is no built-in encoder for Webp and AVIF.

The exception you are getting from HEIF is quite confusing to me. I am getting

System.Runtime.InteropServices.COMException
  HResult=0xC00D5212
  Message=No suitable transform was found to encode or decode the content. (Exception from HRESULT: 0xC00D5212)
  Source=PresentationCore
  StackTrace:
   at System.Windows.Media.Imaging.BitmapEncoder.SaveFrame(SafeMILHandle frameEncodeHandle, SafeMILHandle encoderOptions, BitmapFrame frame)
   at System.Windows.Media.Imaging.BitmapEncoder.Save(Stream stream)

as a result from UnsafeNativeMethods.WICBitmapFrameEncode.Commit, without StreamAsIStream being called back. I thought this might correspond to a missing component on the way such as HEVC.

However your call stack suggests you are doing possibly better than me. I cannot figure out how you can get NullReferenceException from Write though. The only thing that I can see can be null is dataStream but there are multiple checks to prevent that - in the IStreamFrom, in the constructor, in Verify. Are you able to debug this, most interestingly see whether the buffer actually contains meaningful data?

JPEG XL and JPEG XR do not seem to be supported yet.

JPEG XR seems to be served by WmpBitmapEncoder: https://docs.microsoft.com/en-us/windows/win32/wic/jpeg-xr-codec

[miloush]

To summarize, WPF relies on WIC for decoding and encoding images. Future and 3rd party codecs are supported as long as they implement the WIC interfaces. Free decoders seem to be available for all the requested formats. Many of them are lacking an encoder though (which usually need to be licensed). It is unclear to me whether WPF is the correct place to request new codecs built into Windows.

That said, WPF has some space for improvement:

• Classes for recently added built-in decoders (and encoders) could be added, allowing to set parameters.
• The type system could be opened so that
  • developers can add decoder/encoder classes to existing WIC components
  • non-WIC decoders/encoders can be provided, including purely managed
• support for new pixel formats
• support for IWICDevelopRaw

[miloush]

NullReferenceException

OK I installed HEVC and now get your NullReferenceException, you must have had the encoder from before (I think it used to be part of Windows).

The buffer has data, but it looks like it's the out uint cbWritten parameter of StreamAsIStream.Write!

That does look like a bug.

[petsuter]

In my debugger it also shows the exception on cbWritten = 0;.

[miloush]

Yeah I am thinking the caller is not interested in the number of bytes written and passes null pointer to the method, as allowed per https://docs.microsoft.com/en-us/windows/win32/api/objidl/nf-objidl-isequentialstream-write

And the wrapper does not expect that.

[miloush]

Indeed, I confirm that fixing that in the wrapper works and I get a valid HEIF file.

[miloush]

@petsuter It sounds like we more or less covered the ability to decode the requested formats (or subset of their supported pixel formats). I would suggest closing this issue and opening new ones, probably even one per request - for example AVIF and Webp encoders might have different licensing conditions and different impact. I think it would also be helpful mentioning which other platforms/software support saving these file formats that you cannot compete with.