Better support for different pixel formats in CanvasBitmap and CanvasVirtualBitmap

[benstevens48]

The function DefaultBitmapAdapter::CreateWicBitmapSource attempts to support HDR pixel formats, but I think this function could do better in terms of its pixel format support. Firstly I don't see why it needs to check that the container format is JPEG-XR - surely it is sufficient to look at the pixel format. Secondly, instead of checking for an exact match of pixel formats it could do something like this function.

Windows::Graphics::DirectX::DirectXPixelFormat WicHelper::GetBestDirectXPixelFormatForBitmapSourceFormat(com_ptr<IWICImagingFactory> const& pFactory, WICPixelFormatGUID const& sourcePixelFormat, uint32_t maxBitsPerPixel) {  //Output can be B8G8R8A8UIntNormalized, R16G16B16A16UIntNormalized or R16G16B16A16Float or R32G32B32A32Float. If maxBitsPerPixel is 0 then it is ignored. The minimum bits per pixel is 32.
        //todo: DDSformats?
        if (maxBitsPerPixel != 0 && maxBitsPerPixel < 64) {
            return Windows::Graphics::DirectX::DirectXPixelFormat::B8G8R8A8UIntNormalized;
        }
        try {
            com_ptr<IWICPixelFormatInfo2> pPixelFormatInfo = GetPixelFormatInfo(pFactory, sourcePixelFormat);
            WICPixelFormatNumericRepresentation numericRepresentation;
            check_hresult(pPixelFormatInfo->GetNumericRepresentation(&numericRepresentation));
            uint32_t bitsPerChannel = GetMaxBitsPerChannel(pPixelFormatInfo);
            WICPixelFormatGUID inputFormat;
            check_hresult(pPixelFormatInfo->GetFormatGUID(&inputFormat));
            if (numericRepresentation == WICPixelFormatNumericRepresentationFixed || numericRepresentation == WICPixelFormatNumericRepresentationFloat) {
                if (bitsPerChannel > 16 && (maxBitsPerPixel == 0 || maxBitsPerPixel >= 128)) {
                    return Windows::Graphics::DirectX::DirectXPixelFormat::R32G32B32A32Float;
                } else {
                    return Windows::Graphics::DirectX::DirectXPixelFormat::R16G16B16A16Float;
                }
            } else {
                if (bitsPerChannel > 8) {
                    return Windows::Graphics::DirectX::DirectXPixelFormat::R16G16B16A16UIntNormalized;
                } else {
                    return Windows::Graphics::DirectX::DirectXPixelFormat::B8G8R8A8UIntNormalized;
                }
            }
        } catch (hresult_error ex) {
            return Windows::Graphics::DirectX::DirectXPixelFormat::B8G8R8A8UIntNormalized;
        }
    }

    uint32_t WicHelper::GetMaxBitsPerChannel(com_ptr<IWICPixelFormatInfo2> const& pPixelFormatInfo) {
        try {
            WICPixelFormatGUID pixelFormat;
            check_hresult(pPixelFormatInfo->GetFormatGUID(&pixelFormat));
            if (pixelFormat == GUID_WICPixelFormat16bppBGR565) {
                return 6;
            } else if (pixelFormat == GUID_WICPixelFormat16bppBGRA5551) {
                return 5;
            } else if (pixelFormat == GUID_WICPixelFormat32bppRGBA1010102 || pixelFormat == GUID_WICPixelFormat32bppRGBA1010102XR) {
                return 10;
            }
            UINT bitsPerPixel;
            check_hresult(pPixelFormatInfo->GetBitsPerPixel(&bitsPerPixel));
            UINT channelCount;
            check_hresult(pPixelFormatInfo->GetChannelCount(&channelCount));
            if (channelCount > 0) {
                return bitsPerPixel / channelCount;
            } else {
                return 8;
            }
        } catch (hresult_error ex) {
            return 8; //Assume 8 in case of error
        }
    }

And then use this to get the corresponding WIC formats

    WICPixelFormatGUID WicHelper::GetCorrespondingWicPixelFormat(Windows::Graphics::DirectX::DirectXPixelFormat suggestedPixelFormat) {  //Input can be B8G8R8A8UIntNormalized, R16G16B16A16UIntNormalized or R16G16B16A16Float or R32G32B32A32Float. Assumes premultipled alpha.
        if (suggestedPixelFormat == Windows::Graphics::DirectX::DirectXPixelFormat::R16G16B16A16UIntNormalized) {
            return GUID_WICPixelFormat64bppPRGBA;
        } else if (suggestedPixelFormat == Windows::Graphics::DirectX::DirectXPixelFormat::R16G16B16A16Float) {
            return GUID_WICPixelFormat64bppPRGBAHalf;
        } else if (suggestedPixelFormat == Windows::Graphics::DirectX::DirectXPixelFormat::R32G32B32A32Float) {
            return GUID_WICPixelFormat128bppPRGBAFloat;
        } else {
            return  GUID_WICPixelFormat32bppPBGRA;
        }
    }

Note this function does not consider the DDS formats. You could also add some sort of option to the CanvasVirtualBitmap and CanvasBitmap LoadAsync methods to specify the format (for example here I have added a max pixel size because loading a full 128bpp image takes a lot of memory and is unnecessary if it is going to be drawn to a 32bpp swap chain), but then again for simplicity you might want to omit this option.

In addition, it would then be helpful to add a PixelFormat property to the CanvasVirtualBitmap so it is possible to tell what format is being used. This is actually necessary in order to determine what color space conversion to then apply. (I already created an issue for this).

I have been trying to solve this using interop, but I have run into another issue whereby interop causes the app to hang (I've already submitted that issue). I hope you will consider implementing this. I am happy to do a bit of work on it, although I am not very familiar with WRL (I'm using C++/WinRT).

[benstevens48]

I realised my initial title was incorrect - I was testing CreateImageSourceFromWic directly without setting the alpha mode to premultiplied which was the problem. However, the remaining comments still stand.

[shawnhar]

The HDR support in Win2D image loading is definitely on the simplistic side. I don't remember exactly why we limited this to Jpeg-XR format - probably a combination of caution (avoid risk of regressing existing functionality while adding HDR support) plus this is the only WIC format used by Win2D that is actually capable of encoding HDR data.

Automatic load time format conversions is a big can of worms, particularly if you look at things like going between unorm and float formats. What's the right behavior if range is reduced and data must be truncated? Should the conversion tonemap, and if so in what colorspace? Advanced HDR apps need to deal with these questions (or might just pick a specific format and require that the hardware support it) but that's a high level of complexity beyond what we felt made sense for the high-level-and-simple-but-doesn't-expose-all-the-knobs style of Win2D image loading helper. So we just took the easy option of preserving HDR data only if that could map directly to the hardware.

What cases are you seeing where image files don't map directly to GPU format capabilities? It would be helpful to understand details of what the formats involved here are.

[benstevens48]

Hi Shawn, Ok, I agree automatic conversion might be complicated. So perhaps the better suggestion would be to have an addition Load overload that specifies a DirectX pixel format (only a few would be supported) and the load function would convert to the corresponding WIC format before creating the virtual bitmap from the WIC bitmap source. The unknown pixel format could correspond to the existing automatic conversion behaviour. In addition, you could expose the pixel format used at a property of the VirtualCanvasBitmap (although not sure how it would work with interop).

In terms of use cases - I'm developing a photo viewing app and would like to support high-quality rendering or image processing of as many formats as possible. In terms of formats I have actually seen people using, I guess 64bpp uint is the main high bit-depth format I have seen, which is supported by PNG, TIFF and HEIC as well as JPEG-XR.