is possible convert YUV surface to CanvasBitmap ?

[PatrickSCLin]

I'm developing a video streaming app via Win2D and FFmpeg,

cus Win2D just work in D2D format, so I have to do sws_scale,

a worst case like a big resolution 4000 x 3000 h264 streaming

it's really too slow if convert it from yuv to bgra in FFmpeg

I just noticed a test "Direct3DSurfaceInteropTests",

so I don't know the detail, but is possible that I bind yuv data to a d3d surface, and convert yuv to bgra in GPU or even better if Win2D can draw the surface directly

[shawnhar]

Win2D does not directly support YUV data formats. If you are willing to interop and include some native C++ code, you have several options:

1. Use ID3D11VideoContext::VideoProcessorBlt to convert YUV surfaces to BGRA, prior to processing them with Win2D. This is a fast hardware accelerated conversion path.
2. Use ID2D1DeviceContext2::CreateImageSourceFromDxgi to wrap an ID2D1ImageSource around your YUV surface(s), then interop the ID2D1ImageSource to a CanvasVirtualBitmap, which can be fed directly into any Win2D image processing operations. This is likely slightly more efficient than option 1, but requires Windows 10.
3. Potentially the fastest but also by far the most work is to keep your Y data in a separate image from the UV, and process the two parts separately. This avoids ever having to expand the data into RGB format, which can allow UV processing to run at reduced resolution compared to the Y plane, but is only an option if your image manipulation algorithms allow the planes to be processed separately in this way (not an option if you are just using built-in effects like sepia or highlights and shadows). You can then use the YCbCr Effect (not exposed in Win2D, but part of native D2D) to recombine your Y and UV data into a single image, or (depending on what format you need the output in) just keep these as separate planar bitmaps.

[PatrickSCLin]

my application is windows 10 universal app, is option 2 more easier and better option for me ?

[shawnhar]

Option 2 is probably the easiest, but these are all somewhat equivalent in that they require stepping outside of Win2D and writing some C++ D2D code to use alongside it.

[PatrickSCLin]

which D3D11_BIND_FLAG for this purpose ?

[shawnhar]

I haven't actually tried this, but I'd expect these need to be shader resource (the same as to use a texture with D3D).

[PatrickSCLin]

How to interop ID2D1ImageSource to CanvasVirtualBitmap?

I have already created ID2D1ImageSource but no idea how to do the last step.

[PatrickSCLin]

I just commit the code on github https://github.com/PatrickSCLin/YUVSample

[shawnhar]

Interop is documented here: http://microsoft.github.io/Win2D/html/Interop.htm

[PatrickSCLin]

I just finish the interop, but the image which be drawn is not incorrect.

I guess the UV is wrong, I'm confused, since I have to interop Win2D & Direct2D

I should use the CanvasDrawSession and send it into C++ to do interop it to D2D1Context to do CreateImageSourceFromDxgi

or I should use the D2D1Device which I created in C++ and send it to C# , interop it to CanvasDevice ?

this sample is what I done for now, https://github.com/PatrickSCLin/YUVSample

maybe you give me some hint from the code

[PatrickSCLin]

I can draw YUV frame on win2d now, but ...

does option 2 only based on NV12 format ?

I mean, most h264 file is decoded to be yuv420p frame from FFmpeg,

but position of UV is kind different with NV12, can I use yuv420p format directly in option 2?

[PatrickSCLin]

thanks for help :)

I just have to do a little function to swap UV by myself, and it display perfect.

[shawnhar]

Great to hear you got this working! How is the performance?

The incoming surfaces for CreateImageSourceFromDxgi don't have to be NV12. https://msdn.microsoft.com/en-us/library/windows/desktop/dn890791%28v=vs.85%29.aspx has a table documenting the various options - input can either be a single planar format texture, or a pair of textures containing separate Y and UV data.

[PatrickSCLin]

https://msdn.microsoft.com/en-us/library/windows/desktop/dn890791(v=vs.85).aspx

sorry, I'm trying to put yuv420p frame data directly to a CanvasVirturalBitmap by CreateImageSourceFromDxgi

since yuv420p is a planlar format, so in this case, I should create 3 textures for Y, U, V, all of them are R8 format, right ?

also I saw the document said

If multiple surfaces are provided, this method infers whether chroma planes are subsampled (by 2x) from the relative sizes of each corresponding source rectangle (or if the source rectangles parameter is NULL, the bounds of each surface). The second and third rectangle must each be equal in size to the first rectangle, or to the first rectangle with one or both dimensions scaled by 0.5 (while rounding up).

HRESULT CreateImageSourceFromDxgi(
  [in]                 IDXGISurface                        **surfaces,
  [in, optional] const D2D1_RECT_U                         *sourceRectangles,
                       UINT32                              surfaceCount,
                       DXGI_COLOR_SPACE_TYPE               colorSpace,
                       D2D1_IMAGE_SOURCE_FROM_DXGI_OPTIONS options,
  [out]                ID2D1ImageSource                    **imageSource
);

sourceRectangles [in, optional]
Type: const D2D1_RECT_U*
The regions of the surfaces to create the image source from.

the problem is, I can not see the argument sourceRectagles in my ID2D1DeviceContext2 interface

the function in my VisualStudio 2015 Update 1 is looks like this, and no other overload function:

HRESULT CreateImageSourceFromDxgi(
  [in]                 IDXGISurface                        **surfaces,
                       UINT32                              surfaceCount,
                       DXGI_COLOR_SPACE_TYPE               colorSpace,
                       D2D1_IMAGE_SOURCE_FROM_DXGI_OPTIONS options,
  [out]                ID2D1ImageSource                    **imageSource
);

what should I do ?

[PatrickSCLin]

the option3 only work when UV is packet format, right ?

[PatrickSCLin]

I'm using option2 for my project now, if the decoder runs well,

it should be fine to draw in 30fps, 4000 * 3000, on CPU i5 4460

thanks for help :)