search

microsoft / Windows-Machine-Learning

Samples and Tools for Windows ML.
https://docs.microsoft.com/en-us/windows/ai/
MIT License
1.03k stars 446 forks source link

How to generate a VideoFrame array using a normalized cv::Mat Image (float pixel format). #463

Closed iqedgarmg closed 2 years ago

iqedgarmg commented 2 years ago

I'm submitting a…

Sample app request : How to create a VideoFrame using a cv::Mat with CV_64FC3 Format.

Current behavior

There is only documentation available of how to convert a CV_8UC3 format.

Expected behavior

Further documentation of how to introduce a preprocessed image (with normalized pixels) as input to a ::Windows::AI::MachineLearning::LearningModel.

Minimal reproduction of the problem with instructions

At present, there is documentation available of how to convert a cv::Mat array with pixels in format CV_8UC3 (i.e, between the range [0, 255]) to a VideoFrame array:

https://github.com/Microsoft/Windows-Machine-Learning/tree/master/Samples/CustomTensorization

Nevertheless, my model was trained using normalized images (i.,e, pixels in the range [-1.0, 1.0] after normalization). After looking in the documentation for several days, I cannot a find a way to achieve this conversion.

Environment

Windows Build Number:

App min and target version: min: 10.0.17763.0 target: 10.0.19041.0,

OS Version (Server, IoT Core, Desktop, etc): Windows 10 Desktop

WinMLTools specific:

Visual Studio

smk2007 commented 2 years ago

It seems like you are trying to bind a 3-channel double precision (64bit) floating point image in the range of [-1, 1] (since you mentioned that the model is preprocessed).

Since you mentioned that your model was created "using normalized images... in the range [-1.0, 1.0] after normalization," can you pass in the byte buffer natively using TensorDouble without needing to wrap it in a VideoFrame?

nums11 commented 2 years ago

Closing due to inactivity.

ALONGNEU commented 2 years ago

Hi @smk2007 I have a similar question, how to create a VideoFrame object using a cv::Mat with CV_32FC3 Format. Based on your answer, do you mean I need to create a TensorFloat object and bind it with the input layer instead of the VideoFrame object? So, is there any introduction or samples for that? thx

smk2007 commented 2 years ago

Hi @ALONGNEU,

Please take a look at the Windows ML Samples Gallery: OpenCV Interop sample.

It contains examples of how to wrap cv::Mat in various ways that can be bound into WinML.

1) Bind as a VideoFrame (via SoftwareBitmap): https://github.com/microsoft/Windows-Machine-Learning/blob/673410d94fb1c344ca21f98b1e2ea5ac692069ce/Samples/WinMLSamplesGallery/WinMLSamplesGalleryNative/OpenCVImage.cpp#L58 You can create a VideoFrame from the SoftwareBitmap like so: https://github.com/microsoft/Windows-Machine-Learning/blob/aa5b5b8bbde5554573e8151f07a0bc4fdde81ef9/Samples/StyleTransfer/ImageHelper.cs#L168

2) Bind as a Tensor: https://github.com/microsoft/Windows-Machine-Learning/blob/673410d94fb1c344ca21f98b1e2ea5ac692069ce/Samples/WinMLSamplesGallery/WinMLSamplesGalleryNative/OpenCVImage.cpp#L46

ALONGNEU commented 2 years ago

Hi @ALONGNEU,

Please take a look at the Windows ML Samples Gallery: OpenCV Interop sample.

It contains examples of how to wrap cv::Mat in various ways that can be bound into WinML.

  1. Bind as a VideoFrame (via SoftwareBitmap): https://github.com/microsoft/Windows-Machine-Learning/blob/673410d94fb1c344ca21f98b1e2ea5ac692069ce/Samples/WinMLSamplesGallery/WinMLSamplesGalleryNative/OpenCVImage.cpp#L58

    You can create a VideoFrame from the SoftwareBitmap like so: https://github.com/microsoft/Windows-Machine-Learning/blob/aa5b5b8bbde5554573e8151f07a0bc4fdde81ef9/Samples/StyleTransfer/ImageHelper.cs#L168

  2. Bind as a Tensor: https://github.com/microsoft/Windows-Machine-Learning/blob/673410d94fb1c344ca21f98b1e2ea5ac692069ce/Samples/WinMLSamplesGallery/WinMLSamplesGalleryNative/OpenCVImage.cpp#L46

Thanks a lot, I will study it and have a try.