Radiometric data from Windows .Net application

[claybar]

Hi,

I am trying to get raw Y16 data from a Lepton2.5 module with radiometric mode enabled. Unfortunately I can't seem to find a combo of libraries that gives the raw data stream as Y16. I thought OpenCV (via Emgu) would allow this, but no matter what settings are applied to OpenCV it appears to automatically convert the incoming stream from Y16 to RGB24. This produces a strange colour mapping, best guess is the Y16 data has been assumed to be YUV data.

Basic OpenCV code with attempted non-conversion looks a bit like this:

Capture capture = new Capture(deviceIndex);
capture.SetCaptureProperty(Emgu.CV.CvEnum.CapProp.ConvertRgb, 0);

There is also something a bit strange with the FourCC setting of the camera, retrieving the

double fourCCd = capture.GetCaptureProperty(Emgu.CV.CvEnum.CapProp.FourCC);
uint fourCCui = (uint)fourCCd;
string s = new string(System.Text.Encoding.UTF8.GetString(BitConverter.GetBytes(fourCCui)).ToCharArray());
Console.WriteLine(string.Format("FourCC: double: {0}, string: {1}", fourCCd, s));

gives a result of some strange value:

FourCC: double: -466162819, string: }?6?

Attempting to set the FourCC value make snot difference to the returned value.

I have compiled the firmware with the Y16 symbol set and can successfully receive Y16 data on a linux platform using V4L2, but no such luck on windows. I have attempted to compile groupgets/libuvc but this seems to be a tricky prospect on windows, especially for use within .net applications.

Any ideas?

[kekiefer]

Yeah, this is a problem for OpenCV's Capture module on every platform, not just Windows. It insists on trying color formats no matter what you do. There's at least two options, either patch OpenCV's Capture module code to allow you to actually download Y16, or use DirectShow. The devs for FLIR who ported their Windows Lepton GUI to work with the PT1 used DirectShow, but unfortunately they didn't release the source for that effort. But we sure would appreciate some example code running with DirectShow if you do end up going down this road!

[kekiefer]

@claybar any updates on your work to do radiometry from Windows?

[claybar]

I almost got this working using directshowlib, but never found the right combo in the filter graph. I suspect one of the undocumented media subtypes will work, but it might be trial and error to find which one.

[kekiefer]

Not sure if it helps (I know nothing about the Windows APIs for this), but someone told me about media foundation as well. I took a quick look through the docs and did notice MFVideoFormat_L16, but I'm unsure what UVC GUID this would be looking for.

This is also particularly tantalizing! https://docs.microsoft.com/en-us/windows-hardware/drivers/stream/infrared-stream-support-in-uvc

If you do find anything more about these formats and if anything from the firmware might be required to use them please reopen this issue.

[ghellwig]

Dear @kekiefer,

I know this issue is quite old, but from Microsoft's documentation it looks like the following frame descriptor needs to be included in the firmware

// Example Format Descriptor for UVC 1.1 frame based format

typedef struct _VIDEO_FORMAT_FRAME
{
    UCHAR bLength;
    UCHAR bDescriptorType;
    UCHAR bDescriptorSubtype;
    UCHAR bFormatIndex;
    UCHAR bNumFrameDescriptors;
    GUID  guidFormat;  // this field should contain the IR subtype GUID
    UCHAR bBitsPerPixel;
    UCHAR bDefaultFrameIndex;
    UCHAR bAspectRatioX;
    UCHAR bAspectRatioY;
    UCHAR bmInterlaceFlags;
    UCHAR bCopyProtect;
    UCHAR bVariableSize;
} VIDEO_FORMAT_FRAME, *PVIDEO_FORMAT_FRAME;

I am neither an expert in firmware development nor in the usage of Media Foundation, however I would be interested in the raw images on Windows with the standard usbvideo.sys driver.

I got it working with your recipe using pthreads4w and libuvc, but it would be really nice to avoid the installation of the libusbk driver which seems unmaintained since 2014.

[kekiefer]

Maybe @Sheyne has some more insight here. Current versions of FLIR's Lepton sdk are using the usbvideo.sys driver, and I'm pretty sure they have access to raw frame data?

[Sheyne]

@ghellwig there's an SDK at https://lepton.flir.com/software-sdk/ (Lepton Windows SDK) that can do this. There's a .NET library we export that uses usbvideo.sys. If you need pure c++ (not wrapped up in .NET/source code I can see if I can get permission to give you that).

Note that you want the Lepton Windows SDK not the Lepton Embedded SDK.

[ghellwig]

Hi @Sheyne,

thank you for your response! I am aware of the .NET SDK, but as you correctly guessed, I am interested in pure C/C++, although I could of course load the .NET dll in my code. But this would be a bit weird, i.e. wrapping C++ in .NET to wrap it back into C++. If publishing the source code is not an option for your company, I would also be happy with C/C++ SDK, similar to Intel's librealsense. In addition, it should be available for commercial use.

[ghellwig]

Hi again,

@kekiefer, @Sheyne I managed to get raw data using media foundation from the camera, however only the 160x120 frame because only the frame with index 0x01 seems to be recognized. Using usbtreeview I get this output:

        ------- VS Uncompressed Format Type Descriptor --------
bLength                  : 0x1B (27 bytes)
bDescriptorType          : 0x24 (Video Streaming Interface)
bDescriptorSubtype       : 0x04 (Uncompressed Format Type)
bFormatIndex             : 0x02
bNumFrameDescriptors     : 0x01
guidFormat               : {20363159-0000-0010-8000-00AA00389B71} (unknown)
bBitsPerPixel            : 0x10
bDefaultFrameIndex       : 0x01
bAspectRatioX            : 0x00
bAspectRatioY            : 0x00
bmInterlaceFlags         : 0x00
 D0 IL stream or variable: 0 (no)
 D1 Fields per frame     : 0 (2 fields)
 D2 Field 1 first        : 0 (no)
 D3 Reserved             : 0
 D4..5 Field pattern     : 0 (Field 1 only)
 D6..7 Display Mode      : 0 (Bob only)
bCopyProtect             : 0x00 (No restrictions)
*!*ERROR:  Found 2 frame descriptors (should be 1)

        -------- VS Uncompressed Frame Type Descriptor --------
---> This is the Default (optimum) Frame index
bLength                  : 0x1E (30 bytes)
bDescriptorType          : 0x24 (Video Streaming Interface)
bDescriptorSubtype       : 0x05 (Uncompressed Frame Type)
bFrameIndex              : 0x01
bmCapabilities           : 0x02
wWidth                   : 0x00A0 (160)
wHeight                  : 0x0078 (120)
dwMinBitRate             : 0x002A3000 (345.5 KB/s)
dwMaxBitRate             : 0x002A3000 (345.5 KB/s)
dwMaxVideoFrameBufferSize: 0x00009600
dwDefaultFrameInterval   : 0x0010F447 (111 ms -> 9.00 fps)
bFrameIntervalType       : 0x01
adwFrameInterval[1]      : 0x0010F447 (111 ms -> 9.00 fps)

        -------- VS Uncompressed Frame Type Descriptor --------
bLength                  : 0x1E (30 bytes)
bDescriptorType          : 0x24 (Video Streaming Interface)
bDescriptorSubtype       : 0x05 (Uncompressed Frame Type)
bFrameIndex              : 0x02
bmCapabilities           : 0x02
wWidth                   : 0x00A0 (160)
wHeight                  : 0x007A (122)
dwMinBitRate             : 0x002AE400 (351.2 KB/s)
dwMaxBitRate             : 0x002AE400 (351.2 KB/s)
dwMaxVideoFrameBufferSize: 0x00009880
dwDefaultFrameInterval   : 0x0010F447 (111 ms -> 9.00 fps)
bFrameIntervalType       : 0x01
adwFrameInterval[1]      : 0x0010F447 (111 ms -> 9.00 fps)

In particular, the tool complains about two frame descriptors, while in the format descriptor there is only 1 listed:

bNumFrameDescriptors     : 0x01

I believe, this is caused by these lines:

• lepton 3: https://github.com/groupgets/purethermal1-firmware/blob/master/Middlewares/ST/STM32_USB_Device_Library/Class/video/Inc/uvc_desc_vs_l3.h#L52
• lepton 2: https://github.com/groupgets/purethermal1-firmware/blob/master/Middlewares/ST/STM32_USB_Device_Library/Class/video/Inc/uvc_desc_vs_l2.h#L52

I believe this is a bug in the firmware, right? However, I don't know, if this would allow me to stream the frames with telemetry data included.

Best, Gregor

[ghellwig]

@kekiefer if you agree with my analysis: Should I create a separate issue for this?

edit: I created a PR for this: https://github.com/groupgets/purethermal1-firmware/pull/22

[ghellwig]

@Sheyne in addition to streaming from the camera, I also want to use the extension units. However, I fail to understand how this should be done from the MSDN documentation.

It looks, like I have to write an extension unit interface and somehow register the extension unit GUIDs in the windows registry. But, when I try to use an extension control, I get an error, that the property set (should be the extension unit GUID, from what I understand) is not available for my object.

Am I actually going in the right direction? Do you have any hints on what tools to use and maybe some common pitfalls which I should care about?

Thank you in advance, Gregor

[kekiefer]

PR #22 merged and it looks like you were able to access extension units through Windows. That's great. What additional documentation do you think would be helpful to link to (or write up), or is there an example that would show someone else how to do this?

[ghellwig]

@kekiefer I think another directory in your uvc capture repo would be a good idea. However, I have to clean up the code a bit before that. I guess I need a few days until I find the time to do this. A few lines of example code would likely be quicker. Where should I add this?

[kekiefer]

The uvc capture repo is definitely the place for this, and I would really appreciate anything you can come up with here. If this is a big diversion for you, and you would rather do this as a small contract, shoot me an email (kurt at groupgets) and we'll work something out.

[Sheyne]

@ghellwig Did you figure out UVC extension units? Sorry I lost this thread until I saw your PR. Let me know and I can probably send you some code to get you started.

[ghellwig]

@Sheyne Yes, I figured it out. The only issue I had in the end was solved by looking at your PR #13. Thank you!

[sayezz]

@kekiefer @ghellwig I'm quite late to the show. I fight with getting Lepton 3.5 running under windows using C++. I'm special interested in the radiometric data. It looks like you found a solution in C++ but I lost track. If some of you sees this, can you please reach out to me and give me a hint.

PS: I tried to get in running with WinUSB driver... but I think this was a dead end.

[ghellwig]

Hi @sayezz,

you may have a look here: https://github.com/groupgets/purethermal1-uvc-capture/tree/master/mediafoundation/PureThermal

[sayezz]

Thanks! I appreciate! That really helped! I can compile the solution and let it run under Windows 10 with Visual Studio 2017.