using on Windows

[MTavana]

Hi: Can I use this library on python installed on Windows?

[stawel]

will it work on windows: I don't know, I don't own a windows machine,but probably yes Just try it out.

[rajeshmag]

Great work @stawel

I don't know python. Can you please help me solve this error, while executing in Windows:

D:\Users\ramarg\Downloads\ht301_hacklib-master\ht301_hacklib-master>python opencv.py
[ WARN:0] global C:\Users\runneradmin\AppData\Local\Temp\pip-req-build-sn_xpupm\opencv\modules\videoio\src\cap_msmf.cpp (438) `anonymous-namespace'::SourceReaderCB::~SourceReaderCB terminating async callback
Traceback (most recent call last):
  File "D:\Users\ramarg\Downloads\ht301_hacklib-master\ht301_hacklib-master\opencv.py", line 15, in <module>
    ret, frame = cap.read()
  File "D:\Users\ramarg\Downloads\ht301_hacklib-master\ht301_hacklib-master\ht301_hacklib.py", line 287, in read
    ret, frame_raw, frame, meta = self.read_()
  File "D:\Users\ramarg\Downloads\ht301_hacklib-master\ht301_hacklib-master\ht301_hacklib.py", line 278, in read_
    frame = frame.view(dtype=dt).reshape((frame.shape[:2]))
ValueError: cannot reshape array of size 112128 into shape (1,224256)
[ WARN:1] global C:\Users\runneradmin\AppData\Local\Temp\pip-req-build-sn_xpupm\opencv\modules\videoio\src\cap_msmf.c
D:\Users\ramarg\Downloads\ht301_hacklib-master\ht301_hacklib-master>

[stawel]

@rajeshmag interesting, could you run opencv.py from this branch: https://github.com/stawel/ht301_hacklib/tree/test_windows

for me (linux) the output is:

ht301_lib$ python opencv.py
...
cap shape: (292, 384, 2)
uint16 shape: (292, 384, 1)
final shape: (292, 384)
...

I'm interested what the output is in windows.

[ungutknut]

I tested it on Win10 too but wasn't able to get it running. Being a linux guy myself (where I got it running fine) I lack of experience on python under Win; so perhaps I'm doing something wrong. Anyhow here's the opency.py output on Win:

[ WARN:0] global C:\Users\runneradmin\AppData\Local\Temp\pip-req-build-ep71p_ws\opencv\modules\videoio\src\cap_msmf.cpp (438) anonymous-namespace'::SourceReaderCB::~SourceReaderCB terminating async callback
[ WARN:0] global C:\Users\runneradmin\AppData\Local\Temp\pip-req-build-ep71p_ws\opencv\modules\videoio\src\cap_msmf.cpp (438) anonymous-namespace'::SourceReaderCB::~SourceReaderCB terminating async callback

cap shape: (1, 224256)
uint16 shape: (1, 112128)
final shape: (1, 112128)
Traceback (most recent call last):
  File "opencv.py", line 15, in <module>
    ret, frame = cap.read()
  File "C:\Users\xxx\Downloads\ht301_hacklib-test_windows\ht301_hacklib.py", line 292, in read
    device_strings = device_info(meta)
  File "C:\Users\xxx\Downloads\ht301_hacklib-test_windows\ht301_hacklib.py", line 226, in device_info
    meta3 = meta[3]
IndexError: index 3 is out of bounds for axis 0 with size 1
[ WARN:1] global C:\Users\runneradmin\AppData\Local\Temp\pip-req-build-ep71p_ws\opencv\modules\videoio\src\cap_msmf.cpp (438) anonymous-namespace'::SourceReaderCB::~SourceReaderCB terminating async callback 

Matplotlib and opencv installed as expected via pip BTW.

Here's the output when I try to start pyplot.py:

[ WARN:0] global C:\Users\runneradmin\AppData\Local\Temp\pip-req-build-ep71p_ws\opencv\modules\videoio\src\cap_msmf.cpp (438) `anonymous-namespace'::SourceReaderCB::~SourceReaderCB terminating async callback
[ WARN:0] global C:\Users\runneradmin\AppData\Local\Temp\pip-req-build-ep71p_ws\opencv\modules\videoio\src\cap_msmf.cpp (438) `anonymous-namespace'::SourceReaderCB::~SourceReaderCB terminating async callback

cap shape: (1, 224256)
uint16 shape: (1, 112128)
final shape: (1, 112128)
Traceback (most recent call last):
  File "opencv.py", line 15, in <module>
    ret, frame = cap.read()
  File "C:\Users\xxx\Downloads\ht301_hacklib-test_windows\ht301_hacklib.py", line 292, in read
    device_strings = device_info(meta)
  File "C:\Users\xxx\Downloads\ht301_hacklib-test_windows\ht301_hacklib.py", line 226, in device_info
    meta3 = meta[3]
IndexError: index 3 is out of bounds for axis 0 with size 1
[ WARN:1] global C:\Users\runneradmin\AppData\Local\Temp\pip-req-build-ep71p_ws\opencv\modules\videoio\src\cap_msmf.cpp (438) `anonymous-namespace'::SourceReaderCB::~SourceReaderCB terminating async callback

Things I noticed without being sure if they matter or not: Some references to "C:\Users\runneradmin..." which doesnt exist. And the shebang in each script still has the linux path set "#!/usr/bin/python3"

[ungutknut]

A colleague of mine got it running on windows after debugging.

Attached you can find the fixed ht301_hacklib.py: ht301_hacklib.zip

[stawel]

Thank You @ungutknut and your colleague, great work! I've updated the master branch

[canberkdurmus]

Hello,

I am having a problem with Windows too. It works just fine on Linux 20.04 but in Windows, it apparently has a problem with setting cap properties.

Firstly the error I am getting:

When I check the frame shapes In Windows:

Frame shape and content in Ubuntu 20.04:

And this is from the library on Ubuntu 20.04:

It seems like the cap properties are not effective (not changing the shapes, still three channels) since in Linux dtype conversion function (frame = frame.view(dtype=dt)) takes a frame with two channels.

Is there any way to set properties correctly or is there any way to process (slice it with numpy or cv2.cvtColor) the three-channel image into two-channel one after capturing the frame? It seems that Windows OpenCV is still trying to convert it into RGB (or BGR) despite cap.set(cv2.CAP_PROP_CONVERT_RGB, 0)

What is the format that captured in Ubuntu 20 (The uint8 frame shaped 292, 384, 2 that we are going to convert into uint16) is it YUV or something like it?

Thank you for your time

[stawel]

interesting, I guess it has something to do with you opencv version (maybe you need to upgrade it?), this is what I found: https://github.com/opencv/opencv/issues/11398

back to you questions:

Is there any way to set properties correctly or is there any way to process (slice it with numpy or cv2.cvtColor) the three-channel image into two-channel one after capturing the frame?

I don't know, I'm also not sure if a back and forth conversion is a good idea since some of the "temperature lookup table" data is encoded on the image.

What is the format that captured in Ubuntu 20 (The uint8 frame shaped 292, 384, 2 that we are going to convert into uint16) is it YUV or something like it?

my test program:

#!/usr/bin/python3
import numpy as np
import cv2

def decode_fourcc(cc):
    return "".join([chr((int(cc) >> 8 * i) & 0xFF) for i in range(4)])

def info(cap):
    ret, frame = cap.read()
    print(frame.shape, frame.dtype)
    print('  backend: ', cap.getBackendName())
    print('  CAP_PROP_FOURCC:', decode_fourcc(cap.get(cv2.CAP_PROP_FOURCC)))
    print('  CAP_PROP_FORMAT:', cap.get(cv2.CAP_PROP_FORMAT));

cv2.namedWindow("HT301", cv2.WINDOW_NORMAL)
cap = cv2.VideoCapture(2)
info(cap)
print('cap.set(cv2.CAP_PROP_CONVERT_RGB, 0)');
cap.set(cv2.CAP_PROP_CONVERT_RGB, 0)
info(cap)

# Use raw mode
print('cap.set(cv2.CAP_PROP_ZOOM, 0x8004)');
cap.set(cv2.CAP_PROP_ZOOM, 0x8004)
info(cap)

while(True):
    ret, frame = cap.read()
    print('read:', frame.shape, frame.dtype)
    frame = cv2.cvtColor(frame, cv2.COLOR_YUV2BGR_YUYV);
    print('cvtColor:', frame.shape, frame.dtype)

    cv2.imshow('HT301',frame)
    key = cv2.waitKey(1) & 0xFF
    if key == ord('q'):
        break

output:

(292, 384, 3) uint8
  backend:  V4L2
  CAP_PROP_FOURCC: YUYV
  CAP_PROP_FORMAT: 16.0
cap.set(cv2.CAP_PROP_CONVERT_RGB, 0)
(292, 384, 2) uint8
  backend:  V4L2
  CAP_PROP_FOURCC: YUYV
  CAP_PROP_FORMAT: 8.0
cap.set(cv2.CAP_PROP_ZOOM, 0x8004)
(292, 384, 2) uint8
  backend:  V4L2
  CAP_PROP_FOURCC: YUYV
  CAP_PROP_FORMAT: 8.0
read: (292, 384, 2) uint8
cvtColor: (292, 384, 3) uint8
...

it looks like the output format is YUYV, unfortunately I don't see a COLOR_BGR2YUV_YUYV (back) conversion.

side note: if you just set cap.set(cv2.CAP_PROP_CONVERT_RGB, 0) without cap.set(cv2.CAP_PROP_ZOOM, 0x8004) (without entering raw mode - in raw mode each pixel represents a ADC value), the output is a gray temperature image (encoded in YUYV) - I also remember that there was a possibility to change the gray mapping to a different one (unfortunately I don't remember how)

[canberkdurmus]

Thank you so much for your time, I tried to convert it into YUYV after capturing as BGR but it corrupts the metadata stripe on the bottom 4 rows and all the mapping and interpretation process gives numerous errors.

How can I find the mapping of the metadata stripe on the bottom 4 rows? I would like to try to parse it myself.

[JoJoBond]

OpenCV effectively always uses DirectShow as capture API on Windows. On DirectShow OpenCV will request a RGB24 video as per default. The HT301 doesn't have a RGB24 mode via DirectShow, it only has YUY2 (YUYV). DirectShow will still play nicely and will convert the YUY2 to RGB24 before passing it to the caller. You can't convert back to YUY2 accurately because the conversion from YUY2 to RGB24 already involved rounding. The only we to get this to work properly is to request YUY2 from the beginning.

Instead of just self.cap.set(cv2.CAP_PROP_CONVERT_RGB, 0) could you try:

self.cap.set(cv2.CAP_PROP_FOURCC, 0x32595559)
self.cap.set(cv2.CAP_PROP_CONVERT_RGB, 0)

[canberkdurmus]

Hello, unfortunately, this does not work either. DirectShow keeps giving me RGB24 even I call these;

self.cap.set(cv2.CAP_PROP_FOURCC, 0x32595559)
self.cap.set(cv2.CAP_PROP_CONVERT_RGB, 0)

Thank you for your time and help.

[JoJoBond]

There could possibly be an issue with the value type for the set method. It's being converted from fp64 to int64 and from int64 to int32. But then it has to match another int64 value (though the upper 32 bits are all zero so it should work). Maybe you could try other FourCC values: Y422 : 0x32323459 YUYV: 0x56595559 YVYU: 0x55595659

[JoJoBond]

Oh, it seems that you have to set CAP_PROP_FRAME_WIDTH and CAP_PROP_FRAME_HEIGHT as well, before trying to set CAP_PROP_FOURCC. Maybe this would do:

self.cap.set(cv2.CAP_PROP_FRAME_WIDTH, 384)
self.cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 292)
self.cap.set(cv2.CAP_PROP_FOURCC, 0x32595559)
self.cap.set(cv2.CAP_PROP_CONVERT_RGB, 0)

[canberkdurmus]

Unfortunately setting the frame size didn't help either. Thank you

[JoJoBond]

If I understand correctly setting the OPENCV_DSHOW_DEBUG environment variable to 1 would yield debug output on stdout. Sadly the DirectShow capturing in OpenCV is poorly documented. So I have to work through the sources to find things out.

[JoJoBond]

Looking at the code further ( https://github.com/opencv/opencv/blob/4.x/modules/videoio/src/cap_dshow.cpp ) it seems that requesting anything other than RGB24 is pretty buggy in OpenCV. What will almost certainly work is libuvc/pyuvc ( https://github.com/pupil-labs/pyuvc ), but it requires 64bit python environment and replacing the HT301 driver with libusbK.

[canberkdurmus]

I will be giving this a shot when I have time. I will update here if it works, thank you again!

[JoJoBond]

You can use the 'original' project as guideline for the libuvc stuff: https://github.com/mcguire-steve/ht301_ircam/blob/master/src/camera_driver.cpp#L297 just the HT301CameraDriver::OpenCamera() method.

[JoJoBond]

I rebuild my Windows Python environment and did some more testing. It seems that when using CAP_MSMF instead of CAP_DSHOW it works without any further modifications:

self.cap = cv2.VideoCapture(video_dev, cv2.CAP_MSMF)

I'm running it on the Windows 10 21H1 with Python 3.10.2 x64 and OpenCV 4.5.5.62 cp36 abi3 x64.

[canberkdurmus]

Hey, it works with this backend! Thank you so much for the help.

My system also if anyone needs it: Win 10 Pro 21H2 Python 3.10.0 x64 and OpenCV 4.5.5.62