Error when trying to use IR sensors on Manjaro (Arch-based) with HP Spectre

[rdivanji]

Just wanted to say first thanks for making this utility. My /dev/video0 device is the main camera and the /dev/video2 is the IR sensors. video1 and 3 are inactive even though v4l2-ctl --list-devices --all lists all 4 (0-3) as "HP Wide Vision FHD Camera: HP W (usb-0000:00:14.0-5)." If I test using the camera, my face is outlined in red and I can add my face just fine, however if I use video2 (IR sensors), testing works (I have to get a bit closer to the sensors for it to outline my face but that's okay) but trying to add my face returns this error and my face is not added.

Traceback (most recent call last): File "/usr/bin/howdy", line 90, in import cli.add File "/usr/lib/security/howdy/cli/add.py", line 110, in enc = face_recognition.face_encodings(frame) File "/usr/lib/python3.7/site-packages/face_recognition/api.py", line 210, in face_encodings return [np.array(face_encoder.compute_face_descriptor(face_image, raw_landmark_set, num_jitters)) for raw_landmark_set in raw_landmarks] File "/usr/lib/python3.7/site-packages/face_recognition/api.py", line 210, in return [np.array(face_encoder.compute_face_descriptor(face_image, raw_landmark_set, num_jitters)) for raw_landmark_set in raw_landmarks] TypeError: compute_face_descriptor(): incompatible function arguments. The following argument types are supported:

1. (self: dlib.face_recognition_model_v1, img: numpy.ndarray[(rows,cols,3),uint8], face: dlib.full_object_detection, num_jitters: int=0) -> dlib.vector
2. (self: dlib.face_recognition_model_v1, img: numpy.ndarray[(rows,cols,3),uint8], faces: dlib.full_object_detections, num_jitters: int=0) -> dlib.vectors
3. (self: dlib.face_recognition_model_v1, batch_img: List[numpy.ndarray[(rows,cols,3),uint8]], batch_faces: List[dlib.full_object_detections], num_jitters: int=0) -> dlib.vectorss Invoked with: <dlib.face_recognition_model_v1 object at 0x7f15f4c09730>, array([[22, 22, 22, ..., 24, 24, 24], [22, 22, 22, ..., 24, 24, 24], [21, 21, 21, ..., 24, 24, 24], ..., [74, 74, 74, ..., 77, 77, 78], [73, 74, 74, ..., 77, 78, 78], [72, 71, 75, ..., 77, 77, 78]], dtype=uint8), <dlib.full_object_detection object at 0x7f15f40c6d88>, 1

Any suggestions? Thanks!

[boltgolt]

That's an error i've never seed before, does it throw it every time you try to add your face? Is your face close enough to the camera?

[rdivanji]

That's an error i've never seed before, does it throw it every time you try to add your face? Is your face close enough to the camera?

Yes it throws that every time I try to add my face using the IR sensors. As far as I know my face is close enough. When I test using the IR sensors, it definitely recognizes that my face is there and it is circled like how it should be.

[boltgolt]

I've found this issue from the underlying library, looks like your capturing empty (null) frames. That's weird, because it seems to be working fine with the test command. Did you set a different camera input for the test command? Did you set the frame_width or frame_height config options?

[rdivanji]

Hey there, sorry for the late response. Laptop shit itself last weekend so I had to wipe and start again and I also had exams earlier, but they are over now :) I tried installing Howdy today and the build fails:

Preparing...

Building howdy...
Cloning into 'howdy'...
==> Making package: howdy 2.3.1-1 (Wed 26 Sep 2018 11:42:32 PM CDT)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving sources...
  -> Downloading v2.3.1.tar.gz...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   122    0   122    0     0     23      0 --:--:--  0:00:05 --:--:--    25
100 19635    0 19635    0     0   1846      0 --:--:--  0:00:10 --:--:--  4829
  -> Downloading pam-python-1.0.6.tar.gz...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   383  100   383    0     0     71      0  0:00:05  0:00:05 --:--:--    84
100 47119  100 47119    0     0   4369      0  0:00:10  0:00:10 --:--:-- 11898
  -> Downloading pam-python-1.0.6-fedora.patch...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1179    0  1179    0     0    219      0 --:--:--  0:00:05 --:--:--   273
  -> Downloading pam-python-1.0.6-gcc8.patch...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   788    0   788    0     0    146      0 --:--:--  0:00:05 --:--:--   167
==> Validating source files with sha256sums...
    v2.3.1.tar.gz ... Passed
    pam-python-1.0.6.tar.gz ... Passed
    pam-python-1.0.6-fedora.patch ... Passed
    pam-python-1.0.6-gcc8.patch ... Passed
==> Extracting sources...
  -> Extracting v2.3.1.tar.gz with bsdtar
  -> Extracting pam-python-1.0.6.tar.gz with bsdtar
==> Starting prepare()...
Cloning into 'dlib_clone'...
patching file src/pam_python.c
Hunk #1 succeeded at 39 (offset 7 lines).
Hunk #2 succeeded at 1526 (offset 29 lines).
patching file src/pam_python.c
Hunk #1 succeeded at 2227 (offset 53 lines).
patching file src/Makefile
Hunk #1 succeeded at 1 with fuzz 1.
==> ERROR: A failure occurred in prepare().
    Aborting...

Any suggestions? I think the error is kind of vague. Thanks again!

[KelleyMcChes]

Hmm... that has literally no information about what went wrong (it's not you, its Arch being vague). Are you installing it with an AUR helper? If you are, try installing it the more traditional way and see if it works

[rdivanji]

Sorry, new to this stuff. I read the readme and there aren't any instructions, so what would be the proper way to install it traditionally? I'm using pamac as my AUR helper.

[KelleyMcChes]

I could have worded it better. Just install it without using the AUR helper (makepkg -si)

[rdivanji]

Okay, got it installed! Thanks :) Okay so I tried it again with the default config (except with the device_id tag set to 2 for my IR sensors) and the circle around my face is black as opposed to red when I use just the camera. Trying to add my face results in the same error as above.

[rdivanji]

Okay so I discovered something today. I opened Cheese, gnomes webcam app, and it started showing an odd image. Turns out that the input that Cheese uses is some "HP Wide Vision HD Camera" which seems to be a combination of my video0 and video2 (webcam and IR) device. But using gstreamer-properties, I can tell that my v4l2 device is just my camera. Hopefully that gives some insight.

[sapjunior]

@boltgolt I think this error is cause by some IR Cameras return "Grayscale" instead of "RGB" image. Mine uses XPS13 9370 and this machine IR camera return grayscale while dlib requires RGB image. May be we have to emply some sanity check for input image dimension before feed into dlib? For example,

ret, frame = (video_capture.read())
if len(frame.shape) == 2:
  frame = cv2.cvtColor(frame,cv2.COLOR_GRAY2BGR)

to test.py add.py and compare.py

[boltgolt]

Sounds good! Did you get the same error as the OP and does that code fix it?

[sapjunior]

Yes, I got same error

[stuartmarsden]

I also had this issue on arch with a HP Spectre 360.

I manually added the lines @sapjunior suggested and it works great now. My IR camera was only just added to the latest kernel so this was the first time I could try.

Great work thanks.

[budgie05]

I can confirm the code from @sapjunior works with my XPS13 9370

Thank you

[boltgolt]

Sorry, reopening this because the fix still has to be implemented in the master branch.

[dmig]

Recognition chain looks like this: face_detector -> pose_predictor -> face_encoder -> comparision with reference model. This error occurred because dlib' face_detector can work with grayscale images, but pose_predictor and face_encoder expect only 24bit color.

BTW using grayscale for face_detector speeds up this stage by ~30%, from 65 to 45ms in my case.

[dmig]

@sapjunior, @Droidpro1 can you test this instead?

#after this line
video_capture = cv2.VideoCapture(config.get("video", "device_path"))
# add this one
video_capture.set(cv2.CAP_PROP_CONVERT_RGB, True)

And here is an example of proper frame reading: https://github.com/opencv/opencv/blob/b39cd06249213220e802bb64260727711d9fc98c/samples/python/video_v4l2.py#L44

[rdivanji]

@sapjunior, @Droidpro1 can you test this instead?

#after this line
video_capture = cv2.VideoCapture(config.get("video", "device_path"))
# add this one
video_capture.set(cv2.CAP_PROP_CONVERT_RGB, True)

And here is an example of proper frame reading: https://github.com/opencv/opencv/blob/b39cd06249213220e802bb64260727711d9fc98c/samples/python/video_v4l2.py#L44

Where do I add this? test.py? I added that after this line video_capture = cv2.VideoCapture(int(config.get("video", "device_id")))

And I don't think I'm noticing anything different.

[dmig]

@Droidpro1 no, to the add.py or compare.py. test.py isn't affected.

Where do I add this? test.py? I added that after this line video_capture = cv2.VideoCapture(int(config.get("video", "device_id")))

That's correct.

According to its source, opencv should do this by default, but in your case it doesn't for some reason.