Anyway to use libcamera instead of v4l?

[k4ever]

I'm running Arch Linux on a Microsoft Surface Pro 4. The cameras on the Surface Pro 4 use libcamera instead of v4l. However, I was able to get Howdy to work with the Surface Pro 4 by creating a loopback device to /dev/video42, and setting this as the Howdy video device. The loopback was created using the Gstreamer Loopback Device for other Applications instructions on this website: https://github.com/linux-surface/linux-surface/wiki/Camera-Support

The problem is that this command needs to be ran to initial the camera prior to using Howdy:

gst-launch-1.0 libcamerasrc camera-name='\SB.PCI0.I2C2.CAMF' ! \ video/x-raw,width=1280,height=720,framerate=30/1,format=NV12 \ ! videoconvert ! video/x-raw,format=YUY2 ! queue ! \ v4l2sink device=/dev/video42

Is there a way to get Howdy to also use libcamera?

[boltgolt]

Howdy does not currenty support that, but has a concept called "recorders". Every recorder is responsible for getting the video feed from the configured video device and delivering it to OpenCV. If you're interested i'm sure it's possible to write a recorder for your usecase and add it to v3.0.0

[afalout]

How does the "recorder" concept work - does it depend on capturing video stream into a file and then processing it? Could you please point me to the documentation/howto for it? MS Surface Pro tablets will soon get a support for IR sensor so being able to use Howdy on Surface would be a dream come true... Thanks!

[Svampebob1]

Hi, I'm also facing this issue. Did you solve this?

I'm looking into how I can integrate this into /etc/pam.d/common-auth, but are struggling a bit :-)

[afalout]

Dan Scally on Surface Linux camera project was working on libcamera support for Howdy, but that was maybe 6 months ago and then there was silence... Needless to say, we wont have Howdy on any recent built-in cameras until there is libcamera support in Howdy. And/or gstreamer or pipewire, once they have libcamera support. OK so Im not being very exact but something like that. Lower level lib/HW support > Higher level lib abstraction layer > Applications. Until this is a reality, we need libcamera support in Howdy. Sigh.

[Svampebob1]

Dan Scally on Surface Linux camera project was working on libcamera support for Howdy, but that was maybe 6 months ago and then there was silence... Needless to say, we wont have Howdy on any recent built-in cameras until there is libcamera support in Howdy. And/or gstreamer or pipewire, once they have libcamera support. OK so Im not being very exact but something like that. Lower level lib/HW support > Higher level lib abstraction layer > Applications. Until this is a reality, we need libcamera support in Howdy. Sigh.

I might have a workaround for this. I have made a systemd service for this command: gst-launch-1.0 libcamerasrc camera-name='\SB.PCI0.I2C2.CAMF' ! video/x-raw,width=1280,height=720,framerate=30/1,format=NV12 ! videoconvert ! video/x-raw,format=YUY2 ! queue ! v4l2sink device=/dev/video42

I just need to find a way to start the service through pam, before triggering Howdy. The plan is to stop the service again after a successful login, so that the webcam isn't on constantly.

[djrscally]

Dan Scally on Surface Linux camera project was working on libcamera support for Howdy, but that was maybe 6 months ago and then there was silence...

Yeah sorry; I haven't had much spare time lately (my wife had our third child 6 and a half months ago; from one kid to two was a doddle but it turns out that going from two to three really kills your time). I am actively working on both sides of this problem though; extending libcamera to handle the IR cameras on IPU3 devices and adding a libcamera recorder to Howdy. I recognise you've been waiting a long time @afalout - I'll try to get everything working even if not upstreamable and share it.

I might have a workaround for this. I have made a systemd service for this command: gst-launch-1.0 libcamerasrc camera-name='\SB.PCI0.I2C2.CAMF' ! video/x-raw,width=1280,height=720,framerate=30/1,format=NV12 ! videoconvert ! video/x-raw,format=YUY2 ! queue ! v4l2sink device=/dev/video42

I just need to find a way to start the service through pam, before triggering Howdy. The plan is to stop the service again after a successful login, so that the webcam isn't on constantly.

This kind of thing should work in the interim as long as you don't mind not using the IR camera.

[Svampebob1]

Yeah sorry; I haven't had much spare time lately (my wife had our third child 6 and a half months ago; from one kid to two was a doddle but it turns out that going from two to three really kills your time). I am actively working on both sides of this problem though; extending libcamera to handle the IR cameras on IPU3 devices and adding a libcamera recorder to Howdy. I recognise you've been waiting a long time @afalout - I'll try to get everything working even if not upstreamable and share it.

I totally understand this. No worries. Take care of your kids and wife :-).

This kind of thing should work in the interim as long as you don't mind not using the IR camera.

Thanks. I'll share the results later.

[Svampebob1]

I've figured it out, but this is quick and dirty! Do this on your own risk! It might be smart to have root access while setting this up and testing it, so that you can disable Howdy if it fails.

Feel free to ask if you have any questions.

1. Set up and make sure that libcamera with v4l2loopback works according to the Wiki from this tutorial: https://github.com/linux-surface/linux-surface/wiki/Camera-Support#gstreamer-loopback-device-for-other-applications Certainly! Here’s the revised document with the markdown issues fixed:
2. Set up loopback to video device:

   sudo vi /etc/systemd/system/libcamera_loopback.service

3. Paste the following:

   [Unit]
   Description=libcamera loopback
   
   [Service]
   Type=simple
   User=
   #Group=
   ExecStart=/usr/bin/gst-launch-1.0 libcamerasrc camera-name='\\\_SB_.PCI0.I2C2.CAMF' ! video/x-raw,width=1280,height=720,framerate=30/1,format=NV12 ! videoconvert ! video/x-raw,format=YUY2 ! v4l2sink device=/dev/video42
   Restart=on-failure
   
   [Install]
   WantedBy=multi-user.target

4. Create a policy kit exception so that it can be run by all users in the system:

   1. sudo vi /etc/polkit-1/rules.d/libcamera.rules

   2. Paste the following:

      polkit.addRule(function(action, subject) {
        if (action.id == "org.freedesktop.systemd1.manage-units" &&
            action.lookup("unit") == "libcamera_loopback.service" &&
            (action.lookup("verb") == "start" || action.lookup("verb") == "stop" || action.lookup("verb") == "restart")) {
            return polkit.Result.YES;
        }
      });

5. Test that the service works and that the webcam light turns on and off when the service starts and stops:

   sudo systemctl daemon-reload
   sudo systemctl restart polkit
   sudo systemctl start libcamera_loopback.service
   sudo systemctl status libcamera_loopback.service
   sudo systemctl stop libcamera_loopback.service

6. Install Howdy according to the instructions.
7. Create scripts for starting and stopping:

   1. sudo vi /bin/start-libcamera-loopback.sh

   2. Paste the following, close and save:

      #!/bin/bash
      systemctl start libcamera_loopback.service

   3. sudo vi /bin/stop-libcamera-loopback.sh

   4. Paste the following, close and save:

      #!/bin/bash
      systemctl stop libcamera_loopback.service

   5. sudo chmod +x /bin/*-libcamera-loopback.sh

8. Set up PAM with the following:
   1. Place this at the top:

      auth    required    pam_exec.so seteuid /bin/start-libcamera-loopback.sh

   2. Place this at the bottom:

      auth    optional     pam_exec.so seteuid /bin/stop-libcamera-loopback.sh

[djrscally]

Sweet - thanks for that. For what it's worth I tried to hack the work I'd done so far into working and I think it should...but I haven't had time to test it on my surface yet and likely won't now for at least another week (I'll be away at a conference all next week and probably busy prepping until then). The actual image capture through libcamera works fine, the bit I've been stuck on is convincing opencv to actually convert the image buffers that you receive. To work around that I've just set it to slice out the Y plane from the NV12 buffers that the IPU3 produces and treat it as a gray buffer...it ought to work but as I say I haven't actually managed to test it yet.

If you wanted to test this before then (bearing in mind that I haven't...) you'd need to have built the python bindings in libcamera (meson configure -D pycamera=enabled, requires libpython3-dev pybind11-dev) and in config.ini set recording_plugin=libcamera and libcamera_camera_id="your camera id here". The camera IDs you can find using cam -l:

$ cam -l
[2:51:00.296914792] [21301]  INFO Camera camera_manager.cpp:284 libcamera v0.2.0+139-66f0c2fe-dirty (2024-05-09T10:11:19+01:00)
Available cameras:
1: 'USB  Live camera: USB  Live cam' (\_SB_.PCI0.GP17.XHC0.RHUB.PRT3-3:1.0-0c45:6366)
2: Internal front camera (\_SB_.PCI0.GP17.XHC1.RHUB.PRT3-3:1.0-13d3:5458)

Those are USB cameras on my PC...the IDs for the Surface cameras aren't quite so horrific. I'll get the conversion working once I'm back and then at least the non-IR camera would be available.

[afalout]

Much appreciated, Dan. I will wait for the IR camera as plain photo one is not usable for any form of authentication in real world. Please keep us updated, your work is very important for a lot of people. Cheers

[afalout]

Those are USB cameras on my PC...the IDs for the Surface cameras aren't quite so horrific. I'll get the conversion working once I'm back and then at least the non-IR camera would be available.

Hi Dan,

just a quick check if you had time to move this forward?

Thanks Andrej

[djrscally]

Those are USB cameras on my PC...the IDs for the Surface cameras aren't quite so horrific. I'll get the conversion working once I'm back and then at least the non-IR camera would be available.

Hi Dan,

just a quick check if you had time to move this forward?

Thanks Andrej

Yes, some progress - I can capture from the IR camera with the unpacking into a normal greyscale format done by libcamera. The other problem is that since we can't drive the IR LEDs the image is too dark; I have libcamera mostly correcting that now but it's not quite right. Once that's working I'll share it so you can try it out - it still won't be in a finished state until I have time to do a bunch of cleanup, but it should work at that point.

Thanks Dan