OpenSeeFace fails to utilize the built in webcam

[RossComputerGuy]

When using OpenSeeFace with the built in webcam, it fails to work properly. It works correctly when using an external webcam.

$ python3 facetracker.py -c 0 -W 1280 -H 720 --discard-after 0 --scan-every 0 --no-3d-adapt 1 --max-feature-updates 900
2024-10-24 20:20:13.703091095 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:2158 CreateInferencePybindStateModule] Init provider bridge failed.
[ WARN:0@0.033] global cap_gstreamer.cpp:1173 isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Took 7.39ms
Took 7.14ms
Took 7.45ms
Took 7.41ms
Took 9.74ms
[ WARN:0@11.599] global cap_v4l.cpp:1134 tryIoctl VIDEOIO(V4L2:/dev/video0): select() timeout.
[ WARN:0@21.629] global cap_v4l.cpp:1134 tryIoctl VIDEOIO(V4L2:/dev/video0): select() timeout.
[ WARN:0@31.656] global cap_v4l.cpp:1134 tryIoctl VIDEOIO(V4L2:/dev/video0): select() timeout.
[ WARN:0@31.676] global cap_gstreamer.cpp:1173 isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
[ WARN:0@31.676] global cap_v4l.cpp:887 requestBuffers VIDEOIO(V4L2:/dev/video0): failed VIDIOC_REQBUFS: errno=16 (Device or resource busy)
[ WARN:0@31.676] global cap_v4l.cpp:887 requestBuffers VIDEOIO(V4L2:/dev/video0): failed VIDIOC_REQBUFS: errno=16 (Device or resource busy)
[ WARN:0@31.676] global cap_v4l.cpp:887 requestBuffers VIDEOIO(V4L2:/dev/video0): failed VIDIOC_REQBUFS: errno=16 (Device or resource busy)
[ WARN:0@31.676] global cap_v4l.cpp:887 requestBuffers VIDEOIO(V4L2:/dev/video0): failed VIDIOC_REQBUFS: errno=16 (Device or resource busy)
[ WARN:0@31.676] global cap_v4l.cpp:862 requestBuffers VIDEOIO(V4L2:/dev/video0): Insufficient buffer memory
[ WARN:0@31.676] global cap_v4l.cpp:997 open VIDEOIO(V4L2:/dev/video0): can't open camera by index
[ERROR:0@31.677] global obsensor_uvc_stream_channel.cpp:159 getStreamChannelGroup Camera index out of range
There was no valid input.

Likely due to this because it shows up in dmesg right when OpenSeeFace runs into the problem:

[42989.715679] apple-isp 384000000.isp: Direct firmware load for apple/isp_1921_01XX.dat failed with error -2
[42989.715701] apple-isp 384000000.isp: failed to request setfile 'apple/isp_1921_01XX.dat': -2
[42989.723230] apple-isp 384000000.isp: warning: calibration data not loaded: -2

Flake for shell:

{
  outputs = { self, nixpkgs }: {
    devShells.aarch64-linux.default = let
      pkgs = nixpkgs.legacyPackages.aarch64-linux;
    in pkgs.mkShell {
      packages = [
        (pkgs.python3.withPackages (p: with p; [
          onnxruntime
          (opencv4.override {
            enableGtk3 = true;
          })
          pillow
          numpy
        ]))
      ];
    };
  };
}

[tpwrules]

This is probably an upstream driver problem or configuration issue. I get a couple frames and then it seems to stop and give the same sorts of errors. The firmware missing will result in poor webcam quality but you will still get something. I don't see any such warning (though I am sure I have the firmware present) and my webcam (on 16" M1 Max Macbook) works well in Firefox on e.g. webcamtests.com.

If your install is very old, you might need to update the firmware blobs in the EFI stub to get rid of that complaint? (see the last paragraph here). But I don't expect it to solve the problem and I'm not sure there's a lot I can do.

[tpwrules]

It doesn't get stuck if I tell OpenSeeFace to use 1920x1080 (which is what webcamtests.com detected).

Seems to accurately detect if both eyes are closed too (though not if only one is).

[RossComputerGuy]

If your install is very old, you might need to update the firmware blobs in the EFI stub to get rid of that complaint? (see the last paragraph here). But I don't expect it to solve the problem and I'm not sure there's a lot I can do.

It's not clear to me how to upgrade the firmware blobs from that section.

[RossComputerGuy]

It doesn't get stuck if I tell OpenSeeFace to use 1920x1080 (which is what webcamtests.com detected).

Just tried this and it indeed works, thanks.

[tpwrules]

If your install is very old, you might need to update the firmware blobs in the EFI stub to get rid of that complaint? (see the last paragraph here). But I don't expect it to solve the problem and I'm not sure there's a lot I can do.

It's not clear to me how to upgrade the firmware blobs from that section.

You need to delete the ESP and macOS stub then reinstall them and fix up NixOS as described there. The need isn't super clear (sorry) but the process should be. The latest installer will write the latest firmware files.

There's also some info here: https://github.com/tpwrules/nixos-apple-silicon/blob/main/docs/release-notes.md#2023-10-21 . I haven't been able to get an automatic way together.