Please upstream modules into mainline kernel

[bk2204]

I recently discovered that my ThinkPad X1 Carbon needs an out-of-tree driver (this one) for my camera. That seems atypical for Intel hardware, which is typically well supported upstream.

I've tried with both Debian sid's 5.18.0-2-amd64 and Ubuntu jammy's 5.15, but the driver is included in neither distro, probably because it isn't upstream. Since this hardware is considered important for most people who use an Alder Lake laptop, could you consider upstreaming this driver (and any dependent pieces) into the mainline kernel relatively soon, even if only in the staging area?

I did try to build this myself with DKMS, but due to #13 (both the missing header and then the modpost error), it's not possible to do so, so there is presently no way to make this hardware work on Debian.

[pinchartl]

That sounds technically doable. The end result should provide images of similar quality as a cheap webcam, which is better than nothing.

I believe there's some effort on a proof-of-concept, using the CPU to process data. I would then like to see how it could be accelerated with the GPU.

[DemiMarie]

That sounds technically doable. The end result should provide images of similar quality as a cheap webcam, which is better than nothing.

Why can the PSYS produce better quality images? Is it because the PSYS has access to data that the CPU does not? Or would achieving equivalent quality be prohibitively expensive?

I believe there's some effort on a proof-of-concept, using the CPU to process data. I would then like to see how it could be accelerated with the GPU.

GPU acceleration would be great, but it should be optional.

[pinchartl]

Because it would be prohibitively expensive to apply the same level of processing using the CPU, or even the GPU.

[DemiMarie]

Would it be possible to use PCI passthrough to assign the ISYS and PSYS to a VM? Are the ISYS and PSYS in an IOMMU group with any other devices, except perhaps each other? Could firmware loading and authentication be done on the host before the device is assigned to the VM? Obviously the VM has no access to the CSE, so it cannot request firmware authentication itself.

[pinchartl]

For the PSYS, perhaps, I don't know. For the ISYS, likely not. You would need to also assign to the VM the I2C controllers for the buses on which the sensors are connected, and there could be other devices on those buses. Even if this could be done, you wouldn't be able in the VM to power the camera sensors (and other ancillary devices such as lens controllers) up and down, as that's handled by ACPI.

[jwrdegoede]

@netllama

After reading through most of the old comments in this issue, I discovered that the problem was that v4l2-relayd.service had failed (at boot?). After restarting it manually, the problem is fixed. I checked the logs, and the service was failing to start with the following output:

Hmm, so the grep: /sys/devices/virtual/video4linux/*/name: No such file or directory error suggests that the v4l2loopback kernel module has not loaded yet.

I guess this might hit on the first boot of a new kernel when rebooting directly after installing a kernel-update and thus before akmod has build the v4l2loopback module. Then the module will be build on boot which I guess may cause this.

Are you still seeing this?

[netllama]

yes, that is exactly the scenario !

On November 22, 2023 4:51:56 PM GMT+01:00, Hans de Goede @.***> wrote:

@netllama

After reading through most of the old comments in this issue, I discovered that the problem was that v4l2-relayd.service had failed (at boot?). After restarting it manually, the problem is fixed. I checked the logs, and the service was failing to start with the following output:

Hmm, so the grep: /sys/devices/virtual/video4linux/*/name: No such file or directory error suggests that the v4l2loopback kernel module has not loaded yet.

I guess this might hit on the first boot of a new kernel when rebooting directly after installing a kernel-update and thus before akmod has build the v4l2loopback module. Then the module will be build on boot which I guess may cause this.

[jwrdegoede]

yes, that is exactly the scenario !

Ok, there is not much which can be done about this. akmod will try to build the modules for the new kernel in the background after installing the new kernel. But if you reboot before akmod has completed the build then the module will be build at boot triggering this.

I believe that dnf / akmod even inhibits normal reboots through the UI while the modules are being build. But if you do "sudo reboot" from a terminal that does not honor the inhibit and then you may hit this.

[SmokinCaterpillar]

Small question / reminder, first post of this issue was on Jun 21, 2022 that is now over a year and a half ago. What has happened in the meantime? Any update from intel?

[arpad-m]

There is ongoing upstreaming efforts for various parts of the webcam driver stack:

• https://lore.kernel.org/linux-media/20240111065531.2418836-1-bingbu.cao@intel.com/
• https://lore.kernel.org/linux-media/20231204123947.5754-1-hdegoede@redhat.com/
• https://lists.libcamera.org/pipermail/libcamera-devel/2024-January/040113.html

jwrdegoede will also have a talk at fosdem about it. See also their recent blog post.

[paulmenzel]

Thank you Red Hat and Linaro, and, therefore, to all organizations buying RHEL licenses and not being parasites by using CentOS or other clones. Unfortunately that’s often banks and insurance companies. I am curious, if Dell or Google ChromeOS participated in this effort.

Anyway, big kudos again to Red Hat (@jwrdegoede) and Linaro.

[pinchartl]

I have good hopes that the ISYS driver will land upstream in a reasonably near future.

The PSYS is a different story. It seems much more likely to me that IPU6-based platforms will be supported by libcamera using a software-based ISP (possibly GPU-accelerated at some point). That will incur some performance and power consumption costs, but should provide decent-enough images for basic video conferencing needs.

[DemiMarie]

I have good hopes that the ISYS driver will land upstream in a reasonably near future.

The PSYS is a different story. It seems much more likely to me that IPU6-based platforms will be supported by libcamera using a software-based ISP (possibly GPU-accelerated at some point). That will incur some performance and power consumption costs, but should provide decent-enough images for basic video conferencing needs.

What is the reason for this? Is it because Linux has no framework for PSYS drivers, or because Intel is not willing to provide a driver or documentation, and therefore extensive reverse engineering would be required?

[pinchartl]

Only Intel can officially answer that question. You may be able to form your personal opinion on the matter by looking at how much information the existing PSYS driver exposes (or doesn't expose) about the parameters the ISP receives, the statistics it produces, and how it operates in general.

[DemiMarie]

Ah, so this is a situation where the actual driver runs in userspace, with the kernel driver just being a pipe between userspace and the on-device firmware.

[paulmenzel]

Slide 7 of the FOSDEM 2024 presentation slides says:

Reluctance/refusal of some vendors to disclose “secret sauce”

Doesn’t that contradict Intel’s pledge to openness in 2021?

[DemiMarie]

I have good hopes that the ISYS driver will land upstream in a reasonably near future.

The PSYS is a different story. It seems much more likely to me that IPU6-based platforms will be supported by libcamera using a software-based ISP (possibly GPU-accelerated at some point). That will incur some performance and power consumption costs, but should provide decent-enough images for basic video conferencing needs.

Does that mean that for those who need high-quality images or good battery life, an external high-end USB webcam will be needed for the forseeable future?

[pinchartl]

For those who need high-quality images and good battery life with upstream drivers, yes. That is my understanding, but I would be very happy if Intel proved me wrong :-)

[acegallagher]

See here for discussion and status on kernel mailing list:

https://lwn.net/ml/linux-media/008f30be-7d1b-498c-88fa-d8cf061e19fa@redhat.com/#t

[oblique]

The v6 of the patch was posted for review two days ago: https://lwn.net/ml/linux-media/20240426095822.946453-1-sakari.ailus@linux.intel.com/

[arpad-m]

Also, v8 of the softisp patch has been posted recently https://lists.libcamera.org/pipermail/libcamera-devel/2024-April/041448.html

the patches I've linked above have made it into Linux 6.8.

[chromer030]

the driver upstreamed in linux 6.10.

https://lore.kernel.org/lkml/20240516080159.76e8b45d@sal.lan/

EDIT: relative commit which contains codes & documentation about how driver works:

https://github.com/torvalds/linux/commit/6fd600d742744dc7ef7fc65ca26daa2b1163158a

[prochac]

the driver upstreamed in linux 6.10.

https://lore.kernel.org/lkml/20240516080159.76e8b45d@sal.lan/

My for-sure-last Intel-based NTB will have a working webcam in 2024, yay.

[pinchartl]

the driver upstreamed in linux 6.10. https://lore.kernel.org/lkml/20240516080159.76e8b45d@sal.lan/

While it's an important part of the puzzle, the ISYS driver alone isn't enough. The PSYS driver is still missing, and that will take more time. In the meantime, libcamera has a software-based ISP implementation (contributed by Redhat and Linaro, kudos to them) that will be usable as a stop-gap measure.

My for-sure-last Intel-based NTB will have a working webcam in 2024, yay.

You will also need a driver for the camera sensor in your device. With some luck, there will already be one in the kernel. It may also need integration in libcamera, if not done yet.

TL;DR: Avoiding the extra CPU and power consumption of the software ISP will have to wait for next year, but the good news is that there's a chance you'll get images out of your camera in 2024 :-)

[DemiMarie]

the driver upstreamed in linux 6.10. https://lore.kernel.org/lkml/20240516080159.76e8b45d@sal.lan/

While it's an important part of the puzzle, the ISYS driver alone isn't enough. The PSYS driver is still missing, and that will take more time.

Does this mean that a PSYS driver is at least planned? I thought it was completely blocked by Intel not being willing to release any userspace for the PSYS and upstream (rightly) refusing to take a driver with no working userspace.

(...)

My for-sure-last Intel-based NTB will have a working webcam in 2024, yay.

You will also need a driver for the camera sensor in your device. With some luck, there will already be one in the kernel. It may also need integration in libcamera, if not done yet.

Does Linux have drivers for the sensors most commonly found in laptops?

[pinchartl]

Does this mean that a PSYS driver is at least planned? I thought it was completely blocked by Intel not being willing to release any userspace for the PSYS and upstream (rightly) refusing to take a driver with no working userspace.

I'm not sure I would say "planned". The effort isn't dead, but the relevant upstream kernel communities have not blessed any architecture and technical solution yet. Whether or not a driver could land upstream, and at what time, are not questions that can be answered with a complete guarantee at this point.

(...)

My for-sure-last Intel-based NTB will have a working webcam in 2024, yay.

You will also need a driver for the camera sensor in your device. With some luck, there will already be one in the kernel. It may also need integration in libcamera, if not done yet.

Does Linux have drivers for the sensors most commonly found in laptops?

Some of them are supported. Intel has upstreamed sensor drivers in the past few of years, so I would say they have done work in that area, but I don't know what shares of the Intel-based laptops that would cover, or what models.

[DemiMarie]

Does this mean that a PSYS driver is at least planned? I thought it was completely blocked by Intel not being willing to release any userspace for the PSYS and upstream (rightly) refusing to take a driver with no working userspace.

I'm not sure I would say "planned". The effort isn't dead, but the relevant upstream kernel communities have not blessed any architecture and technical solution yet. Whether or not a driver could land upstream, and at what time, are not questions that can be answered with a complete guarantee at this point.

I know that drivers for other ISPs have been accepted upstream. Is the lack of an open userspace for the PSYS still the main blocker, or are there other problems that also need to be resolved?

[Vascro]

the driver upstreamed in linux 6.10.

https://lore.kernel.org/lkml/20240516080159.76e8b45d@sal.lan/

EDIT: relative commit which contains codes & documentation about how driver works:

torvalds/linux@6fd600d

Is there any talk of backporting the driver to 6.9? I'm aware that the PSYS driver would still be needed, but with 6.9 being so new, it makes sense to me to try supporting it.

[norru]

I've updated my Debian Sid to latest, which brought in kernel 6.10.6.

There is no binary kernel module for IPU6 (but I can see the ipu6-fw in the firmware package).

Any idea why the driver could be missing from the distro?

[rk9qn3j]

I'm running Fedora 40 with 6.10.6 - same here..

[otetard]

Any idea why the driver could be missing from the distro?

IPU6 was enabled in the 6.11~rc4-1~exp1 release of the Linux kernel (available in experimental). Unfortunately, at the moment, the only available module is ov2740.

More information on the next steps required to have support in Debian available in this bug: #1074441.

[norru]

I'll keep waiting, thanks for the update!