Please upstream modules into mainline kernel

[bk2204]

I recently discovered that my ThinkPad X1 Carbon needs an out-of-tree driver (this one) for my camera. That seems atypical for Intel hardware, which is typically well supported upstream.

I've tried with both Debian sid's 5.18.0-2-amd64 and Ubuntu jammy's 5.15, but the driver is included in neither distro, probably because it isn't upstream. Since this hardware is considered important for most people who use an Alder Lake laptop, could you consider upstreaming this driver (and any dependent pieces) into the mainline kernel relatively soon, even if only in the staging area?

I did try to build this myself with DKMS, but due to #13 (both the missing header and then the modpost error), it's not possible to do so, so there is presently no way to make this hardware work on Debian.

[mkobel]

Lenovo X1 10Gen / Linux 6.2.9-arch1-1 / OV2740:

• DKMS install of https://github.com/intel/ipu6-drivers
• deployed IPU6ep of https://github.com/intel/ipu6-camera-bins
• built https://github.com/intel/ipu6-camera-hal with
  • IPU6_VER=ipu6ep
  • ffmpeg -i ~/example.JPG -pix_fmt nv12 privacy_image_ov2740-uf_1280_720.yuv
• build & installed https://github.com/intel/icamerasrc/tree/icamerasrc_slim_api

The gst-launch example is not working:

$ sudo -E gst-launch-1.0 icamerasrc buffer-count=7 ! video/x-raw,format=NV12,width=1280,height=720 ! videoconvert ! ximagesink

Setting pipeline to PAUSED ...
[04-02 22:36:26.104] CamHAL[INF] aiqb file name ov13b10.aiqb
[04-02 22:36:26.104] CamHAL[ERR] there is no aiqb file:ov13b10
[04-02 22:36:26.104] CamHAL[INF] aiqb file name ov13b10.aiqb
[04-02 22:36:26.104] CamHAL[ERR] there is no aiqb file:ov13b10
[04-02 22:36:26.104] CamHAL[INF] aiqb file name ov8856.aiqb
[04-02 22:36:26.104] CamHAL[ERR] there is no aiqb file:ov8856
[04-02 22:36:26.104] CamHAL[INF] aiqb file name ov8856.aiqb
[04-02 22:36:26.104] CamHAL[ERR] there is no aiqb file:ov8856
[04-02 22:36:26.104] CamHAL[INF] aiqb file name ov01a10.aiqb
[04-02 22:36:26.104] CamHAL[INF] aiqb file name ov01a10.aiqb
[04-02 22:36:26.104] CamHAL[INF] aiqb file name ov01a10.aiqb
[04-02 22:36:26.104] CamHAL[INF] aiqb file name ov01a10.aiqb
[04-02 22:36:26.104] CamHAL[INF] aiqb file name OV02C10_1BG203N3_ADL.aiqb
[04-02 22:36:26.104] CamHAL[INF] aiqb file name OV02C10_1BG203N3_ADL.aiqb
[04-02 22:36:26.104] CamHAL[INF] aiqb file name OV02C10_1SG204N3_ADL.aiqb
[04-02 22:36:26.104] CamHAL[INF] aiqb file name OV02C10_1SG204N3_ADL.aiqb
[04-02 22:36:26.104] CamHAL[INF] aiqb file name OV02C10_CIFME14_ADL.aiqb
[04-02 22:36:26.104] CamHAL[INF] aiqb file name OV02C10_CIFME14_ADL.aiqb
[04-02 22:36:26.104] CamHAL[INF] aiqb file name OV02C10_1BG203N3_ADL.aiqb
[04-02 22:36:26.105] CamHAL[INF] aiqb file name OV02C10_1BG203N3_ADL.aiqb
[04-02 22:36:26.105] CamHAL[INF] aiqb file name OV02C10_1SG204N3_ADL.aiqb
[04-02 22:36:26.105] CamHAL[INF] aiqb file name OV02C10_1SG204N3_ADL.aiqb
[04-02 22:36:26.105] CamHAL[INF] aiqb file name OV02C10_CIFME14_ADL.aiqb
[04-02 22:36:26.105] CamHAL[INF] aiqb file name OV02C10_CIFME14_ADL.aiqb
[04-02 22:36:26.105] CamHAL[INF] aiqb file name OV2740_CJFLE23_ADL.aiqb
[04-02 22:36:26.105] CamHAL[INF] aiqb file name OV2740_CJFLE23_ADL.aiqb
[04-02 22:36:26.105] CamHAL[INF] aiqb file name HM2170_1SG205N3_ADL.aiqb
[04-02 22:36:26.105] CamHAL[INF] aiqb file name HM2170_1SG205N3_ADL.aiqb
[04-02 22:36:26.105] CamHAL[INF] aiqb file name HM2170_CJFME18_ADL.aiqb
[04-02 22:36:26.105] CamHAL[INF] aiqb file name HM2170_CJFME18_ADL.aiqb
[04-02 22:36:26.105] CamHAL[INF] aiqb file name HI556_1BG502T3_ADL.aiqb
[04-02 22:36:26.105] CamHAL[INF] aiqb file name HI556_1BG502T3_ADL.aiqb
[04-02 22:36:26.105] CamHAL[INF] aiqb file name HI556_CJFLE25_ADL.aiqb
[04-02 22:36:26.105] CamHAL[INF] aiqb file name HI556_CJFLE25_ADL.aiqb
[04-02 22:36:26.105] CamHAL[INF] aiqb file name ov01a1s.aiqb
[04-02 22:36:26.106] CamHAL[INF] aiqb file name ov01a1s.aiqb
[04-02 22:36:26.106] CamHAL[ERR] Failed to find DevName for cameraId: 0, get video node: ov13b10 , devname: /dev/v4l-subdev1
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[04-02 22:36:26.112] CamHAL[ERR] Get entity fail for calling getEntityById
[04-02 22:36:26.112] CamHAL[ERR] Get entity fail for calling getEntityById
[04-02 22:36:26.112] CamHAL[ERR] setup Link ov13b10  [-1:0] ==> Intel IPU6 CSI-2  [-1x0] enable 1 failed.
[04-02 22:36:26.112] CamHAL[ERR] set MediaCtlConf McLink failed: ret = -1
[04-02 22:36:26.112] CamHAL[ERR] set up mediaCtl failed
[04-02 22:36:26.112] CamHAL[ERR] @configure Device Configure failed
[04-02 22:36:26.112] CamHAL[ERR] failed to config streams.
ERROR: from element /GstPipeline:pipeline0/Gstcamerasrc:camerasrc0: src pad: Internal data flow error.
Additional debug info:
gstcambasesrc.cpp(3143): gst_cam_base_src_loop (): /GstPipeline:pipeline0/Gstcamerasrc:camerasrc0:
streaming task paused, reason not-negotiated (-4)
Execution ended after 0:00:00.003394833
Setting pipeline to NULL ...
Freeing pipeline ...

[mrhpearson]

@mrhpearson thank you for quick response. Intel is a big company, I am working for RealSense team and we have d457 mipi camera fully integrated with ipu6 working like charm as regular webcam. We have hard time with ipu6 team - working on "impossible" features with great success.

Hi @dmipx - can we discuss offline? Ping me on mpearson-lenovo at squebb dot ca

[SmokinCaterpillar]

@SmokinCaterpillar @karolszk can someone of you attach text file with following command:

media-ctl --print-dot

I can try to estimate work to be done.

I am on 5.19.0-38-genericwith Ubuntu 22.04, a ThinkPad X1 Gen10, and sensor ov2740.

$ media-ctl --print-dot

gives

digraph board {
    rankdir=TB
    n00000001 [label="{{<port0> 0} | Intel IPU6 CSI-2 0\n/dev/v4l-subdev0 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
    n00000001:port1 -> n00000019:port0 [style=dashed]
    n00000004 [label="{{<port0> 0} | Intel IPU6 CSI-2 1\n/dev/v4l-subdev1 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
    n00000004:port1 -> n00000019:port0
    n00000007 [label="{{<port0> 0} | Intel IPU6 CSI-2 2\n/dev/v4l-subdev2 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
    n00000007:port1 -> n00000019:port0 [style=dashed]
    n0000000a [label="{{<port0> 0} | Intel IPU6 CSI-2 3\n/dev/v4l-subdev3 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
    n0000000a:port1 -> n00000019:port0 [style=dashed]
    n0000000d [label="{{<port0> 0} | Intel IPU6 CSI-2 4\n/dev/v4l-subdev4 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
    n0000000d:port1 -> n00000019:port0 [style=dashed]
    n00000010 [label="{{<port0> 0} | Intel IPU6 CSI-2 5\n/dev/v4l-subdev5 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
    n00000010:port1 -> n00000019:port0 [style=dashed]
    n00000013 [label="{{<port0> 0} | Intel IPU6 CSI-2 6\n/dev/v4l-subdev6 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
    n00000013:port1 -> n00000019:port0 [style=dashed]
    n00000016 [label="{{<port0> 0} | Intel IPU6 CSI-2 7\n/dev/v4l-subdev7 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
    n00000016:port1 -> n00000019:port0 [style=dashed]
    n00000019 [label="{{<port0> 0} | Intel IPU6 CSI2 BE SOC 0\n/dev/v4l-subdev8 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
    n00000019:port1 -> n0000001c
    n0000001c [label="Intel IPU6 BE SOC capture 0\n/dev/video0", shape=box, style=filled, fillcolor=yellow]
    n00000032 [label="{{} | ov2740 18-0036\n/dev/v4l-subdev9 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
    n00000032:port0 -> n00000004:port0
}

[pinchartl]

@dmipx Do I understand correctly that the RealSense camera on Alder Lake uses the input system of the IPU6, but not the processing system ?

[naps62]

@SmokinCaterpillar @karolszk can someone of you attach text file with following command:

media-ctl --print-dot

I can try to estimate work to be done.

Dell XPS 13 Plus 2022 with Ubuntu

❯ uname -a
Linux arrakis 5.19.0-38-generic #39~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Mar 17 21:16:15 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

❯ media-ctl --print-dot
digraph board {
        rankdir=TB
        n00000001 [label="{{<port0> 0} | Intel IPU6 CSI-2 0\n/dev/v4l-subdev0 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
        n00000001:port1 -> n00000019:port0 [style=dashed]
        n00000004 [label="{{<port0> 0} | Intel IPU6 CSI-2 1\n/dev/v4l-subdev1 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
        n00000004:port1 -> n00000019:port0 [style=dashed]
        n00000007 [label="{{<port0> 0} | Intel IPU6 CSI-2 2\n/dev/v4l-subdev2 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
        n00000007:port1 -> n00000019:port0 [style=dashed]
        n0000000a [label="{{<port0> 0} | Intel IPU6 CSI-2 3\n/dev/v4l-subdev3 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
        n0000000a:port1 -> n00000019:port0 [style=dashed]
        n0000000d [label="{{<port0> 0} | Intel IPU6 CSI-2 4\n/dev/v4l-subdev4 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
        n0000000d:port1 -> n00000019:port0 [style=dashed]
        n00000010 [label="{{<port0> 0} | Intel IPU6 CSI-2 5\n/dev/v4l-subdev5 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
        n00000010:port1 -> n00000019:port0 [style=dashed]
        n00000013 [label="{{<port0> 0} | Intel IPU6 CSI-2 6\n/dev/v4l-subdev6 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
        n00000013:port1 -> n00000019:port0 [style=dashed]
        n00000016 [label="{{<port0> 0} | Intel IPU6 CSI-2 7\n/dev/v4l-subdev7 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
        n00000016:port1 -> n00000019:port0 [style=dashed]
        n00000019 [label="{{<port0> 0} | Intel IPU6 CSI2 BE SOC 0\n/dev/v4l-subdev8 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
        n00000019:port1 -> n0000001c [style=dashed]
        n0000001c [label="Intel IPU6 BE SOC capture 0\n/dev/video1", shape=box, style=filled, fillcolor=yellow]
        n00000044 [label="{{} | ov01a10 17-0036\n/dev/v4l-subdev9 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
        n00000044:port0 -> n00000007:port0 [style=dashed]
}

[dmipx]

ov01a10

This looks like have binding to ipu6 and I think it is also working with gstreamer plug-ins although the media links are not active.

ov2740

Has active links on ThinkPad.

I see that everyone have some outdated driver as the latest one have much more complicated binding scheme. Thanks for your input, I can start to investigate. You can use online tool to visualize media-ctl dot output http://magjac.com/graphviz-visual-editor/

[s-ailus]

We're working on upstreaming the IPU6 ISYS driver, I'd think it may be merged during the summer this year (this is tentative as the needed APIs aren't fully enabled in the kernel yet). Work is ongoing on IVSC, too, but I don't have an estimate for that yet.

[SmokinCaterpillar]

Hey, were there any important updates in the meantime? Thanks :-)

[s-ailus]

Well, now that you ask, the ISYS driver RFC set has landed to linux-media:

https://lore.kernel.org/linux-media/20230413100429.919622-1-bingbu.cao@intel.com/T/#t

See the cover letter for details.

[norru]

Just got an X1 Carbon Gen 11 which I believe has an IPU6 camera. I was fully expecting the machine to have rough edges on Linux. Running vanilla Debian Sid with kernel 6.1.0-9-amd64, camera is not detected.

A working integrated webcam is not a priority for me (I can attach an external one) but I'd like to get it working, eventually.

Happy to help with simple testing/validation.

[paulmenzel]

@norru, Is the X1 Carbon Gen 11 advertised with GNU/Linux support?

The camera won’t work without hassle for several more months. @norru, I recommend to return the device so Lenovo won’t make any money without offering good GNU/Linux support.

[norru]

@norru, Is the X1 Carbon Gen 11 advertised with GNU/Linux support?

I see where you are coming from.

It might have, but you know what, I don't trust anything that any vendor says about "Linux support". I'm somehow prepared to forgive some minor teething issues when a new device is released.

Said that, eventually, any item I buy from Lenovo could be my last, if user experience is bad in the long run.

I've done it with Apple before. I was very happy with their devices until I wasn't and suddenly I was not an Apple customer anymore.

[drogenlied]

@norru Lenovo sold me a Thinkpad X1 Yoga Gen7 with predeployed Ubuntu.

The Ubuntu autoinstaller shit itself on the first start and after repairing it I found out that the camera was unusable.

So be careful when asking "official support", it means literally nothing.

[norru]

So be careful when asking "official support", it means literally nothing.

Yes, I agree, that's what I meant.

All in all I need my Linux devices to work. If I can't trust a hardware vendor to satisfy that need, I'll just look for another vendor which I can trust.

It sure doesn't do them any good in the long term if they claim they "support Linux", but then users find out that's a lie, even if they may sell a few more pieces of hardware in the immediate term.

Trust erodes.

I have enough patience to tolerate some teething problems when a brand new product is released but patience doesn't come in endless supply.

[mrhpearson]

Hi

The MIPI camera has been a nightmare; but please note that specific config has not been certified with Linux and we won't sell the Ubuntu or Fedora preload with it. We can't support it - Intel don't have upstream drivers for it yet.

It really sucks having a configuration which isn't certified - not something any of us wanted - but Intel do not have Linux support ready yet and realistically we can't just block Windows because of lacking Linux support (though I wish we could sometimes!). We've been working with Intel to get it fixed - it's very slow going but there is progress. I have some concerns about the final solution and how well it will play in an open-source environment - but it will be possible to get it working if you're willing to accepts closed source binary blobs.

Our aim with the Linux program is to support exactly the same HW and FW as Windows gets - I don't want special Linux configs. I don't think it's good for the Linux ecosystem to have limited HW compatibility, but it does make delivering and supporting Linux harder. MIPI has broken the rule for us and it's disappointing.

@drogenlied - you should not have been able to buy the X1 Yoga G7 with a MIPI camera and Ubuntu. If you're willing to send me details on that order (I think I'd just need the serial number if you have it - send to markpearson at lenovo dot com) then I can find out what happened.

I'll actually second what paulmenzel above says - return the system and get one without MIPI (called 'computer vision' in the specs). MIPI support will come eventually but it's going to be a while.

Mark

[pinchartl]

but it will be possible to get it working if you're willing to accepts closed source binary blobs.

Are you talking about camera support or other devices (or both) ?

[paulmenzel]

@mrhpearson, thank you for taking the time to comment with a useful answer and status update. That’s very much appreciated.

We've been working with Intel to get it fixed - it's very slow going but there is progress. I have some concerns about the final solution and how well it will play in an open-source environment - but it will be possible to get it working if you're willing to accepts closed source binary blobs.

Has the proposed solution been discussed in public, so I can read up on it? I must have missed it. Closed source binary blobs would be a great downside.

[mrhpearson]

@pinchartl - Camera support.

@paulmenzel : There's some details in the Intel repo's: https://github.com/intel/ipu6-drivers but it's a bit limited on the nitty gritty details.

Just found out, there will be a blog from one of the distro's soon with a lot more details on the challenges (and steps needed) to enable it - but it's not public yet. If you can hold on a bit longer I think it will be useful for this topic (and a bit longer should be in the days/weeks range not months).

Mark

[norru]

Hi Mark,

Happy, long-term, Lenovo + Linux user here.

Thanks for the time you've spent addressing the community's concerns. I personally highly appreciate it and believe I'm not alone in thinking this.

When I ordered my hardware I wasn't expecting it to work on day one out of the box. I just double checked it wasn't advertised as "Linux supported" but I was aware about ongoing efforts, so I decided to complete the purchase anyway trusting Lenovo's long term commitment for the platform.

I understand some hardware is harder to work with but I'm happy to wait for a full solution, as long as the effort is continuing.

I'd be willing to accept a solution which includes binary blobs as long as it doesn't require self-signing Linux kernel images. In my case I'd like to keep using the official Debian binaries.

Hope it helps, feel free to follow up with questions :)

On Tue, 23 May 2023 at 14:02, Mark Pearson @.***> wrote:

Hi

The MIPI camera has been a nightmare; but please note that specific config has not been certified with Linux and we won't sell the Ubuntu or Fedora preload with it. We can't support it - Intel don't have upstream drivers for it yet.

It really sucks having a configuration which isn't certified - not something any of us wanted - but Intel do not have Linux support ready yet and realistically we can't just block Windows because of lacking Linux support (though I wish we could sometimes!). We've been working with Intel to get it fixed - it's very slow going but there is progress. I have some concerns about the final solution and how well it will play in an open-source environment - but it will be possible to get it working if you're willing to accepts closed source binary blobs.

Our aim with the Linux program is to support exactly the same HW and FW as Windows gets - I don't want special Linux configs. I don't think it's good for the Linux ecosystem to have limited HW compatibility, but it does make delivering and supporting Linux harder. MIPI has broken the rule for us and it's disappointing.

@drogenlied https://github.com/drogenlied - you should not have been able to buy the X1 Yoga G7 with a MIPI camera and Ubuntu. If you're willing to send me details on that order (I think I'd just need the serial number if you have it - send to markpearson at lenovo dot com) then I can find out what happened.

I'll actually second what paulmenzel above says - return the system and get one without MIPI (called 'computer vision' in the specs). MIPI support will come eventually but it's going to be a while.

Mark

— Reply to this email directly, view it on GitHub https://github.com/intel/ipu6-drivers/issues/22#issuecomment-1559286964, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGZDS4V6E46ZK5ZHMIRXITXHSYPPANCNFSM5ZN5CZEQ . You are receiving this because you were mentioned.Message ID: @.***>

-- Cheers, Nico

[drogenlied]

@mrhpearson I suspect I know why I was able to buy this system as I bought the laptop before the whole camera issue was on LKML and in the press in August 2022, so a return would be a bit late now and there is no 4k OLED screen without that camera anyway.

By November, Intel's horrible contraption started being just about barely useable, mainly thanks to @jwrdegoede fixing use before initialisation in several components.

Nonwithstanding, it was falsely advertised at the time, the Ubuntu preload was more work than a fresh install and Lenovo removed my customer review about the great experience.

Still, I am interested in your take on the process and will send you my serial.

[paulmenzel]

Just found out, there will be a blog from one of the distro's soon with a lot more details on the challenges (and steps needed) to enable it - but it's not public yet. If you can hold on a bit longer I think it will be useful for this topic (and a bit longer should be in the days/weeks range not months).

I guess, you mean @jwrdegoede’s article Fedora IPU6 camera support now available in rpmfusion-nonfree published today.

[mrhpearson]

Yes - you beat me to posting it :)

This works on the X1 Carbon 10, Yoga7 and Nano G2. It won't work on the Carbon 11 or Yoga 8 - we're still figuring out what is needed there (though Intel have gotten those camera's working - we haven't reproduced it in our lab yet).

If you're wanting to use the camera this is the best solution I've seen and kudos to the RH folk for what they've done - it's amazing. I'm sure other's will follow based on this work.

From my perspective this is definitely not the final solution - it's a workaround to unblock users in the meantime. I'm looking forward to the Intel drivers landing fully upstream and then being able to get the user space pieces working.

[norbusan]

I would happily work on packaging this for Arch, but the repos are somehow a mess with too many branches and no clear indication which branch should be used. Would it be possible to summarize which sources and additional patches have been used for the Fedora packaging?

[naps62]

@norbusan I would happily submit my Dell XPS 13 as tribute (OV01A1B sensor, as far as I know, currently unlisted in that blog post) to move to Arch again and help with testing

update: wrong sensor I think. other below have pointed out the right one

[simisimis]

Dell XPS Plus(9320) with a ov01a10 sensor is also unlisted :/

[jwrdegoede]

I would happily work on packaging this for Arch, but the repos are somehow a mess with too many branches and no clear indication which branch should be used. Would it be possible to summarize which sources and additional patches have been used for the Fedora packaging?

First of all your users need to be on kernel >= 6.3 and you need to disable the kernels own ov2740 sensor driver. It is ok to disable this on x86_64 builds since I don't believe it is used on x86 outside of IPU6 setups.

The Fedora packages are using the main/master branch of all involved repositories. For the kernel drivers we carry one small downstream patch: https://github.com/intel/ipu6-drivers/pull/138 . This is used together with this modprobe.conf file: https://pkgs.rpmfusion.org/cgit/nonfree/ipu6-camera-hal.git/tree/intel_ipu6_isys.conf which one of the packages drops in /lib/modprobe.d

To build the kernel drivers you need to merge the contents of the ipu6-drivers and ivsc-driver repositories. Here are the steps which I take when manually building, assuming a dir with 2 clean git checkouts of ipu6-drivers and ivsc-driver:

cd ivsc-driver
cp -pr include backport-include drivers ../ipu6-drivers/
cd  ../ipu6-drivers/
make -j8
sudo make modules_install

For the rest all the tricky / special things we do are described in my blogpost: https://hansdegoede.dreamwidth.org/27235.html

Also see the packaging sources. esp. the .spec files for details:

https://pkgs.rpmfusion.org/cgit/nonfree/ipu6-camera-bins.git/tree/ https://pkgs.rpmfusion.org/cgit/nonfree/ipu6-camera-hal.git/tree/ https://pkgs.rpmfusion.org/cgit/nonfree/gstreamer1-plugins-icamerasrc.git/tree/ https://pkgs.rpmfusion.org/cgit/free/v4l2-relayd.git/tree/

I do see there that we are also carrying a small downstream compile fix for ipu6-camera-hal and a small fix to sync it up with the latest v4l2loopback event definitions for v4l2-relayd. I'm not sure what the status of upstreaming either of those is.

I do believe that there have been other packaging attempts for Arch so you may want to coordinate with those. If you have any more questions about this please send me an email at hdegoede@redhat.com .

[eiglow]

I just tried it on my Dell XPS 13 Plus today and it seems to work fine.

[jwrdegoede]

@simisimis

Dell XPS Plus(9320) with a ov01a10 sensor is also unlisted :/

There is an ov01a10 sensor driver included with the ipu6-drivers and there also is an ov01a10 sensor-profile included in ipu6-camera-hal. So I think the current ipu6-driver stack should work.

If you are not a Fedora user, maybe you can resize a partition to make some space to do a quick Fedora workstation install (installing Fedora from the workstation livecd is pretty quick and easy) and then follow the instructions from my blogpost to install and test the rpmfusion packaged drivers? :
https://hansdegoede.dreamwidth.org/27235.html

Note be careful when installing to either chose custom partitioning or install in free space, you don't want to overwrite your existing install. Fedora should pick up your current distro and at it to its grub menu. If not your laptop's bootmenu shown when pressing F12 should allow you to start your own distro's grub again. After the initial tests you may want to use efibootmgr (or your BIOS setup) to put your own distro's grub first in the boot order and then use the laptop's F12 boot menu to chose Fedora when you want.

[jwrdegoede]

@naps62 the current ipu6-drivers as provided by Intel do not include a driver for the OV01A1B so it seems you are out of luck, sorry.

[jwrdegoede]

@eiglow

I just tried it on my Dell XPS 13 Plus today and it seems to work fine.

Thank you for testing. Did you test with the rpmfusion packages ?

And just to make sure this is the Dell XPS Plus (9320) model, right? So the same laptop as @simisimis has ?

@simisimis I guess that means you don't need to install Fedora to test. Unless you want to see that it works for yourself.

[jwrdegoede]

@norbusan if you scroll back up here then @Thesola10 has a dkms aur for the ipu6-drivers: https://aur.archlinux.org/packages/intel-ipu6-dkms-git I don't know if that includes the full stack including v4l2loopback and v4l2-relayd but you should really coordinate any packaging efforts with @Thesola10 .

[HomelessCoder]

Hi @jwrdegoede, and thank you for your fantastic work and effort!

I'm on Precision 5470. I followed your guide to update the kernel and install the latest packages. I understand the model is not listed, though lsmod shows ov01a10 in the output. All the apps can now detect the camera (it's listed in Cheese, GMeet, Zoom), but only a black screen is shown. If there is anything I can do to help in testing this - I would be more than happy!

[naps62]

@naps62 the current ipu6-drivers as provided by Intel do not include a driver for the OV01A1B so it seems you are out of luck, sorry.

Actually mine is also a XPS plus 9320, so judging from other comments here, I guess I noted down the wrong string here

[jwrdegoede]

@HomelessCoder

I'm on Precision 5470. I followed your guide to update the kernel and install the latest packages. I understand the model is not listed, though lsmod shows ov01a10 in the output. All the apps can now detect the camera (it's listed in Cheese, GMeet, Zoom), but only a black screen is shown. If there is anything I can do to help in testing this - I would be more than happy!

For starters lets first try just the gstreamer icamerasrc plugin without involving v4l2loopback / v4l2-relayd. Please try running:

gst-launch-1.0 icamerasrc buffer-count=7 ! video/x-raw,format=NV12,width=1280,height=720 ! videoconvert ! xvimagesink

from a terminal and see if that works. Then after that please drop me an email at hdegoede@redhat.com with the results and lets discuss this further by email (to avoid hijacking this github issue).

[paulmenzel]

@jwrdegoede, thank you for writing that blog post. Just to be sure, that is basically the same solution, that Dell’s Ubuntu ships with, meaning that not all camera features are working (highest resolution?), and that system usage is higher than in MS Windows due to the loopback hacks?

[jwrdegoede]

@jwrdegoede, thank you for writing that blog post. Just to be sure, that is basically the same solution, that Dell’s Ubuntu ships with, meaning that not all camera features are working (highest resolution?), and that system usage is higher than in MS Windows due to the loopback hacks?

Correct.

There is one big difference though, this works with unmodified distro / upstream kernels where as Dell's Ubuntu pre-install only worked with special patched OEM kernel builds AFAIK.

[mkobel]

I would happily work on packaging this for Arch, but the repos are somehow a mess with too many branches and no clear indication which branch should be used. Would it be possible to summarize which sources and additional patches have been used for the Fedora packaging? @norbusan : Are you aware of https://github.com/stefanpartheym/archlinux-ipu6-webcam/issues/18#issuecomment-1557720342 ? This works for my X1 Carbon Gen 10 with Arch

[eiglow]

@eiglow

I just tried it on my Dell XPS 13 Plus today and it seems to work fine.

Thank you for testing. Did you test with the rpmfusion packages ?

And just to make sure this is the Dell XPS Plus (9320) model, right? So the same laptop as @simisimis has ?

Yes, I have the 9320. I just copy-pasted the commands from your blog post.

[gaul]

Yes, I have the 9320. I just copy-pasted the commands from your blog post.

I tested on the XPS 9315 using the blog post instructions but it cannot find the camera.

[zecakeh]

Hi, I have a Microsoft Surface Pro 8, that uses the ov5693 and ov13858 sensors. I can see in this repository that there is a driver for the ov13858 sensor, but it doesn't seem like it's included in the package generated by RPM Fusion. Is there a reason why?

[jwrdegoede]

Hi, I have a Microsoft Surface Pro 8, that uses the ov5693 and ov13858 sensors. I can see in this repository that there is a driver for the ov13858 sensor, but it doesn't seem like it's included in the package generated by RPM Fusion. Is there a reason why?

I just checked and the Makefile provided in Intel's ipu6-drivers repo does not build a module out of the ov13858_intel.c file. The ov13858 driver is not mentioned in the Makefile or Kconfig files at all.

[PhilDevProg]

There's a pull request for the OV13858 in this repository: https://github.com/intel/ipu6-drivers/pull/99 and one in the HAL repository: https://github.com/intel/ipu6-camera-hal/pull/45 but they don't seem to be active anymore...

[chromer030]

After reading some of the comments, i can tell that this is really a huge mess (and also a shame for Intel), and i hope to be solved quite soon.

[norbusan]

First of all your users need to be on kernel >= 6.3 and you need to disable the kernels own ov2740 sensor driver. It is ok to disable this on x86_64 builds since I don't believe it is used on x86 outside of IPU6 setups. [...]

Thanks @jwrdegoede for the details, in particular the links to the source packages (I searched for them on rpmfusion but that was confusing, and I am not RH trained).

Concerning the Arch intel-ipu6-dkms-git package, I have tried it few times, not a single one with success, while doing all the compile steps myself did work.

@mkobel Thanks for the link to https://github.com/stefanpartheym/archlinux-ipu6-webcam/issues/18#issuecomment-1557720342 I wasn't aware of this, very interesting.

[pinchartl]

After reading some of the comments, i can tell that this is really a huge mess (and also a shame for Intel), and i hope to be solved quite soon.

I wouldn't hold my breath for the "soon" part, but I have good hopes that we'll eventually have a proper solution, with upstream kernel drivers and open-source userspace code.

As previously discussed, four (main) components are required for an open-source upstream camera stack supporting the IPU6:

• Camera sensor kernel drivers
• IPU6 input system kernel driver
• IPU6 processing system kernel driver
• libcamera integration for all of the above

The input system driver has been submitted to the linux-media mailing list, and while it will take time for the multiple review-and-resubmit cycles, I think it will get merged at some point, hopefully before the end of the year. Once that's done, systems that integrate a camera sensor supported by the upstream kernel should be able to capture raw images without any closed-source code.

The last two steps will take more time. Until we have an IPU6 processing system kernel driver with a documented userspace API, we won't be able to start work in libcamera. There's no ETA at this point, so it's likely too early to recruit volunteers to implement IPU6 support in libcamera. I'm hoping that we could get a working solution within two years.

[pinchartl]

@pinchartl - Camera support.

(Context: closed source binary blobs)

It will probably not come as a surprise that this is not an acceptable solution for me at all :) While I understand other users and developers may have a different opinion, and don't mind spending time and effort to support a closed-source solution, this does nothing to cleanse Intel of its original sin when it dumped that blob onto us. I would even go as far as saying that integrating the closed-source solution into distribution to offer an easy out-of-the-box experience to users is counterproductive in the battle to get vendors to provide camera support without closed-source components.

[dmipx]

As all ipu6 laptops have raw bayer sensors, you will be able (with tambourine) to capture bayer images with upstreamed driver (in a year or so). PSYS role is convert bayer image to nv12. there is no acceleration for yuyv format for now (another year). One of feasible opensource solutions is to use SIMD accelerated conversion of bayer frames. My proposal is as follow:

1. Use RealSense development #73 for ipu6 native camera support.
2. Implement bayer transfer function in isys driver. Once I will get my hands on alderlake laptop, it is two week of effort for full integration.

[pinchartl]

As all ipu6 laptops have raw bayer sensors, you will be able (with tambourine) to capture bayer images with upstreamed driver (in a year or so). PSYS role is convert bayer image to nv12. there is no acceleration for yuyv format for now (another year). One of feasible opensource solutions is to use SIMD accelerated conversion of bayer frames. My proposal is as follow:

1. Use RealSense development [ipu6: ipu-isys-video video node support for camera subdev #73](https://github.com/intel/ipu6-drivers/pull/73) for ipu6 native camera support.

If your goal is to support RAW capture only as a starting point, why should we use that out-of-tree code instead of working on getting the ISYS driver posted to the linux-media mailing list (https://lore.kernel.org/linux-media/20230413100429.919622-1-bingbu.cao@intel.com/) upstream ? We could do with testers, and more importantly, reviewers. If anyone is interested in accelerating upstream of the IPU6 ISYS kernel driver, please review the submitted patches on the linux-media mailing list.

2. Implement bayer transfer function in isys driver.

Do you mean implementing RAW bayer capture, or implementing CFA interpolation (a.k.a. debayering) in software in the kernel driver ? The latter won't make it upstream, and won't be enough to get usable images anyway as you'll have no auto-exposure, auto-white balance or auto-focus.

   Once I will get my hands on alderlake laptop, it is two week of effort for full integration.

[dmipx]

If your goal is to support RAW capture only as a starting point, why should we use that out-of-tree code instead of working on getting the ISYS driver posted to the linux-media mailing list (https://lore.kernel.org/linux-media/20230413100429.919622-1-bingbu.cao@intel.com/) upstream ? We could do with testers, and more importantly, reviewers. If anyone is interested in accelerating upstream of the IPU6 ISYS kernel driver, please review the submitted patches on the linux-media mailing list. This is good idea, the suggested patch can be used on plain ipu6 as it's only enhances integration with subdevice - inherits controls, report formats and framerates, redirects all v4l2 controls through video node to subdevices in media link chain. This simplifies integration with userspace applications throwing off libcamera.

2. Implement bayer transfer function in isys driver.

Do you mean implementing RAW bayer capture, or implementing CFA interpolation (a.k.a. debayering) in software in the kernel driver ? The latter won't make it upstream, and won't be enough to get usable images anyway as you'll have no auto-exposure, auto-white balance or auto-focus.

ISYS captures camera sensor image of bayer format, ov2740 for example, can stream only SGRBG10 which should be converted to NV12 or YUYV in order to be supported with userspace applications. How is auto-exposure, auto-white balance or auto-focus implemented? Isn't thats function of application to control camera sensor? ov2740 has no "auto" functions but manual controls.

The kernel implementation is simple but this is indeed ugly solution. I doubt that other than suggested userspace solution can supply upstreamable solution. There is again number of options:

1. Use libcamera to convert bayer to application firendly format. In that case - applications that can use libcamera in win situation, but all others still need v4l2-loopback solution and we do not want it.
2. Implement full integration of isys with psys and forward captured frame from isys to psys within ipu6 capture driver. This will require Intel blobs with psys preload. (These blobs can be rendered as binary firmware?)
3. TBD

[pinchartl]

ISYS captures camera sensor image of bayer format, ov2740 for example, can stream only SGRBG10 which should be converted to NV12 or YUYV in order to be supported with userspace applications.

Demosaicing and RGB to YUV conversion is only a small part of the processing pipeline needed to produce a decent image. You need lens shading (and possibly distortion) correction, various types of denoising, tone mapping, ... All this is performed by the PSYS. Without that, this is what you can expect out of the box when capturing raw frames and "converting" them to YUV in software:

https://user-images.githubusercontent.com/70438257/108349559-6753eb80-71e3-11eb-8b0b-cc44f3021800.png

With ISP (PSYS) support and rudimentary control algorithms, the image becomes usable:

https://user-images.githubusercontent.com/70438257/108349353-2c51b800-71e3-11eb-8caf-ff77205ed6d7.png

(Those are taken on an IPU3-based system)

How is auto-exposure, auto-white balance or auto-focus implemented? Isn't thats function of application to control camera sensor? ov2740 has no "auto" functions but manual controls.

The PSYS computes statistics on every image, and control algorithms running in userspace consumes those statistics and compute the appropriate sensor analog gain and exposure time, focus lens position, and hundreds of PSYS parameters (such as the colour gains for auto-white balance). The statistics could be computed in software, but that's very costly if you want to achieve good quality.

While the control algorithms could in theory be implemented by applications, they are complex, and applications developers shouldn't have to deal with them (not to mention that they are platform-specific). That's why we have libcamera, which handles all that behinds the scenes, and delivers processed images to applications.

The kernel implementation is simple but this is indeed ugly solution. I doubt that other than suggested userspace solution can supply upstreamable solution. There is again number of options:

1. Use libcamera to convert bayer to application firendly format. In that case - applications that can use libcamera in win situation, but all others still need v4l2-loopback solution and we do not want it.

We could implement software-based processing of images in libcamera to improve the raw images a little bit, but it will be costly in terms of CPU time. It's a stop-gap option though. Another option is to accelerate this using the GPU (still in libcamera).

libcamera offers a GStreamer element, and a V4L2 compatibility layer in the form of a library that is LD_PRELOAD-ed. This allows supporting existing applications, and it supports zero-copy, unlike v4l2-loopback.

2. Implement full integration of isys with psys and forward captured frame from isys to psys within ipu6 capture driver. This will require Intel blobs with psys preload. (These blobs can be rendered as binary firmware?)

Once we have a proper PSYS driver, we plan to integrate support for that in libcamera, and implement open-source ISP control algorithms completely independently from the Intel binary blobs. This is what we have done for the IPU3.

3. TBD