4k video does not work

[doublej472]

When I send a 4k video feed to my camlink 4k, it changes the correct color space from YUYV 4:2:2 to Planar YUV 4:2:0, which means that this library stops working.

Is there a way to catch the resolution as well when blocking color spaces? so that if the resolution is 3840x2160, only allow through Planar YUV 4:2:0, otherwise only allow through YUYV 4:2:2

[xkahn]

@doublej472 Hi Jonathan, This is certainly possible, but has various levels of difficulty depending on what you would need it to do. Auto-detection of the video stream could get very complicated. The very small script is making blind assumptions about the video feed right now.

The rule you suggest should be possible easily, but I'm not sure it's general purpose enough to work for others. Here's the outline of how the application works now:

1. We intercept the all ioctl calls made by the application and pass along unchanged all calls that are NOT VIDIOC_ENUM_FMT
2. If we find that call, we call another ioctl (VIDIOC_QUERYCAP) to see if we are looking at a Cam Link device. (If not we forward the call along to the kernel.)
3. Next, we call the ioctl the application wanted and check to see if the return includes the color space we want, if so we return the result. If not, we try again until we get the result we want or run out of options. (We also leak the colorspace # back to the calling application so it can call us again, if needed.)

Your problem could be solved by:

4. Calling ANOTHER ioctl : VIDIOC_ENUM_FRAMESIZES in a loop to determine if the video size 3840x2160 is available when the color space is Planar YUV 4:2:0. If so, return that color space. If the video size 3840x2160 is available when the color space is YUYV 4:2:2 remove it from the list.

It's probably not a complex patch, but, again, it doesn't feel like a universal solution. Maybe this would need a configuration file?

[cosimo]

As a side note, Planar YUV 4:2:0 does not seem to work in any case, at least with my camera and camlink 4K.

[bdrung]

There is a bug in the firmware which breaks all pixel formats except the first one. The first pixel format changes depending on the HDMI input (YUYV 4:2:2 or Planar YUV 4:2:0). I developed a quirk to fix the buggy firmware and submitted it to the upstream Linux: https://lore.kernel.org/lkml/20210325213458.51309-1-bdrung@posteo.de/

[xkahn]

@bdrung Are you saying that the FIRST pixel format is always the right answer from the camlink? Even if the source video pixel format changes?

[bdrung]

Changing to the first pixel format will always work, because the malformed response of the buggy firmware is accidentally identical to tho corrected response. So in case of

v4l2-ctl --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
    Type: Video Capture

    [0]: 'NV12' (Y/CbCr 4:2:0)
        Size: Discrete 3840x2160
            Interval: Discrete 0.033s (29.970 fps)
    [1]: 'NV12' (Y/CbCr 4:2:0)
        Size: Discrete 3840x2160
            Interval: Discrete 0.033s (29.970 fps)
    [2]: 'YU12' (Planar YUV 4:2:0)
        Size: Discrete 3840x2160
            Interval: Discrete 0.033s (29.970 fps)

using NV12 will work. For FullHD input, the first format is normally YUYV.

[xkahn]

Perfect. I'll update the camlink LD_LIBRARY to reflect that reality (which will resolve this ticket.)

Obviously a kernel quirks patch will solve the issue permanently, but may take longer to get deployed to people.

[xkahn]

@doublej472 Fix pushed. Let me know if this solves your issue.

[ooduor]

suggestion (niche): This may not be a technical solution but I found out, depending on camera, I had success with the color-space by changing the mode settings (onSony A6100) from "Video" mode to any other mode e.g. S&Q, M, A, P. You can find more details in my fork wiki.

[xkahn]

What happens when it doesn't work?

On March 26, 2021 6:43:43 PM Antony Oduor @.***> wrote:

suggestion (niche): This may not be a technical solution but I found out, depending on camera, I had success with the color-space by changing the mode settings (onSony A6100) from "Video" mode to any other mode e.g. S&Q, M, A, P. You can find more details in my fork wiki. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[bjornedstrom]

Thanks xkahn for a nice little project!

I want to share some observations. I've tried the Cam Link and my camera (Fuji X-T3) in two modes.

• If I set my camera to 4K 29.97fps, I get the same output as bdrung above (index 0 is 'NV12' (Y/CbCr 4:2:0), index 2 is Planar YUV)
• If I set my camera to FullHD (1920x1080) 25fps, I get the following order:

  $ v4l2-ctl --list-formats-ext -d /dev/video5
  ioctl: VIDIOC_ENUM_FMT
  Index       : 0
  Type        : Video Capture
  Pixel Format: 'YUYV'
  Name        : YUYV 4:2:2
      Size: Discrete 1920x1080
          Interval: Discrete 0.040s (25.000 fps)
  Index       : 1
  Type        : Video Capture
  Pixel Format: 'NV12'
  Name        : Y/CbCr 4:2:0
      Size: Discrete 1920x1080
          Interval: Discrete 0.040s (25.000 fps)
  Index       : 2
  Type        : Video Capture
  Pixel Format: 'YU12'
  Name        : Planar YUV 4:2:0
      Size: Discrete 1920x1080
          Interval: Discrete 0.040s (25.000 fps)

Observation 1

The latest Zoom client for Linux works well and captures from the Cam Link without any modification at all, both in 4K and "FullHD". That is, I do not need to use camlink.so in Zoom.

Observation 2

Now for everything else (firefox, etc), I need to use camlink.so otherwise nothing works (errors like "camera not found" or completely black output). However:

• If shoot in 4K 29.9fps, In Firefox with camlink.so, I get image but the color is broken (black and white-ish, with some very faint color once in a while)
• If I change to 1920x1800 25fps, everything works perfectly.

Hypothesis?

I'm gonna continue to play around a little bit, but it seems to me that, in 4K for my particular setup, index 0 is not the obvious best option (Zoom picks a different one?)- I will see if index 1 or index 2 works better, when I have some time, if that gets rid of the black-and-white problem.

[Snuggle]

suggestion (niche): This may not be a technical solution but I found out, depending on camera, I had success with the color-space by changing the mode settings (onSony A6100) from "Video" mode to any other mode e.g. S&Q, M, A, P. You can find more details in my fork wiki.

I own a Sony Alpha 6300 and it works perfectly in movie mode only when I plug it in via a dedicated USB 3.0 port. (Assuming I also pick the first colour space [0]: 'NV12' (Y/CbCr 4:2:0) as discussed above) If I plug it into a slower 2.0 port, it fails to produce video under movie mode and instead shows a blank black video. This is likely because movie mode is the only mode that will actually output full 3840x2160 video whereas other modes aren't as high quality and require less bandwidth over USB?

Hope this helps!