image sync problem?

[lerabot]

Hello, first of all thank you for your pcb layout :) I fab'd 5 board last month and they work great.

I have an image sync issue and I'm wonderig you you had the same problem, check the screenshot.

I'm capturing using s-video NTSC, but I can see this problem even on composite.

[rniwase]

Hi, @lerabot. Thanks for making sure it works.

I am aware of this problem and apologize for not announcing it in advance.

I have tried to solve this problem by directly rewriting the registers of the ADV7280A-M, but I have not been able to solve it yet.

I believe that the problem is probably in the hardware design. In particular, there may be a fault in the power supply or the MIPI CSI transmission line.

I am too busy with other work to tackle this problem right now, but I will try to get an evaluation board from Analog Devices someday to compare the operation.

[lerabot]

Thank for the quick feedback. I might try to write a script that try to detect and stitch the image together for the time being.

Also, I posted on the raspberry pi forum about this. https://forums.raspberrypi.com/viewtopic.php?p=2111105#p2110939

I might try to hack together something with the ADM7280 driver at some point to try to correct the power up timing sequence.

[lerabot]

Hello again. I'm resuming work on this and will try hacking with the register again. If you have any clues on how to fix this in software, I'll gladly take them.

[rniwase]

Thanks for waiting so long. I'd like to share what I've learned in the past few weeks.

As answered in the Raspberry Pi or Analog Devices forums, the data output by the ADV7280-M with NTSC as input format, ITU-R BT.656-3 as output format, and I2P enabled is not 480 lines per frame, but The data output by ADV7280-M is not 480 lines per frame, but 507 lines per frame including blanking data.

For this, I actually confirmed using an FPGA board (TE0726-03M) and MIPI CSI-2 receiver that 507 YUV422 packets come per frame. The relationship between video format settings and output resolution is described in detail in Table 72 of the device manual (UG-1176). https://www.analog.com/media/en/technical-documentation/user-guides/ADV7280A-7281A-7282A-UG-1176.pdf

However, the Linux kernel 6.1.y device driver running on the Raspberry Pi OS Bullseye is set to a height that reserves a buffer of 480 lines per frame, so it appears that the 27 lines of difference are overwritten at the beginning of the buffer during recording. The result is a problem that results in images that appear to be out of sync.

To solve the problem, I changed the height in the device driver (adv7180.c) to 507 instead of 480.

This allows the user to obtain a complete image, although the upper 27 lines contain extra blanking data.

I feel that to solve all of these problems, not only the ADV7280 device driver, but also unicam and V4L2 API must be investigated thoroughly.

(Dear @6by9, ~author of the great adv7180 driver.~ If you have any advice on this area, we would appreciate your comments...)

My environment and kernel module build procedure are shown below.

• Board: Raspberry Pi 4B 8GB
• OS: Raspberry Pi OS Lite 64-bit (Bullseye)

  $ uname -a
  Linux svr-tommib 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023 aarch64 GNU/Linux
  $ cat /etc/os-release
  PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
  NAME="Debian GNU/Linux"
  VERSION_ID="11"
  VERSION="11 (bullseye)"
  VERSION_CODENAME=bullseye
  ID=debian
  HOME_URL="https://www.debian.org/"
  SUPPORT_URL="https://www.debian.org/support"
  BUG_REPORT_URL="https://bugs.debian.org/"

  
  # Enter root login shell
  sudo su -

Install dependencies

apt-get update apt-get -y upgrade apt-get -y install git bc bison flex libssl-dev

Download the corresponding Linux kernel source from the Git hash of the specific firmware

Note: The following is for kernel version 6.1.21-v8+

FW_GIT=1440b3e0b52075a9ec244216cddcf56099c28dfa KERNEL_GIT=$(curl -fL https://github.com/raspberrypi/rpi-firmware/raw/${FW_GIT}/git_hash) curl -fL https://github.com/raspberrypi/linux/archive/${KERNEL_GIT}.tar.gz | tar -zx -C /usr/src/

Download symbol file

curl -fL https://github.com/raspberrypi/rpi-firmware/raw/${FW_GIT}/Module8.symvers > /usr/src/linux-${KERNEL_GIT}/Module.symvers

Porting configs from current system

modprobe configs zcat /proc/config.gz > /usr/src/linux-${KERNEL_GIT}/.config

Generate files needed to build kernel modules

cd /usr/src/linux-${KERNEL_GIT}/ make -j$(nproc) modules_prepare

Preparing for adv7180 kernel module build

mkdir /root/adv7180_hack/ cd /root/adv7180_hack/ cp /usr/src/linux-${KERNEL_GIT}/drivers/media/i2c/adv7180.c ./ echo "obj-m := adv7180.o" > Makefile

Modify adv7180.c

sed -i.bak -e "s/480 : 576/507 : 576/" adv7180.c diff adv7180.c.bak adv7180.c 742c742 < fmt->height = state->curr_norm & V4L2_STD_525_60 ? 507 : 576;

  fmt->height = state->curr_norm & V4L2_STD_525_60 ? 480 : 576;

Build modified adv7180 kernel module

make -C /usr/src/linux-${KERNEL_GIT}/ M=$PWD

Install modified module

mv /lib/modules/$(uname -r)/kernel/drivers/media/i2c/adv7180.ko.xz /lib/modules/$(uname -r)/kernel/drivers/media/i2c/adv7180.ko.xz.bak xz -c /root/adv7180_hack/adv7180.ko > /lib/modules/$(uname -r)/kernel/drivers/media/i2c/adv7180.ko.xz

Setup dtoverlay and reboot

echo "dtoverlay=adv728x-m,adv7280m=1" >> /boot/config.txt reboot

Video recording test

sudo apt-get -y install ffmpeg ffmpeg -an -video_size 720x507 -r 29.97 -i /dev/video0 -c:v rawvideo -vf realtime -t 5 out.asf ffmpeg -an -video_size 720x507 -r 29.97 -i /dev/video0 -c:v rawvideo -vf realtime,crop=720:480:0:27 -t 5 out_crop.asf

[6by9]

(Dear @6by9, author of the great adv7180 driver. If you have any advice on this area, we would appreciate your comments...)

I'm not the author of it. It's a mainline Linux driver, originally written by Mocean Laboratories, with copyright listed as originally belonging to Intel. MAINTAINERS lists "Lars-Peter Clausen lars@metafoo.de" as the maintainer of the driver, with linux-media@vger.kernel.org being the correct mailing list for requesting assistance.

The data output by ADV7280-M is not 480 lines per frame, but 507 lines per frame including blanking data.

Previously referenced on the forums https://forums.raspberrypi.com/viewtopic.php?t=308902, and acknowledged by ADI in https://ez.analog.com/video/f/q-a/6495/adv7280-frame-timing-on-the-digital-output-video-bus/20601#20601

I feel that to solve all of these problems, not only the ADV7280 device driver, but also unicam and V4L2 API must be investigated thoroughly.

It's down to ADI to come up with some answers really, but it may actually be down to the NTSC spec based on ADI's comment "Unlike PAL, the NTSC standard does not have fixed vertical blanking specifications". It's nothing wrong in Unicam or V4L2.

Trying to fix it in the adv7180 driver is going to be slightly ugly as it only wants to be fixed for the CSI based chips when the I2P is active. Lots of conditionals to mess with. Simplest initial approach is to ping linux-media and Lars-Peter Clausen with the above link from ez.analog.com and see what they have to say, however nicer if you can send a patch that fixes the issue. It'd probably be along the lines of

diff --git a/drivers/media/i2c/adv7180.c b/drivers/media/i2c/adv7180.c
index 413021887f96..bdec357d69aa 100644
--- a/drivers/media/i2c/adv7180.c
+++ b/drivers/media/i2c/adv7180.c
@@ -773,7 +773,11 @@ static int adv7180_mbus_fmt(struct v4l2_subdev *sd,
                fmt->code = MEDIA_BUS_FMT_UYVY8_2X8;
        fmt->colorspace = V4L2_COLORSPACE_SMPTE170M;
        fmt->width = 720;
-       fmt->height = state->curr_norm & V4L2_STD_525_60 ? 480 : 576;
+       if (state->chip_info->flags & ADV7180_FLAG_MIPI_CSI2 &&
+           state->field == V4L2_FIELD_NONE)
+               fmt->height = state->curr_norm & V4L2_STD_525_60 ? 525 : 576;
+       else
+               fmt->height = state->curr_norm & V4L2_STD_525_60 ? 480 : 576;

        if (state->field == V4L2_FIELD_ALTERNATE)
                fmt->height /= 2;

however you have no way of telling userspace to ignore the top and bottom N lines of blanking that the chip produces. (Oh, I'd written almost exactly the same patch in https://forums.raspberrypi.com/viewtopic.php?p=1854418#p1854418, but hadn't included checking the CSI2 flag)

Observation: Page 51 of https://www.analog.com/media/en/technical-documentation/user-guides/adv7280_7281_7282_7283_ug-637.pdf

Note the BT.656-4 bit also affects the MIPI CSI-2 active video output resolution from ADV728x-M parts.

So you could potentially get 720x487 out of the chip by setting the BT656-4 flag - that would be the ADV7180_REG_EXTENDED_OUTPUT_CONTROL write, which looks to be set to 0x57 for CSI2 chips (https://github.com/torvalds/linux/blob/master/drivers/media/i2c/adv7180.c#L1038). Alter that to 0xd7 and you should get the 487 line mode. I'd ideally like to know what the ADI engineer was seeing captured for the blanking lines after the image when they analysed it. If it sends CSI2 packets for the image data, then we have to have the memory to receive them.

[rniwase]

Hi @6by9, thank you for leaving your detailed comments! and please forgive me for my misunderstanding that you are the author of the driver.

Also, thanks for pointing us to the specific kernel patches.

however you have no way of telling userspace to ignore the top and bottom N lines of blanking that the chip produces.

This is exactly the information I wanted to know, and for that I thought I needed to look into unicam or V4L2 implementations.

So you could potentially get 720x487 out of the chip by setting the BT656-4 flag - that would be the ADV7180_REG_EXTENDED_OUTPUT_CONTROL write, which looks to be set to 0x57 for CSI2 chips

Actually, there is a slight problem with this. I have confirmed that the number of YUV422 packets obtained by setting the BT656-4 flag to 1 when I2P is disabled is 243 or 244, but for some reason the packet count remains 507 instead of 487 when I2P is enabled. (these were verified by a CSI receiver implemented on an FPGA and a logic analyzer.) However, I have not yet confirmed the difference in images obtained by enabling/disabling BT656-4 flag and I2P, and there is a possibility that I may have made an operational error in the first place, so I will investigate further.

I'd ideally like to know what the ADI engineer was seeing captured for the blanking lines after the image when they analysed it. If it sends CSI2 packets for the image data, then we have to have the memory to receive them.

I know that the ADV7280A-M sends a YUV422 long packet containing video data and then sends several blanking packets before sending a Frame End packet. but I have not confirmed what data is contained in the payload of the blanking packets, so I will try to set up a debugging environment to dump all the packets to confirm the details.

[rniwase]

@lerabot I have updated the procedure in the README of this commit (694bd97).

I initially expected that there was something wrong with the hardware (PCB) design, but it turns out that I can work around this problem by rewriting the device driver.

so please refer to the procedure and check the operation.

[lerabot]

Hello to both of you and thank for all your research. In the past month or so I've followed and read a bunch of those topic and applied a very similar patch myself.

I'll test your version of the patch tomorrow or today and see if I can get better result.

[6by9]

however you have no way of telling userspace to ignore the top and bottom N lines of blanking that the chip produces.

This is exactly the information I wanted to know, and for that I thought I needed to look into unicam or V4L2 implementations.

So you could potentially get 720x487 out of the chip by setting the BT656-4 flag - that would be the ADV7180_REG_EXTENDED_OUTPUT_CONTROL write, which looks to be set to 0x57 for CSI2 chips

Actually, there is a slight problem with this. I have confirmed that the number of YUV422 packets obtained by setting the BT656-4 flag to 1 when I2P is disabled is 243 or 244, but for some reason the packet count remains 507 instead of 487 when I2P is enabled. (these were verified by a CSI receiver implemented on an FPGA and a logic analyzer.) However, I have not yet confirmed the difference in images obtained by enabling/disabling BT656-4 flag and I2P, and there is a possibility that I may have made an operational error in the first place, so I will investigate further.

Whilst I did have a branch that allowed reception of interlaced content (ie I2P disabled), it was a bit too hacky to be merged as the Unicam driver needed knowledge of how the ADV signaled the field (using the frame count value IIRC).

I'll see if I can find a few minutes to hack together testing with BT656-4 register tweaked, and whether it makes a useful difference.

Producing 507 lines feels wrong - it'd be nice to investigate further, but I doubt I'll have time. ADI may answer further on their forums if the question is asked.

Seeing as I have a couple of patches for this chip that would be worth sending upstream, I may send the trivial fix above along with it to at least start the discussion. I don't know how the crop should be advertised though, and want to confirm 507 vs 525 lines as well.

[lerabot]

Been working on this for the past couple of days. It's a shame that we can't get proper interlaced capture because I'm fairly sure we could get a better quality out of software deinterlacer. I've been looking at the interlaced unicam driver and it does seem quite hacky. I'll try to give a try later.

[6by9]

The issue is that there is no standardisation as to how things like interlacing should be signaled on CSI2, and V4L2's frameworks don't provide a mechanism for the receiver to pass side-band information back to the source device driver to give it a chance to interpret it and amend buffer descriptors accordingly.

Looking at my branch at https://github.com/6by9/linux/tree/rpi-5.10.y-unicam-interlaced, ADV728x uses the 16bit frame_count field in the frame start and end packets.

The Toshiba TC358743 HDMI to CSI2 bridge can also receive interlaced content, however it sends the 2 fields with different CSI2 data types. It would therefore need a similar routine to set the interlacing flags, but based on UNICAM_CDT instead of UNICAM_CWC. (We'd also need to programme both DTs that represent the interlaced image data).

IIRC The I2P block is a relatively simple line doubler, or possibly interpolate between the 2 lines in each field. Yes software could probably do a more comprehensive job. Please be aware that the VideoCore VPU deinterlacer doesn't handle YUYV data (only planar YUV420), so you would have to be running it on the ARM cores. As the buffers that are used by the CSI2 receiver are uncached, you may have a performance penalty there.