Interlaced issues? Never saw this before.

[TheSashmo]

Excuse the posts tonight everyone.

Here is another issue. I never noticed before, but my source is interlaced 1080i59.94 10-bit and I can see on the output of the decoder interlacing..... Crazy. Anyone know why and how to get around this?

[TheSashmo]

Top picture is input. Bottom picture is the output.

Tested on vaapi and non vaapi, same result.

Encoder: ./UltraGrid-1.8.2-x86_64.AppImage -t decklink:0 -c libavcodec:encoder=h264_vaapi:bitrate=18000k -s embedded --audio-codec=MP3:sample_rate=48000:bitrate=256k --audio-capture-format channels=16 -l unlimited -m 1316 -P 22816 192.168.99.209

Decoder: ./UltraGrid-1.8.2-x86_64.AppImage -d decklink:device=0:drift_fix -r embedded 192.168.99.209 -P 22816

[TheSashmo]

Tried mjpeg for fun and its the same result. Maybe I am crazy, but I never noticed this before.

[MartinPulec]

I'll try to test next week. Anyways, just initial thoughts:

1) mjpeg and h264_vaapi doesn't have interlaced mode but libx264 does. Anyways, even when those doesn't have interlaced mode, it works encoding as progressive (keeping track that the signal is interlaced), so there should not be a problem, it works fine all time. 2) could you check the output that the captured signal is really recognized as interlaced? I was thinking if it doesn't for some reason think that it is PsF 3) lower/upper field first - this should be observable even on steady picture. I don't know when, but BMD may provide lower field first. Than the fields would be swapped, I've just added a warning to decklink capture. It is not handled otherwise - I personally haven't encounter that and I don't know if it occurs.

[TheSashmo]

[DeckLink capture] Lower field first format detected, fields can be switched! If so, please report a bug to ultragrid-dev@cesnet.cz

Is that the output you are looking for? There is clearly an issue. When I try to add arguments decklink:device=0:mode=11:codec=v210:p_not_i it turns the output to progressive and reduced framerate.

I tried also to use :x264-params=bff=1 or :x264-params=tff=1 in the encoding, but same result....

[TheSashmo]

@MartinPulec I can give you remote ssh access into the device and you could check. I do think that the signal could be coming out as PSF but at the moment I don't have an easy way to confirm that other than visually. I honestly believe this is the same issue that I have been complaining about for a while. I can see a "skipping" of frames on the output of the SDI, but the logs don't show any reduction in processed frames in 5 seconds.

I have tried changing presets to reduce the power consumption, as well dropping to 8 bit 4:2:0, and no matter what I do, I see skipping of video frames on the output.

Any input here would be greatly appreciated.

[MartinPulec]

[DeckLink capture] Lower field first format detected, fields can be switched! If so, please report a bug to ultragrid-dev@cesnet.cz

Is that the output you are looking for?

Exactly!

There is clearly an issue. When I try to add arguments decklink:device=0:mode=11:codec=v210:p_not_i it turns the output to progressive and reduced framerate.

Agree, but there is something that looks suspicious to me. Would you be able to run with development version (either a continuous build or including the commit 52608911):

uv -t decklink:fullhelp

and post the output here? Remote access isn't necessary. This could be enough to evaluate the situation and the information will remain in this thread.

To explain what I need – almost all DeckLink interlaced modes used upper-field-first field order, with the exception of NTSC, that obviously is very rarely used (as far as I know we haven't got a report for it). But you doesn't seem to be using NTSC, because it seems to use always index 0. So it is unclear to me why now it is lower-field-first (unless the device supports both variants, in which case I'd be curious what is the default auto-detected).

Your workaround with bff=1 is clever but I think it won't work by now, because the receiver is configured to set upper-field-first mode. But it depends whether the receiving device supports tff, bff or both variants of the mode.

[TheSashmo]

Thanks @MartinPulec here you go:

`root@user_test:~# ./UltraGrid-continuous-x86_64.AppImage -t decklink:fullhelp UltraGrid 1.8+ (master rev 169bfee built Apr 11 2023 10:33:09)

Decklink options: -t decklink{:mode=|:device=|:codec=...=}* | decklink:[full]help or -t decklink[:<device_index(indices)>[::[:3D][:sync_timecode][:connection=][:audio_level={line|mic}][:detect-format][:conversion=]] (mode specification is mandatory if your card does not support format autodetection; syntax on the first line is recommended, the second is obsolescent)

3D Use this to capture 3D from supported card (eg. DeckLink HD 3D Extreme). Do not use it for eg. Quad or Duo. Availability of the mode is indicated in video format listing above by flag "3D".

audio_level line - the selected analog input gain levels are used mic - analog audio levels are set to maximum gain on audio input.

detect-format Try to detect input video format even if the device doesn't support autodetect, eg. "-t decklink:connection=HDMI:detect-format".

fullhelp Print description of all available options.

half-duplex Set a profile that allows maximal number of simultaneous IOs.

conversion none - No video input conversion 10lb - HD1080 to SD video input down conversion 10am - Anamorphic from HD1080 to SD video input down conversion 72lb - Letter box from HD720 to SD video input down conversion 72ab - Letterbox video input up conversion amup - Anamorphic video input up conversion Then use the set the resulting mode (!) for capture, eg. for 1080p to PAL conversion: -t decklink:mode=pal:conversion=10lb

p_not_i Incoming signal should be treated as progressive even if detected as interlaced (PsF).

Use1080PsF[=true|false] Incoming signal should be treated as PsF instead of progressive.

nosig-send Send video even if no signal was detected (useful when video interrupts but the video stream needs to be preserved, eg. to keep sync with audio).

[no]passthrough[=keep] Disables/enables/keeps capture passthrough (default is disable).

profile=|profile=keep Use desired device profile: 1dfd, 1dhd, 2dfd, 2dhd or 4dhd. See SDK manual for details. Use keep to disable automatic selection.

sync_timecode Try to synchronize inputs based on timecode (for multiple inputs, eg. tiled 4K)

keep-settings do not apply any DeckLink settings by UG than required (keep user-selected defaults)

Available color spaces: R12L, R10k, v210, RGBA, UYVY

Devices: 0) DeckLink SDI Micro capture modes: 0 (ntsc)) 525i59.94 NTSC 720 x 486 29.97 FPS lowr, flags: Rec601 1 (pal )) 625i50 PAL 720 x 576 25.00 FPS uppr, flags: Rec601 2 (23ps)) 1080p23.98 1920 x 1080 23.98 FPS prog, flags: Rec709 3 (24ps)) 1080p24 1920 x 1080 24.00 FPS prog, flags: Rec709 4 (Hp25)) 1080p25 1920 x 1080 25.00 FPS prog, flags: Rec709 5 (Hp29)) 1080p29.97 1920 x 1080 29.97 FPS prog, flags: Rec709 6 (Hp30)) 1080p30 1920 x 1080 30.00 FPS prog, flags: Rec709 7 (Hp50)) 1080p50 1920 x 1080 50.00 FPS prog, flags: Rec709 8 (Hp59)) 1080p59.94 1920 x 1080 59.94 FPS prog, flags: Rec709 9 (Hp60)) 1080p60 1920 x 1080 60.00 FPS prog, flags: Rec709 10 (Hi50)) 1080i50 1920 x 1080 25.00 FPS uppr, flags: Rec709 11 (Hi59)) 1080i59.94 1920 x 1080 29.97 FPS uppr, flags: Rec709 12 (Hi60)) 1080i60 1920 x 1080 30.00 FPS uppr, flags: Rec709 13 (hp50)) 720p50 1280 x 720 50.00 FPS prog, flags: Rec709 14 (hp59)) 720p59.94 1280 x 720 59.94 FPS prog, flags: Rec709 15 (hp60)) 720p60 1280 x 720 60.00 FPS prog, flags: Rec709

Connection can be one of following:
    SDI

Examples: /tmp/.mount_UltraGGIKhcG/usr/bin/uv -t decklink # captures autodetected video from first DeckLink in system /tmp/.mount_UltraGGIKhcG/usr/bin/uv -t decklink:0:Hi50:UYVY # captures 1080i50, 8-bit yuv /tmp/.mount_UltraGGIKhcG/usr/bin/uv -t decklink:0:10:v210:connection=HDMI # captures 10th format from a card (alternative syntax), 10-bit YUV, from HDMI /tmp/.mount_UltraGGIKhcG/usr/bin/uv -t decklink:mode=23ps # captures 1080p24, 8-bit yuv from first device /tmp/.mount_UltraGGIKhcG/usr/bin/uv -t "decklink:mode=Hp30:codec=v210:device=DeckLink HD Extreme 3D+" # captures 1080p30, 10-bit yuv from DeckLink HD Extreme 3D+

This UltraGrid version was compiled against DeckLink SDK 11.6. System version is 11.7.

Exit root@user_test:~#`

[TheSashmo]

Let me explain in a bit more details to explain what I am seeing.

If I use a hyperdeck with some pre-recorded content "high motion" like a hockey or American football in Hi59 and just do a local playback, i.e. encode and decode on the same device, I am not able to notice any visual playback problems, other than the interlacing issue I expressed, and any bitrate compressions which is logical. BUT, when I take high end camera with SDI output or from high end production switcher and use that as my source, in the same resolution and framerate Hi59, I can visually see frames dropped on the output with gentle motion. The logs output for encode and decode report back a constant 150 frames in 5 seconds, but no matter what I do the video is choppy during motion. In fact, is doesn't matter if I do hp59 or hp60 from the camera, I can see on the sdi output visually a frame dropping here or there and all I do is move camera left to right on the tripod, or if I put someone in front of it and they are swaying side to side. I thought maybe that the issue was not enough CPU, so I tried even using lower presets like ultrafast, and forcing 8-bit 4:2:0 instead of the default of what was detected, and its the same darn result. Load average on the machine is 6 on a 12 core machine, but again no matter what I do, I still get this stutter effect. Any guidance here would be grateful, sorry for the long rant, I am driving and using voice to text.

[MartinPulec]

I am sorry to say, but it doesn't seem to be the problem with the field order, in the end. I got it when looking into your modes where the only bff is NTSC. But NTSC is actually picked for autodetection. So, I don't have your output context, but I believe it should have been caused by it. I believe that now the warning should not occur (unless you are indeed capturing NTSC).

Just to rule out that this is the problem, the warning doesn't occur on the receiver/display, neither, correct?

If the above is really not the problem, I am afraid that I don't have currently an idea, what could be the problem. Maybe could you attach the output of sender and receiver from the beginning until it works steadily (if you can, please attach it as a file, not to flood this issue). I'll see if I won't see something suspicious. Otherwise the remote access wouldn't be so bad idea in the end.

[TheSashmo]

If you have the time. I can email you a ssh key to access. I am in front of the device all day today. You can do two ssh sessions or use screen to run the multiple instances.

[TheSashmo]

@MartinPulec I emailed you basic credentials to log in. Let me know if you have any issues.

[TheSashmo]

Additional information for you. I made slow motion recording from my phone so that I can show the missing frames. Here is the link You can see on the left is the source timecode and on the right is the output timecode on another screen, you can see at:

10:48:33:24 missing 10:48:34:15 missing 10:48:35:12 missing 10:48:36:11 missing

Additionally you can see the output is only showing the top field .1 on the timecode.

Commands I am running: ./UltraGrid-continuous-x86_64.AppImage -t decklink:0:Hi59:UYVY -c libavcodec:encoder=libx264:bitrate=18000k:subsampling=420:preset=superfast -s embedded --audio-codec=MP3:sample_rate=48000:bitrate=256k --audio-capture-format channels=16 -l unlimited -m 1316 -P 22816 192.168.99.228

./UltraGrid-continuous-x86_64.AppImage -d decklink:device=0 -r embedded 192.168.99.228 -P 22816

Both latest release and continuous have same result. Going to try older versions to see if there is any difference.

UPDATE:......... I think I found the dropping frame issue. The packet shaper -l unlimited looks like it was causing the frame drop. Now that I have that removed. It seems to be working flawless. But for some reason I can now no longer reproduce the interlaced bug! I think I need some rest.

[MartinPulec]

I think I found the dropping frame issue. The packet shaper -l unlimited looks like it was causing the frame drop. Now that I have that removed. It seems to be working flawless. But for some reason I can now no longer reproduce the interlaced bug!

Great, thanks for the info. Just an idea ­– wasn't the interlacing problem caused actually by the loss itself? Don't know how exactly does the BMD runtime handle that but I'd suppose that it can use an older field if it doesn't have a more recent one, which AFAIK may cause artifact even for progressive but more observable for interlaced.

Please let me know if you notice that without missing frames. I believe that missing frames should be also observable from receiving UltraGrid output.

[TheSashmo]

To be clear, the UG output shows perfectly fine, no lost frames. But on the output itself it would drop frames, thats why I was pulling my hair out. (Whats left of it). The issue was from -l unlimited, and the only way I found that was from the different versions of UG, don't actually support that command, so when I backtracked checking different versions I saw that lost frame issue went away. So we are finally good! At least good enough with this appliance I am trying to work with. I can do 10bit 4:2:2 encode and decode in ultrafast preset.

What I can not replicate anymore is the interlacing issue..... I've tried a bunch of different setups and for some god awful reason I am unable to do that. But I will keep testing.

Update reading back, it looks like it was vaapi causing the issue, but I need to track back and test that one again.

Thanks for the help!

[MartinPulec]

Thanks for the info. It's a shame if the terminal output looks ok but the picture doesn't.

Just for your info, there are some ideas that cross my mind. Eg. it can be due to bad timing of frames caused by the compression. But it should be AFAIK observable in the terminal on either FPS or dropped frames etc. (except of late frames warning, which has been only in verbose mode until now).

[TheSashmo]

So this issue still exists. I'm remote right now and not able to get my identical setup that I had when I first opened this, but if you look at these images. You can see the lines in the text is not as clean as the pre encode, and the other image has what looks like a line missing.

[MartinPulec]

I am sorry but I still don't know where is the problem. From first 2 pictures I cannot deduce anything. The 3rd looks obviously as interlaced (merged) picture presented as a progressive. Last one is clearly not sharp but it is unclear to me why – it can be merely compression artifacts.

It may help if you record some small sample of your input that looks wrong with --record parameter. It will produce file for every frame but you can without any problems concatenate them with cat(1) (especially if it is H.264 or HEVC).

[TheSashmo]

@MartinPulec let me try to explain...

This goes up to my original topic when opening this ticket.

For some reason interlaced video is not handled correctly on the output. Its not compression artifacts, but in fact looks like incorrect field processing. The two images, one of the circle and one with the colour bars and the white line at the top, both look like there is a field missing.

These two images are from a direct capture using the BMD capture tool in auto detect mode. I have cropped out the rest of the image so that you that I can bring your attention to the quality. These have no problems at all.

If you look at these two same images, which are POST encode and decode and using the same BMD capture tool, but looped output into the input so that you can see the direct image that I see on a reference monitor. It looks like the output is interlaced just how I would see a interlaced video on my computer. Example playing via vlc or ffplay a source that is interlaced, on a progressive computer screen. I would have to enable de-interlacing on software to be able to see this clean on my computer screen. The problem is thats now how it should look on interlaced output on SDI. From what it look like that somewhere in the encoding or in fact maybe the decoding chain, its not separating the fields and outputting them correctly. It looks like in the encoding that its creating a progressive frame "because I can see the interlacing lines".

I am remote right now, but once I return I will try to find a way to supply you samples of the record and or output.

Since I have not had the need to do that before, if you can suggest the best method for getting you source and output that would be great.

Update: you might have to put the images side by side to see the difference. The one with the colour bars only you can clearly see that they are not the same, it looks like there is a line missing at the top. I had the pleasure to meet up with alatteri and he was able to generate a test image on his setup and we could both see that it looked what seems to me like the top field was missing on the output. We didnt do a recording of it to validate if it was being dropped at the encode or the decode. I will attempt that when I get back.

[MartinPulec]

I've created an interlaced testcard pattern: uv -t testcard:pattern=interlaced:fps=60i which you can use to test just on the decoder (of course with development version of UG; I've promised it already on Friday but updated the post because I figured out that it was not fully functional).

It looks like in the encoding that its creating a progressive frame "because I can see the interlacing lines".

To be clear how interlaced is handled in UG (and also by BMD API) - almost always the odd&even fields are handled together. Eg. BMD api gives them interleaved into a single frame, UG handles that also as a single frame and if the compression supports that, it does it as well (libx264 is the notable example, maybe also some other H.264 encoders).

with respect to that:

The two images, one of the circle and one with the colour bars and the white line at the top, both look like there is a field missing.

It doesn't seem to me to be the case. Knowing that BMD captures the frames, then odd lines are from one field, even from the other. So it seems there is just a phase/time shift between those the 2 fields.

Since I have not had the need to do that before, if you can suggest the best method for getting you source and output that would be great.

uv -t decklink -c libavcodec:<opts> --record [dir], dir is optional, otherwise export.timestamp is used. Compression is almost required, otherwise you'll end up recording uncompresed video. Then you can just concatenate the frames (if using Linux): `cat export./.h264 > vid.h264

Update: you might have to put the images side by side to see the difference. The one with the colour bars only you can clearly see that they are not the same, it looks like there is a line missing at the top.`

Agree, but I don't know anything about the temporal/spatial properties of the video, neither of the recording tool (does it record both fields? how - line interleaved?). I think that it would be ideal if you recorded 1) captured input (--record works even when sending) and 2) if you have the opportunity to loop back the output of the playback to another card/input to record its input ideally also by UltraGrid. Using MediaExpress might be also OK for me, because I could check its behavior. But I think that we cannot focus just on images, because handling interlaced video is not just spatial but also temporal issue.

[TheSashmo]

Thanks for the input @MartinPulec but you have completely lost me.

The source is clearly interlaced, and the output is clearly interlaced (detected by the hardware) but the video looks like a interlaced frame encoded as progressive and output as interlaced.

I did the testcard, and it seems to me, at least, that I can see the small interlacing lines in the testcard output. I will do the test again tomorrow, and I will try to make some recordings, but from what it seems to me, there is some issue still.

I will add something very odd. I am using a Konvision reference monitor to check the output, and the source video shows 59.94 refresh rate, but the output shows as 59.88 I find this very weird. To make it even more weird, when I run the test card as the source, it shows as 59.94 correctly, even though its set to 60i as the test card.

You mentioned something about temporal/spatial properties of the video, all that I can tell you is the test tool is an industry standard for validating signals, specifically for lip ync measurements. It's called Valid or Hitomi Matchbox. Not to knock anyone, but its a standard tool used all around the world.

[MartinPulec]

Basically, the problem is that I cannot reproduce the issue neither have a clue where it can lie. It needs to be determined if capture/compress/display is to blame. That's why I've advised to record the input and/or use the interlaced pattern.

I did the testcard, and it seems to me, at least, that I can see the small interlacing lines in the testcard output.

If the display is able to display both progressive and interlaced, the following 2 commands should look recognizably different:

uv -t testcard:pattern=interlaced:fps=59.94i -d decklink
uv -t testcard:pattern=interlaced:fps=29.97p -d decklink

The source is clearly interlaced, and the output is clearly interlaced (detected by the hardware) but the video looks like a interlaced frame encoded as progressive and output as interlaced.

The thing is that even this approach isn't necessarily bad. If the picture given to compression is interlaced merged (line-interleaved pairs of fields) and _is_notdeinterelaced (which UG never does automatically) and then decompressed and presented as interlaced, it works quite fine. We use this with GPUJPEG, which doesn't obviously support interlacing, and it works correctly. The interlacing there has just an impact on compression efficiency (not so much on JPEG, I think that much worse on HEVC). You can also test this with the above commands – if interlaced is displayed correctly, you can add eg. -c libavcodec:encoder=mjpeg, which should look approximately the same, even though the compression is interlacing-agnostic.

[TheSashmo]

Alright! I have updates! I can confirm 100% the following, this issue only exists on continuo build dated on April 14th (15th in the archive), but no longer present on April 28th (29th in the archive).

I can also confirm that the issue is in the decoder, not the encoder, I used latest continuous and don't see the same problem. But I can see that the problem exists in the latest release!

So problem is solved, I just don't know the differences between builds. Update that on both versions can result two different outputs. One output where the fields are incorrect and others that the field is accurate. It's honestly hit and miss based on each restart of the process.

[MartinPulec]

Thanks a lot for tracking this down!

It was fixed with the commit 4bd4ed2, the affected feature was actually bmdDeckLinkConfigFieldFlickerRemoval not being disabled for low-latency (not scheduled) playback. I believe that the bug was present since 2022-05-11 (commit 396c445d), so it is still present in 1.8.x releases. Although I don't know if it is reproducible everywhere (it depends on that flag, that may be also switched off via preferences outside UG). The above-mentioned feature isn't probably intended for non-scheduled playback.

[MartinPulec]

The fix should be already included also in UltraGrid 1.8.3.