Regressed deinterlacing support as compared to i965-va-driver

[mamarley]

I have tried out media-driver 19.3.x on a few systems with Broadwell and Skylake CPUs (Specifically, HD Graphics 5500 and Iris 540) (reproduced with Linux 5.3, 5.4, and 5.5-rc) and have found that the hardware deinterlacing support doesn't work as well compared to the i965-va-driver. Here's what I have found:

1. If ENABLE_NONFREE_KERNELS=OFF is set, double-frame-rate deinterlacing doesn't work at all, resulting in 1080i30 content created from a 60FPS source playing jerkily at 30FPS instead of 60FPS as it did with the i965-va-driver. (The previous driver supported double-frame-rate even when the non-free features were disabled.) This is important for many news and sports broadcasts on 1080i ATSC channels.
2. If ENABLE_NONFREE_KERNELS=ON is set, double-frame-rate deinterlacing works only sometimes and with reduced quality compared to i965-va-driver. In VLC, the frame rate is always doubled, but the video seems subjectively fuzzier than with i965-va-driver and horizontal lines seem to "shimmer" up and down. In Kodi, it seems to alternate between no frame rate doubling (resulting in 30FPS) and the degraded frame rate doubling as I saw in VLC every few seconds at random intervals.

I apologize for not including any more information with this report, but there doesn't seem to be anything relevant in the logging output from either application and I'm not sure what else would be helpful. I will certainly provide any requested information though. Thanks!

[XinfengZhang]

thanks reporting, one question: are you using Bob DI or some advance DI?

[mamarley]

VLC doesn't seem to give me an option, but with Kodi I have tried both Bob and Motion-Adaptive and I see the same results with both.

[XinfengZhang]

@FurongZhang , @Jister could you help to check it

[yasij]

I noticed the same issue. I just tested 19.4.0r on Coffee Lake with ENABLE_NONFREE_KERNELS=ON . Motion Adaptive deinterlace looks exactly the same as Bob in Kodi. I notice the flickering line at the top of the screen. Motion-Compensated deinterlacing is not available. i965-va-driver presents Bob, Motion-Adaptive and Motion-Compensated. Motion-Adaptive and Motion-Compensated look much better, and don't have he characteristic Bob artifacts that intel-media-driver does.

[Jister]

Hi, @mamarley @yasij , could you please provide your test clip and reproducing steps so that we can reproduce the issue

[mamarley]

I have an example file, but it is too large to upload here. Is there any place you would prefer me to put it?

[yasij]

Here's a clip from a recording that shows the issue. I just took the first 6 MiB of the TS file: sampledeint.ts.gz

Reproduction steps: gunzip sampledeint.ts.gz env LIBVA_DRIVER_NAME=iHD mpv --fs --hwdec=vaapi --vf=vavpp:deint=motion-adaptive sampledeint.ts

Look at the text. Notice that it bobs up and down like bob deinterlace Now play it with the i965 driver: mpv --fs --hwdec=vaapi --vf=vavpp:deint=motion-adaptive sampledeint.ts The text doesn't not bob with i965.

i965 with the VAAPI bob deinterlacer mpv --fs --hwdec=vaapi --vf=vavpp:deint=bob sampledeint.ts The text bobs like iHD with the motion-adaptive filter.

The hardware is Intel Core i7-8700K UHD Graphics 630.

[yasij]

This might be the same issue as #640 . However, that one mentions that the reference frames aren't passed in VAProcPipelineParameterBuffer. I checked mpv, kodi and gstreamer. All of them are passing reference frames in VAProcPipelineParameterBuffer.

[yasij]

media_libva.h#L82

The driver defines DDI_CODEC_NUM_BK_REF and DDI_CODEC_NUM_FWD_REF as 0. That's why the reference frames aren't passed. The driver reports that max number of FWD and BK references as 0 in DdiMedia_QueryVideoProcPipelineCaps.

[Jister]

Hi, @yasij , I add the correct reference number in #820 , could you help to try the patch? Thanks

[mamarley]

This doesn't seem to have any effect for me. I still get 60fps with shimmering in VLC and alternating between 30fps and 60fps with no option for motion-compensated deinterlacing in Kodi.

[Jister]

Hi, @mamarley , I update #820 , could you help to have a try again? Thanks.

[mamarley]

With that patch both VLC and Kodi crash with a segmentation fault as soon as the video starts playing. That is probably because I am using 19.4 and not git master, but I'm not in a position to be able to try git master easily right now. Sorry. Perhaps @yasij can?

[yasij]

It'll be a few days before I can find time to test it, but I'll try it.

[yasij]

I just tried #820 on top of git master. motion-adaptive still has bad bob artifacts on the sample I submitted, and motion-compensated is not presented as an available option in kodi with iHD (but is with intel-vaapi-driver).

[yasij]

I'll check this again tonight. I might have forgotten to git commit after git apply of that patch before generating the archive to build the package.

[yasij]

I just tried it with #820 actually applied this time. It crashes at media_driver/linux/common/vp/ddi/media_libva_vp.c:2212 because to pSrc->pDeinterlaceParams is NULL. Did you test this before sending it out?

[yasij]

I fixed the crash by moving the allocate for pSrc->pDeinterlaceParams before the switch: 0001-Allocate-pDeinterlaceParams-before-using-it.patch.txt

Now both MADI and MCDI are available in Kodi. The text bobbing is gone, and some content looks better. However, the sample I posted above plays back extremely choppy in mpv and kodi with iHD. I will note this particular clip is a PAFF video recorded from TV. Can the iHD deinterlacer handle PAFF properly?

[Jister]

@yasij Thanks for your debugging. For PAFF video, can i965 driver playback normally? What parameters will app pass to driver for such video?

[yasij]

Yes, the i965 driver plays it back normally. There was a discussion about PAFF in Kodi a while ago when it was not playing correctly. See: https://github.com/xbmc/xbmc/issues/15817 and the code change: https://github.com/xbmc/xbmc/pull/15830 Before that change, kodi was toggling the deinterlace method flag to none and back everytime it encountered a progressive frame in PAFF. This caused stuttering with the i965 driver. From that bug report: "it's a PAFF video. mpv by the way copes with this by deinterlacing every frame no matter whether it is marked as interlaced (in default settings) :-)" They keep the post processing filter on with the deinterlace setting, and send the progressive frames or fields as they come from the PAFF video. Kodi now also does this (and previously did this in v17). This works with the i965 driver.

[yasij]

Is there an update on the status of this? #820 does allow MADI and MCDI to work, so I think it should be merged. A follow up issue can be created to fix the problems with PAFF.

[XinfengZhang]

820 will be merged after some test.

[mamarley]

Any progress on getting this merged?

[Jister]

WIP, under internal test, will merge once test done.

[smp79]

I tried #820 with https://github.com/intel/media-driver/files/4075058/0001-Allocate-pDeinterlaceParams-before-using-it.patch.txt but MADI and MCDI still don't appear in Kodi. Tested with Kodi 19 current master.

[yasij]

@smp79 #820 was updated to fix the memory issue. The 2nd patch isn't needed anymore. With #820 I get MADI and MCDI with iHD in Kodi 18.6 with a Kaby Lake and a Coffee Lake system.

[smp79]

I don't see that #820 was updated to fix the memory issue. I had to use that extra patch, otherwise it crashed. I verified with vainfo that iHD driver is used but no MADI and MCDI options appear. Tested on Gemini Lake J4105.

[yasij]

Link to the raw patch. This does not have the memory issue. It uses a temporary bool, and then sets is after the memory is allocated later. Jister updated that on Jan 17.

[smp79]

Link to the raw patch.

I just tested it, it does not crash but still no MADI and MCDI. Is this patch supposed to fix the issue on all platforms? Definitely does nothing on GLK.

# vainfo
libva info: VA-API version 1.7.0
libva info: Trying to open /usr/lib/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_7
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.7 (libva 2.7.1)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 20.2.pre (38764c846e)

[smp79]

What's the status of this issue? I still don't see the deinterlacing options in Kodi with iHD driver on GLK.

[XinfengZhang]

it should work on all platforms , for GLK, you also need #1005 , hi @Jister when the #820 could be merged?

[smp79]

it should work on all platforms , for GLK, you also need #1005 ,

I just re-tested, It does not work on GLK - MADI or MCDI options do not appear in Kodi. Tested on LibreELEC with current Kodi master. I applied both #820 and #1005 I used media-driver v.20.1.1 because #820 patch applies more or less cleanly on it. Built with -DBUILD_CMRTLIB=OFF -DENABLE_NONFREE_KERNELS=OFF -DBUILD_KERNELS=OFF

[smp79]

I just built with -DENABLE_NONFREE_KERNELS=ON and MADI or MCDI options did appear. However, when MADI or MCDI is enabled the video is extremely jerky. This is when I play an interlaced 1080i50 stream: https://www.dropbox.com/s/st262j6ikkgv1cg/VID_20201025_212840.mp4?dl=1

[FurongZhang]

Need @Jister 's attention.

[heitbaum]

Still seeing this VAAPI error on Matrix master “devel-20201030100043-7f3b20d (Generic.x86_64)” on i5-6260U Skylake CPU with ISO Mpeg2 / TV headend Mpeg2 playback.

cpu details

grep -i intel kodi.log 2020-10-31 15:56:50.438 T:1138 INFO : Host CPU: Intel(R) Core(TM) i5-6260U CPU @ 1.80GHz, 4 cores available 2020-10-31 15:56:51.418 T:1179 INFO : m_displayName : Default (HDA Intel PCH ALC283 Analog) 2020-10-31 15:56:52.506 T:1138 INFO : GL_VENDOR = Intel 2020-10-31 15:56:52.506 T:1138 INFO : GL_RENDERER = Mesa Intel(R) Iris(R) Graphics 540 (SKL GT3)

LibreELEC:~/.kodi/temp # dmesg | grep -i intel BIOS vendor: Intel Corp.; Ver: SYSKLi35.86A.0073.2020.0909.1625; Product Version: smpboot: CPU0: Intel(R) Core(TM) i5-6260U CPU @ 1.80GHz (family: 0x6, model: 0x4e, stepping: 0x3) Performance Events: PEBS fmt3+, Skylake events, 32-deep LBR, full-width counters, Intel PMU driver.

Errors in the logs - just like in https://github.com/xbmc/xbmc/issues/15817

2020-10-31 18:07:16.734 T:19806 ERROR : VAAPI/vpp vaCreateConfig error: the requested VAProfile is not supported (12) 2020-10-31 18:07:16.734 T:19806 ERROR : VAAPI output: Postproc preinit failed

vainfo

libva info: VA-API version 1.9.0 libva info: Trying to open /usr/lib/dri/iHD_drv_video.so libva info: Found init function __vaDriverInit_1_9 libva info: va_openDriver() returns 0 vainfo: VA-API version: 1.9 (libva 2.9.0) vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 20.3.0 (7f3b20d31d) vainfo: Supported profile and entrypoints VAProfileMPEG2Simple : VAEntrypointVLD VAProfileMPEG2Main : VAEntrypointVLD VAProfileH264Main : VAEntrypointVLD VAProfileH264Main : VAEntrypointEncSliceLP VAProfileH264High : VAEntrypointVLD VAProfileH264High : VAEntrypointEncSliceLP VAProfileJPEGBaseline : VAEntrypointVLD VAProfileJPEGBaseline : VAEntrypointEncPicture VAProfileH264ConstrainedBaseline: VAEntrypointVLD VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP VAProfileVP8Version0_3 : VAEntrypointVLD VAProfileHEVCMain : VAEntrypointVLD

[zgb124]

Has there been any movement on this issue? It is my understanding that the i965-va-driver does not work with Tiger Lake, so deinterlacing is effectively broken until this is fixed.

[smp79]

Has there been any movement on this issue?

No one at Intel seem to give a shit. At least on GLK we can use a legacy i965 driver to get a working MCDI. No such luck with TGL.

[XinfengZhang]

it should work on all platforms , for GLK, you also need #1005 ,

I just re-tested, It does not work on GLK - MADI or MCDI options do not appear in Kodi. Tested on LibreELEC with current Kodi master. I applied both #820 and #1005 I used media-driver v.20.1.1 because #820 patch applies more or less cleanly on it. Built with -DBUILD_CMRTLIB=OFF -DENABLE_NONFREE_KERNELS=OFF -DBUILD_KERNELS=OFF

hi @smp79 why the BUILD_KERNELS=OFF? , it means there are no kernel(shaders) support, it will impact VPP capability . I suggest all of these should set to be ON

[XinfengZhang]

@Jister , why we still not merge #820? if the fix is not suitable, is there a better one?

[yasij]

As I commented on #640 the reason I think MADI/MCDI was jerky for me with my video and this patch is because this driver reverses forward and backwards frames compared to the i965-va-driver and the client implementations (including FFMpeg). I found this discussion on the VLC mailing list which states that ""forward_refs" are actually the frames older than the current one, and the "backward_refs" are those later". That might explain why i965-va-driver is setting num_forward_references = 1, and num_backward_references = 0.

I think this driver needs to net num_forward_references = 1, and reverse the frame order logic to match all the client implementations and the previous driver.

[XinfengZhang]

@yasij , https://github.com/intel/libva/pull/426 , @Jister should already change the implementation following the PR

[Jister]

@XinfengZhang We need to change the behavior of media-driver, since there are some other application/users regarding backwork_refs as older frames, if we change such behavior directly, there will be some other compatibility issue.

[yasij]

@yasij , intel/libva#426 , @Jister should already change the implementation following the PR

Looks like e6e7f90d reversed it, but that was reverted with eabb99f5.

The i965-va-driver regards forward_refs as older frames, and this driver has always advertised 0 max_forward_references and 0 max_backward_references so madi/mcdi don't actually work for them. On the driver where they do work, those clients are getting the wrong behavior. Why not match what the existing legacy driver is doing? libva now defines forward_refs as past frames in the header.

For the many clients based on ffmpeg that do regard forward_refs as older frames, you can't do what you currently have. How are you proposing to resolve this? Currently, this driver effectively only supports bob deinterlacing because that's what madi and mcdi give when no backward_ref or forward_ref are provided.

[smp79]

why the BUILD_KERNELS=OFF

I use LibreELEC and the current default for LibreELEC is BUILD_KERNELS=OFF. This PR will change it to BUILD_KERNELS=ON, probably as soon as deinterlacing issues are sorted out.

[Jister]

Please try #1162 . In the meantime, I will check other application/framework's behavior, after all dependencies are clean, then we can merge the change.

[smp79]

Please try #1162

MCDI now works! Tested on GLK with Kodi. Thank you!

[yasij]

Just tested #1162 on KBL with Kodi. MCDI is working well. I tested the PAFF recordings and other 1080i recordings and no issues.

[zgb124]

Very pleased to see there is a fix. What are the chances it will make it into the next release?

[yasij]

Looks like this still hasn't been merged. What's the status of your investigation @Jister ? This driver still does not have functional advanced deinterlacing without this. I'm still puzzled by the comments about backwards compatibility. Both the legacy i965 VAAPI driver and the Mesa VAAPI drivers regard forward_refs as older frames, and this driver has always set the max_forward_references and max_backward_references to 0 which means it has never done advanced DI. If anything, the current definition of forward_refs and backwards_refs in this driver breaks backward compatibility, and needs to be brought in line with the spec and previous drivers. Any userspace software that has it backwards violates the spec, didn't work properly with the other drivers, and can fallback to bob deinterlacing without an issue and have the same behavior as what this current driver does.

[MicroYY]

I'll help check-in @Jister's change.