search

intel / media-driver

Intel Graphics Media Driver to support hardware decode, encode and video processing.
https://github.com/intel/media-driver/wiki
Other
1k stars 347 forks source link

[Bug]: FFmpeg QSV VP9 oneVPL Encodes Produce Faulty Video That Can't Be Played Back Without Breaking/Corrupting Video Stream #1734

Closed warduweram closed 7 months ago

warduweram commented 1 year ago

Which component impacted?

Encode

Is it regression? Good in old configuration?

No, this issue exist a long time

What happened?

This issue was discussed in https://github.com/intel/media-driver/issues/1576 but is now closed since it seems to be fixed for HEVC.

It should be reproducible across multiple platforms (I'm using it on Windows/latest FFmpeg 6 release/i5-13600K), but here is the prompt (among many others I've tried) I've used: ffmpeg -hwaccel_output_format qsv -qsv_device 1 -i "input.mp4" -c:v vp9_qsv -g:v 60 -preset 4 -b:v 50000k -minrate 40000k -maxrate 60000k -profile:v 1 "output.mp4"

Here is my description from the above thread:

There are issues with moving the playhead while playing back the video, this will break the video stream. Sometimes I get artifacts with VP9, sometimes the playback just breaks until I restart the file (hevc seems fine now with the new addition from Intel, but I wouldn't be surprised if there was still a bug here and there, which time will show).

So, VP9 is still unusable and I would emphatically like to ask for proper support for it, since it's a great codec, especially now with the high power efficiency and speed on the newer Intel iGPUs and Arcs. It has a far wider support across different devices than AV1 is going to have in the coming years, also better than hevc. I would also argue that it's better for archiving purposes all around than hevc. I'd very much appreciate if Intel and/or the ffmpeg devs could have another look at this.

What's the usage scenario when you are seeing the problem?

Transcode for media delivery, Playback

What impacted?

The advantages of hardware-encoding hi-quality video environment-friendly efficiently and with low power usage across most platforms/devices are not given in the current state.

Debug Information

No response

Do you want to contribute a patch to fix the issue?

No.

eandrade92630 commented 1 year ago

Not sure if this is the same issue I'm having but I fixed this by encoding only with FFmpeg then remuxing the files with mkvmerge. There's something wrong with the muxing process in FFmpeg.

intel-mediadev commented 1 year ago

Auto Created VSMGWL-69499 for further analysis.

warduweram commented 10 months ago

@leyu-yao @XinfengZhang Has there been any progress on this issue?

Sherry-Lin commented 10 months ago

@xhaihao do you see any issues for ffmpeg-qsv VP9?

xhaihao commented 10 months ago

@Sherry-Lin Yes, I see this issue. The root cause is that VPL runtime doesn't set keyframe flag on returned bitstream. @warduweram Could you try this WA (https://github.com/intel-media-ci/ffmpeg/pull/701) in FFmpeg ?

warduweram commented 10 months ago

@xhaihao I can try, sure, but I don't know how/where to apply that commit, so if you could give me a hand. :D

xhaihao commented 10 months ago

@warduweram you may use the branch directly. $ git clone https://github.com/xhaihao/ffmpeg-1.git -b wa-qsvenc-vp9

warduweram commented 10 months ago

@xhaihao Thanks for the effort, I appreciate it! I understand correctly that I have to build ffmpeg from that cloned repository, right? It might take me a while, since I haven't built something as complex as ffmpeg before. I might have to boot to Linux or figure out how to do it under Windows. So if there's anyone else with the necessary skills and hardware, feel free to try it out before me.

jnko commented 10 months ago

I can confirm the fix is working with QSV VP9. Thanks a lot!

warduweram commented 10 months ago

@jnko Awesome, thanks for trying it out, I couldn't manage to compile under Windows.

@Sherry-Lin How do we continue from here, do we communicate this to the ffmpeg team or do you still wanna fix it in the drivers?

xhaihao commented 9 months ago

@warduweram The patch was submitted to FFmpeg ML, see https://patchwork.ffmpeg.org/project/ffmpeg/patch/20240126042822.111719-1-haihao.xiang@intel.com/

warduweram commented 9 months ago

@xhaihao Awesome, thanks so much!

leyu-yao commented 8 months ago

With this commit https://github.com/oneapi-src/oneVPL-intel-gpu/commit/89edcbacb69b94bc4272598837718ffda523f380 VPL will set IDR flag (keyframe flag) for VP9 I frames.

warduweram commented 8 months ago

@leyu-yao Thanks so much for the lengthy explanation, I appreciate it! Will the fix work for RaptorLake iGPU?

leyu-yao wrote (on other thread):

I checked windows release driver on . Unfortunately, this is fix is not included yet. For VPL and Media sdk, they are two runtime in graphics driver. On some new platforms, VPL is used. And media sdk is used on old platforms and it is not maintained anymore. If you are using recent Intel platforms such as DG2(Arc Axxx) or MTL(Core Ultra xxx), these platforms will use the latest runtime. And this fix will be included the windows release driver in a few weeks. Once you install the driver which includes this fix, a stock ffmpeg build is good to use. However, if you are using some older platforms, this fix won't benefit the windows release driver and you still need a build ffmpeg with Haihao's patch https://patchwork.ffmpeg.org/project/ffmpeg/patch/20240126042822.111719-1-haihao.xiang@intel.com/ I'm not sure when his ffmpeg patch will be included in some ffmpeg pre-build release. May need comment from Haihao.

leyu-yao commented 8 months ago

@leyu-yao Thanks so much for the lengthy explanation, I appreciate it! Will the fix work for RaptorLake iGPU?

leyu-yao wrote (on other thread):

I checked windows release driver on . Unfortunately, this is fix is not included yet. For VPL and Media sdk, they are two runtime in graphics driver. On some new platforms, VPL is used. And media sdk is used on old platforms and it is not maintained anymore. If you are using recent Intel platforms such as DG2(Arc Axxx) or MTL(Core Ultra xxx), these platforms will use the latest runtime. And this fix will be included the windows release driver in a few weeks. Once you install the driver which includes this fix, a stock ffmpeg build is good to use. However, if you are using some older platforms, this fix won't benefit the windows release driver and you still need a build ffmpeg with Haihao's patch https://patchwork.ffmpeg.org/project/ffmpeg/patch/20240126042822.111719-1-haihao.xiang@intel.com/ I'm not sure when his ffmpeg patch will be included in some ffmpeg pre-build release. May need comment from Haihao.

Hi @warduweram As I remember, both VPL and MSDK RT are supported on RaptorLake iGPU. But sorry I'm not very familiar with ffmpeg. Hi @xhaihao, sorry could you help to comment on this? Which RT will ffmpeg qsv plugin load on RaptorLake?

warduweram commented 7 months ago

This seems to be fixed on 5382 WHQL Intel Arc/Iris drivers with ffmpeg 7.0 on RaptorLake if the right bitrate settings for ffmpeg are chosen, I'll close the issue. Thanks again for all the efforts involved.