Can this library play AV1 in webm

[AlacranNeron]

Hello, I'm trying to play some sample AV1 files downloaded from https://www.elecard.com/videos (3840x2160 | AV1 | 13.9 mbps | 4-core Intel i7 2.5 GHz to be exact), but the MediaPlayerCS fails with HRESULT E_FAIL from COM component. The same file plays ok in the Movies and TV app. Is there some config option that I'm missing or AV1 is not supported.

[brabebhin]

Hi @lukasf

It seems out build of ffmpeg that we distribute through nuget does not include AV1 video codec.

[lukasf]

Interesting. It looks like FFmpeg supports AV1 only through external libs (peferrably dav1d). Theoretically, we could try to include it into the build. But there is one problem: The FFmpeg AV1 decoder currently does not support hardware acceleration. So it will not make use of HW AV1 decoders on latest CPUs/GPUs. You would need a very fast CPU to play AV1. It might not be what you expect.

[lukasf]

I just saw that FFmpeg 4.4 is out. They added D3D11 support for AV1, so moving to FFmpeg 4.4 would allow us to use HW accelerated AV1 decoding. From a firt look at the code, it looks like this will work even without adding new libs, but this needs to be verified.

[brabebhin]

I will try to build it and see how it goes.

[brabebhin]

Well I might have been doing something wrong or simply building ffmpeg 4.4 is not enough.

[lukasf]

Do you have hardware acceleration for AV1? In the release notes, I read something about "hw acceleration only". It might be that SW decoding still requires the dav1d library.

[brabebhin]

100% did something wrong as the ffmpeg version reported by API is still 4.3.1

[lukasf]

Just building FFmpeg is not enough. You need to create a NuGet packet and then use that local package from your app (or sample app). Maybe I should create some instructions on that. It's actually pretty easy, but you need to know the right buttons.

[brabebhin]

Yea, I figured. Brings back interesting memories with debugging packages ^^. so I checked out 4.4.7, built it (first problem here), then hacked the local packages cached to inject my dlls.

So the ARM64 build seems broken. Not sure if this is Microsoft problem or ffmpeg problem.

There's also some properties that are no longer accessible in the AVStream class (start_skip_samples and nb_decoded_frames), which seem to break our code.

So the codec is now found indeed, but I don't think my GPU supports AV1 format. I auto pick FFmpegSoftwareDecoder from the hardware acceleration setup.

FFmpeg (Error): [av1 @ 00000243B0D41380] video_get_buffer: image parameters invalid
FFmpeg (Error): [av1 @ 00000243B0D41380] get_buffer() failed
FFmpeg (Error): [av1 @ 00000243B0D41380] thread_get_buffer() failed
FFmpeg (Error): [av1 @ 00000243B0D41380] Failed to allocate space for current frame.
FFmpeg (Error): [av1 @ 00000243B0D41380] Get current frame error

Doing some more research, it seems there is very little hardware available which supports it.

I also tried opening that file with VLC - it is smooth, but it puts the CPU at 44% usages and no load on the GPU video decoder. So I guess that confirms it.

[brabebhin]

Ironically, ARM64 CPUs are more likely to support this than intel/nvidia. And I'm not gonna cash out 4k on a new gpu😂. If you have newer hardware it might work for you.

[lukasf]

Actually I was lucky to get my hands on a RTX 3070, when the prices were not totally out of place (I wanted a 3080, but no chance on that one). So I should be able to test this out.

[brabebhin]

I hope my name change didn't confuse you too much. I never liked the 222 at the end.

[lukasf]

Oh I only noticed now! Thanks for pointing out ^^

[brabebhin]

I will also try it on my rtx2070 machine but I don't have a dev environment there, for building ffmpeg.

From what I understand, hardware from 2019-2020 is unlikely to support it.

[lukasf]

I only see n4.4. Are you sure you checked out the right tag? Trying to build now...

[brabebhin]

Yes, I didn't push my local branch changes. If you check out the submodule and fetch the ffmpeg remote, you can check out the release/4.4 branch. My current commit was from 17 days ago.

I use git extensions for handling the source control, it gives decent control and a useful UI, unlike VS, source tree or other experiments flying around.

[lukasf]

Checking out branches is dangerous. You will get work-in-progress stuff, like work for next release or subrelease. For ffmpeg, only ever work with tags. So for checking out ffmpeg 4.4, use "git checkout n4.4".

[lukasf]

Updating gas-preprocessor fixed the ARM64 build. I uploaded a 4.4.0-pre NuGet package. Indeed they moded some AVStream properties to internal classes. Not sure why. Maybe they take care of encoder padding themselves now. We need to verify this. AV1 playback fails for me as well on first attempt. Will try more tomorrow. Branch ffmpeg-4.4 is online.

[brabebhin]

According to nVidia, 3000 series should support AV1. Are you getting the same errors in the console output?

[lukasf]

Huh strange. Console prints this as first error: "Your platform doesn't suppport hardware accelerated AV1 decoding." (later followed by what you posted)

But the GPU does support AV1 and DXVAChecker confirms this. Not sure why it is not recognized correctly. I only have one GPU in the system.

[brabebhin]

Interesting. Maybe you need the AV1 extension or a specific windows 10 version, 1909?

[lukasf]

I am on latest Windows version. Extensions are not required since ffmpeg directly utilizes D3D11 APIs, they are only needed when using passthrough approach.

[brabebhin]

True, but ffmpeg interacts with the driver in order to do its magic, so if the driver reports the format is not supported, ffmpeg will not know better.

It might still need the external library to demux the packages before sending them to the driver.

[lukasf]

The driver corretly reports DXVA status. DVXA checker shows it, and also MPC-BE uses DXVA without issues. I am already one step further: Ther problem seems to be that the current Windows SDK does not contain the required headers for AV1. The ffmpeg configure script checks for existence of DXVA_PicParams_AV1, and they are not defined (HEVC and others are there in dxva.h). It might be that we need to build against an insider SDK, or get the AV1 headers somewhere else.

[brabebhin]

Uh oh. One safe option would be installing insider in a VM and take the headers out.

[lukasf]

I asked the dev who commited the changeset to ffmpeg. Indeed the only way to get hold of the header is to install the latest Insider SDK on a Windows 10 Insider installation and grab the header from there. They really could have made that easier. It works on official builds as well, not only insider. Setting up a VM right now to get this done.

[lukasf]

Yes!! It actually works. I got HW accelerated AV1 decoding on my RTX 3070 now. Just uploading 4.4.0-pre3 package which has HW acceleration enabled.

For SW decoding, we'd still need to include dav1d. I tried to build it, but I had no success until now. It is so frustrating that every open source lib seems to use a completely different build system. dav1d uses "meson" and I could not get it to work with our MSVC on MSYS2 setup. There is a Visual Studio version of meson, but I really don't want to blow up our requisites once again. I have not tried that. Having it as a pacman command in MSYS2 would be so much easier, if we can only get it to use the MSVC toolchain.

[brabebhin]

Can't we build it once using VS and then link it in? I will also take a look at this if I have the time in the weekend.

Nice work on getting the hardware decoding working.

[lukasf]

Nah, I don't think that's a good idea. I don't want to involve manual steps into our build process. And AV1 decoders are heavily evolving. For them is is especially important to be able to pick up new versions easily as they become available.

[lukasf]

Cool, with the help of some guys from dav1d IRC channel, I could get dav1d compiled on MSYS2 (at least no errors shown). Still some way to go integrating this into our build system and linking with ffmpeg...

[brabebhin]

I'm sorry, I wanted to fiddle around with an msbuild build but didn't get the chance. Nice work on integrating it with MSYS2. I will probably be busy the next 2 to 3 weeks, give or take.

[lukasf]

Oh man this is so weird. Basically, I can compile with dav1d now. But the problem is this: Build with DXVA (AV1), no problems. Build with dav1d (SW only, no DXVA), no problem. Build with both DXVA and dav1d: All platforms build fine, except for x86. On x86, ffmpeg build will hang forever on link.exe. After this occurs once, you will never be able to build ffmpeg x86 again!! I tried to disable dav1d, disable DXVA, roll back to master branch, use differenc VC version, different toolset, delete all temp files. Nothing helps. Only using system restore helped me get compilation working again. But only as long as I don't combine DXVA and dav1d, then the problem is there again. I did this a few times experimenting, then suddenly the clean restore point was away and now my main build machine seems to be lost. Then I tried on my laptop: Same problem there!! Build was fine until I used both x86 and DXVA at the same time, now it also won't build x86 at all. Gaaaah!!

I remember that I had a similar problem maybe a year ago. There it was x64 that refused to build. Even uninstalling VS and reinstalling did not help. I had to reinstall the OS to get things working again (but after that, it never occured again, until now).

[brabebhin]

It might be because you are trying to compile from an x64 machine, and there is a bug in the msbuild cross compiler. Maybe a virtual machine emulating an 32 bit OS would help here.

[brabebhin]

Sadly I didn't get the chance to do anything useful on getting the build running. :(

[brabebhin]

I will likely be busy with some other projects until the end of the year.

[lukasf]

Good luck with your other projects @brabebhin!

I have been thinking, the one thing that could solve all these weird build problems would be: creating a docker file that cleanly installs all required dependencies and then run all ffmpeg builds inside the container. No more dependencies on which VS version you have locally installed, no more discussions on which items you need to check in VS Installer, no more VS updates breaking your builds. And if some build bug (like the one occured to me above) breaks your container, you just recreate it. This would be incredibly helpful.

If I had more free time, this would be somthing I would investigate. Unfortunatly, I don't have much free time right now, and I also don't have much knowledge of docker. Still, maybe I will try to play with this docker idea some day.

[lukasf]

Although they recently started charging money for it (from bigger companies at least), Docker for Windows is in a pretty poor state as it seems. If you try to install MSYS2 into a docker image, docker will freeze during creation of the docker file. This is easily reproducible and an issue is open since a year or two, but no activity. Such a shame. It could have been really useful for us.

[brabebhin]

Strange, we use it at work, but only for fully managed applications, and I haven't heard any complaints.

[lukasf]

We also use it at work, mainly for .NET Core applications. I think for many scenarios it works pretty well. But I find it a bit disturbing that such a major issue with a very well known tool such as MSYS2 is open for so long without any activity. The same kind of hang/freeze seems to happen with a few other tools as well, and it is super easy to reproduce...

[brabebhin]

This is the main reason why I avoid getting too excited about all the new technologies that come (and go) out of nowhere. Probably docker does not properly emulate the low level commands that are necessary for MSYS2, most likely because docker is designed to deal with web apps.

Still, a shame it has so poor support for dev tools.

[lukasf]

AV1 support is available since merge of #244, both SW and HW decoding. Closing this.