search

CasparCG / server

CasparCG Server is a Windows and Linux software used to play out professional graphics, audio and video to multiple outputs. It has been in 24/7 broadcast production since 2006. Ready-to-use downloads are available under the Releases tab https://casparcg.com.
GNU General Public License v3.0
902 stars 269 forks source link

Option to Enable Nvidia Hardware Encoders in FFmpeg Consumers for Linux #1282

Closed LRTNZ closed 4 years ago

LRTNZ commented 4 years ago

Description

Ideally, to allow Linux users to more simply enable GPU based encoding in FFMpeg Consumers, a simple method should be added, by which a user can build the Caspar Server, to allow them to use nvenc_h264 and the like to perform the encoding of their streams. This would also help to soften the blow of potentially having to purchase a dedicated Nvidia GPU, as it would at least be able to be more easily used for more than just allowing the software to run/being used by the mixer - if one was present.

Solution suggestion

Maybe a simple true/false flag in a config file that the user can change, that will allow them to build the Caspar server with the GPU hardware encoder option enabled?

Additional information

I am using Ubuntu Server 18.04, and have been looking into how I can get the FFMpeg build included in caspar server to use GPU encoding, and have not been able to work out how to, let alone how I would make a pull request to enable that.

walterav1984 commented 4 years ago

Related forumpost: https://casparcgforum.org/t/enable-nvenc-h264-encoding-in-linux/3014

LRTNZ commented 4 years ago

Yes I made that forum post just now, however I was also thinking maybe that it is an option that should be looked into for compile time, especially with the increasing prevalence of GPUs with this functionality inbuilt in hardware.

walterav1984 commented 4 years ago

UPDATE: It seems to be working in ubuntu-mate 20.04 for FILE recording with video and audio just a simple test using h264 444 (h265 nor STREAMING not tested yet), more instructions will follow!

Building support doesn't look that hard, I might give it a try since I have access to a geforce 2060 super and have done the following steps for ubuntu 20.04 (live USB works if enough RAM):

sudo apt install nvidia-cuda-toolkit nvidia-modprobe

However if build support for ffmpeg with "--enable-nvenc " is in place for CasparCG what would your ffmpeg argument line look like that will satisfy your(maybe other needs) as in using h264/hevc and 420/422 or even 444 sampling / bitrate / profile / preset parameters?

Having an exact argument which leads to a mp4 that is well tested for compatibility with all kind of NLE edit software AdobePremiere/FinalCutX/Edius/Davinci Resolve/Vegas or hardware players could speed up the test process of this feature. Think of examples for multiple broadcast dimensions and framerates.

Did you already have success for steps 1, 2a, 2b, 3 and 6?

LRTNZ commented 4 years ago

I am currently wanting to perform: -format mpegts -nvenc_h264 -x264opts keyint=25 filter:v format=yuv420p (I think it is that, not in front of my PC to SSH into my server to check).

1 - No, have been trying to work out what is needed to build for Ubuntu, with nvenc. 2a - See 1. 2b - See 1 3 - Yes 4 - Yes, partially. Have not built dependencies seperately, but have built outside of docker. 6 - No, have not. What would be required for that?

I will also add, that I checked the apt-get version of ffmpeg I installed, and it has the nvenc and the like encoders listed under ffmpeg -encoders.

(Sorry, it's 3am, on my phone. Fried my brain trying to work this all out today.)

mint-dewit commented 4 years ago

I don't think any code changes need to be done for this, the stream consumer should pass the parameters already. So long as the included ffmpeg was configured to do this (which is fairly straight forward for linux, less so for windows), it should work.

Here is an example dockerfile on compiling NVENC and ffmpeg: https://github.com/clerk67/docker-nvenc/blob/ubuntu18.04/8.2/ffmpeg4.1/Dockerfile

I will also add, that I checked the apt-get version of ffmpeg I installed, and it has the nvenc and the like encoders listed under ffmpeg -encoders

CasparCG does not use the system ffmpeg version.

LRTNZ commented 4 years ago

Ok, will have to look at compiling it with that. The issue I then have is where to put what to get Caspar to compile with it? I am confident that I could most likely build an ffmpeg binary with the required things included. However I am stumped on how to get Caspar to compile with it, instead of its own one. I am aware it does not use the system version, but I was going to ask would it be possible for it maybe to? Would that potentially work?

Julusian commented 4 years ago

If you follow the development build instructions https://github.com/CasparCG/server/blob/master/BUILDING.md#development Then after step 5 either replace /opt/ffmpeg with your own version, or if you simply delete that folder then the system version will be used instead (if you have all the correct lib*-dev packages installed)

Packaging this up for distribution to another machine may be trickier though, I have not tried doing that

LRTNZ commented 4 years ago

So, after doing the whole extract deps from docker, if I just delete /opt/ffmpeg, the system build will definitely be used in the. Cmake+make -j8 steps?

Julusian commented 4 years ago

yep. I tried it myself, and it picked them up fine

LRTNZ commented 4 years ago

Ok, sweet - will try that later this morning, and report back. By the lib-dev packages, those are just all the ones in the building list? And on a side note - does the nvenc_h264 take -x264opt for parameters?

Julusian commented 4 years ago

No, they wont be in the list there. The extra ones to install should be libavcodec-dev libavformat-dev libavdevice-dev libavutil-dev libavfilter-dev libswscale-dev libpostproc-dev libswresample-dev

I have never tried using nvenc myself so I do not know.

walterav1984 commented 4 years ago

@LRTNZ your subject mentions "Producers" but it should be "Consumer" does it, or do you plan to also ask for "nvdec" Producer support which may also be interresting?

I also noticed ffmpeg in ubuntu 20.04 does ship with --enable-nvenc enabled however I first tried building my own version of ffmpeg and casparcg and noticed that it didn't work but forgot to check if the native apt ffmpeg version actually worked... I guess we still have to verify some nvidia cuda dependencies first. That would be my step 0 next time, get nvidia cuda/driver/runtime installed and check if native ffmpeg does encode with the nvenc options.

LRTNZ commented 4 years ago

@Julusian Ok, will give that a shot! This is partially why I work like this to be potentially added in as a compilation option - to simplify this process for all.

LRTNZ commented 4 years ago

Right, I have an update on this. By deleting the ffmpeg folder in /opt/ I have been able to compile casparcg with the one installed from apt-get. This means I can now use the GPU for h264 encoding - which is wonderful! However, another issue has arisen - audio is not working from these streams. I can specify the audio codecs and everything, yet I get nothing out. The audio meters and progress bar in the client are also not working, if that gives any clues to the issue?

dotarmin commented 4 years ago

Hi @LRTNZ

However, another issue has arisen - audio is not working from these streams. I can specify the > audio codecs and everything, yet I get nothing out.

Please create a new issue for this :)

The audio meters and progress bar in the client are also not working, if that gives any clues to > the issue?

Yes, thats expected behaviour with the current version of CasparCG Client. The OSC format has changed compared to previous versions of the server. Pull requests for the CasparCG Client will be merged during Q2.

LRTNZ commented 4 years ago

@dotarmin currently, the issue is not resolved? I tried getting the program to build with the system version off ffmpeg, as the one included with Caspar does not have the NVIDIA nvenc encoders enabled, and now cannot get sound out of it using that build. Also, I do not see how the progress/audio bars failing is the expected behaviour? One verison that they work with is with everything default, and the version they do not work with is where the only thing that has changed is using the system ffmpeg.

dotarmin commented 4 years ago

currently, the issue is not resolved?

Sorry, I assumed it was solved and the audio part was another issue. I will reopen it again.

Also, I do not see how the progress/audio bars failing is the expected behaviour?

Yes it is expected behaviour based on our decision we did for server >= v2.2.0. However, a new version of the CasparCG Client that works with >= v2.2.0 will be released later in Q2.

LRTNZ commented 4 years ago

Currently looking at having to rebuild the entire server install, as I tried to pull the future ffmpeg version down, and a lot of things broke in the process. I cannot work out how to take a custom build of ffmpeg, and get Caspar to pick that up instead of using either the one extracted from docker, or the system one.

LRTNZ commented 4 years ago

Ok, that is good to hear regarding the client.

However, my primary concern is getting an ffmpeg version with NVIDIA Hardware encoding into a working casparcg server version, to run on Linux.

LRTNZ commented 4 years ago

So the current point that I am at is:

What I would like to achieve: Have ffmpeg consumers in caspar, be able to encode a video file into a mpegts stream, utilising the power of the nvenc encoders, and cuda cores, with the resulting output being a multicast mpegts stream, with audio and video streams. I have previously reached this point, however all the encoding was taking place on the CPU - given there is going to need to be a GPU in this server, it may as well actually do some work. Also, it means the CPU will be able to have work offloaded, so that it can undertake other tasks it needs to perform in this installation.

walterav1984 commented 4 years ago

I have previously reached this point

Is the system native apt ffmpeg version 3.4.6-0ubuntu0.18.04.1 giving you the streaming capabilties as in video (x264/nvenc) and audio (aac not fdk) + multicast TS you want or is it specific combination of nvenc that does the audio vanish? Notice that CasparCG ffmpeg is build with "--enable-libfdk-aac".

LRTNZ commented 4 years ago

Using the native apt ffmpeg, both libx264 and h264_nvenc streams will produce video, but no audio. What is this about "--enable-libfdk-aac"?

walterav1984 commented 4 years ago

Just guessing you use AAC for audio codec which ffmpeg has 2 encoders for, however wikipedia lists TS container audio AAC support as "private" instead of Yes/No.

LRTNZ commented 4 years ago

Yes, the plan is to use the AAC audio codec - however at this point I would take any of them that would work with h264_nvenc.

walterav1984 commented 4 years ago

The extra ones to install should be libavcodec-dev libavformat-dev libavdevice-dev libavutil-dev libavfilter-dev libswscale-dev libpostproc-dev libswresample-dev

Building CasparCG server against distro ffmpeg version without "/opt/ffmpeg" works on ubuntu 20.04 if also package libavresample-dev is installed otherwise linking error /usr/bin/ld: cannot find -lavresample would occur.

LRTNZ commented 4 years ago

The extra ones to install should be libavcodec-dev libavformat-dev libavdevice-dev libavutil-dev libavfilter-dev libswscale-dev libpostproc-dev libswresample-dev

Building CasparCG server against distro ffmpeg version without "/opt/ffmpeg" works on ubuntu 20.04 if also package libavresample-dev is installed otherwise linking error /usr/bin/ld: cannot find -lavresample would occur.

Thats good to know , thank you for that! However, what about Ubuntu server 18.04? How would I build it for that, when the ffmpeg version of CasperCG being 4.4.2 does not match the apt native system version?

LRTNZ commented 4 years ago

Ok, so I have an update on this. After much tinkering and gently cursing at Linux/docker, I can report that between myself and my mate (who is familiar with docker) have managed to get Caspar to build with nvenc encoding enabled in ffmpeg. It only required a (minor) bit of tweaking in the various docker files, building a custom ffmpeg docker container image thing (whatever the correct term is), and then changing Caspar over to using that (by modifying image versions).
Unfortunately, just removing the extracted ffmpeg dir /opt/ffmpeg, whilst it would result in Caspar building , it always meant that while the video encoding would work, the audio stream would not be encoded in and work. I am going to follow the notes I recorded on how to do it from scratch, and see if I noted down everything in order to replicate it. @Julusian if I was to verify these steps, and to record it down in an "Enabling Nvenc on Linux" file, and submitted it as a PR, is that something that would be looked at being pulled into the repo? Just so the reference on how to do that can be stored locally with the codebase.

walterav1984 commented 4 years ago

@LRTNZ could you still post your ffmpeg stream requirements as in exact arguments tested and working with ffmpeg, since I can also confirm "nvenc" is working for me for file recording in casparcg on ubuntu-mate 20.04 with audio (haven't tested streaming yet). However I have difficulty finding the right way to parse ffmpeg arguments to AMCP, parsing a wrong argument kill the whole server.

walterav1984 commented 4 years ago

:man_facepalming: Besides the self build casparcg version, the dated casparcg 2.2 in the ubuntu 20.04 repository just works with nvenc if you meet the requirements at step 0...

LRTNZ commented 4 years ago

🤦‍♂️ Besides the self build casparcg version, the dated casparcg 2.2 in the ubuntu 20.04 repository just works with nvenc if you meet the requirements at step 0...

As I have said more than once, I am using Ubuntu 18.04 server, which is not changing. There is other stuff that will be running on this server, which has been developed and tested for this version.
Also, as I have said, the default ffmpeg version for 18.04 that I can get from apt-get would also work just fine for nvenc when used in caspar - the audio was the issue. And trying to pull ffmpeg from the future just broke everything when I tried doing it. I also have it working, and will look at putting together a PR with a documentation update on how I did it in the next day or so.

LRTNZ commented 4 years ago

@LRTNZ could you still post your ffmpeg stream requirements as in exact arguments tested and working with ffmpeg, since I can also confirm "nvenc" is working for me for file recording in casparcg on ubuntu-mate 20.04 with audio (haven't tested streaming yet). However I have difficulty finding the right way to parse ffmpeg arguments to AMCP, parsing a wrong argument kill the whole server.

The arguments are passed in exactly the same way as you pass all of the other arguments:
-[parameter]:[track to apply it to a/v] [value]
However, I will also add that I have not been using AMCP to parse in the arguments, I have only been using hardcoded ones in the casparcg.config file. So I am not fully sure if there are some nuances to it that may be causing issues.

What I have been using in my casparcg.config:
-codec:v h264_nvenc -preset:v high -cq:v 10 -aud:v 1 -g:v 25 -keyint_min:v 25 -filter:v format=yuv420p -format mpegts (And this is to generate a stream for exoplayer, which is very fussy).

walterav1984 commented 4 years ago

Thanks for writing the the stream command down, have you also fixed the audio issue that seems to be unclear?

LRTNZ commented 4 years ago

Yes, I have fixed the audio issue, by building ffmpeg using the dockerfile in the ~/Linux/ffmpeg/ directory, with a few modifications to it, as well as tweaking a few other files.
I plan to put in a PR on how I did this, as it is a little confusing to work it all out by yourself. Once the steps are written down, it is quite easy to follow.
However this all has drifted quite far from the original intent of this feature request which was to have this be a built in build option that makes it way easier for everyone to do.

Julusian commented 4 years ago

If the change is fairly simple then I would be open to passing an option into the ffmpeg dockerfile to build with nvenc. Perhaps we could we even have it enabled by default, but I suspect it will need the sdk which will have annoying distribution terms? Otherwise, perhaps it should have its own section in the BUILDING.md, and we could make it easier to use a custom prebuilt ffmpeg docker image.

If you could share your notes/modified dockerfile, then we can figure out what approach to take from there

walterav1984 commented 4 years ago

but I suspect it will need the sdk

I was in the assumption that you need it, did use it when building it the first time. But I didn't use it when I build it last time (different USB live session) building the /opt/ffmpeg 4.22 build outside docker with the only difference that I used ubuntu apt version configure rules and used sudo apt build-deb ffmpeg even didn't reference SDKlib locations during configure and it still worked.

However I did satisfy this as the only external source, but I doubt that is even needed:

git clone https://git.videolan.org/git/ffmpeg/nv-codec-headers.git
cd nv-codec-headers
make
sudo make install
Julusian commented 4 years ago

Ok that nv-codec-headers looks to be the sdk headers I was expecting, so adding that in the same way we retrieve the ffmpeg sourcecode should be easy. It is sounding like it would be reasonable to have it enabled by default then. The windows builds already include nvenc, so I dont see there being a licensing problem.

So it should just need a new ffmpeg dockerfile

LRTNZ commented 4 years ago

This is the server/tools/linux/ffmpeg dockerfile I am using for caspar server: Dockerfile.txt
In the custom images file, I changed the ffmpeg version to: export IMAGE_FFMPEG=custom-ffmpeg:latest

Then to build ffmpeg first run: export IMAGE_BASE=version, where version is the version of IMAGE_BASE in the image versions file. Then create a directory at the same level as the root folder containing the server repo files, enter it, and run: docker build --build-arg IMAGE_BASE=$IMAGE_BASE -t custom-ffmpeg ../server/tools/linux/ffmpeg/
After that, you can export the docker dependencies of casparcg, and build using the cmake, make method. This is because docker building with the modified ffmpeg version as in the file above does not work, and fails.

LRTNZ commented 4 years ago

If the change is fairly simple then I would be open to passing an option into the ffmpeg dockerfile to build with nvenc. Perhaps we could we even have it enabled by default, but I suspect it will need the sdk which will have annoying distribution terms? Otherwise, perhaps it should have its own section in the BUILDING.md, and we could make it easier to use a custom prebuilt ffmpeg docker image.

If you could share your notes/modified dockerfile, then we can figure out what approach to take from there

@Julusian The changes required are fairly simple, as can be seen in the attached file in my above reply. However, I have not had much luck running build-in-docker when I have changed the ffmpeg image to use, to be the custom one built with the steps outlined in my above reply. I suspect this is due to my very small amount of Docker experience, so someone else will probably have more luck getting that all to work.

I have realised that I forgot to add that you do also need to have installed the nvidia-cuda-dev and nvidia-cuda-toolkit packages when building CasparCG with this version of ffmpeg outside of docker.

Thinking about it, maybe it would be worth starting a discussion thread specifically about things to work on/add to the Linux documentation? As I have found over the past 1 - 2 weeks of working with this software, trying to get it to behave and run on Linux, there are a number of caveats to get it to run well.

Julusian commented 4 years ago

ok, I shall take a look at merging in your changes when I have some time. Could you try building it again on the latest changed I pushed to 2.3, as I had to remove a couple of existing arguemnts. One question though, do you know what functionality each of those enable flags add? In genuinely curious, as the windows zeranoe builds don't enable most of them.

--enable-libnpp \
--enable-nvenc \
--enable-cuda \
--enable-cuvid \
--enable-cuda-sdk \

Yeah, please do provide some input on the linux documentation. It is very sparse right now, and could do with more info in it. I don't think many people really run this on linux, so it likely hasnt been used much either.

LRTNZ commented 4 years ago

Regarding the flags, I was following the ones laid out here: https://developer.nvidia.com/ffmpeg, as well from here: https://trac.ffmpeg.org/wiki/HWAccelIntro. But in terms of functionality, to the best of my knowledge:

Yes, I will try build the code tomorrow (~12hrs) and check that it all works. I should really throw together a bash script to assist setting up new server builds, copying in the default config/media scanner....

Regarding Linux: Wonderful, will have to work on putting together a PR. Hopefully with a bit more documentation for Linux, some more people will consider that route. Turns out, it is not actually to hard to get it running completely headless (With fully automatic startup) in Ubuntu. However, X11 sure took some pursuading.

LRTNZ commented 4 years ago

@Julusian Was just looking into the changes that you have made to the dockerfiles, to look at the building on Linux. Just wanting to clarify, were you wanting me to go through the process I had with building the custom ffmpeg image for Caspar, to enable NVENC, or just doing the stock build in docker for Linux?

If you are would like me to do the custom ffmpeg route, based on the changes you made to the ffmpeg docker file: https://github.com/CasparCG/server/commit/e236466b398067f5656425ddd0fbd24d5eeda2f3#diff-6f2881e06a33407e5fe4d8d8168f6521 there is no longer the run section where I added in the additional Nvidia packages. Should I add those to the base docker file?

Julusian commented 4 years ago

I did do a build myself and it appeared fine (I didnt try running it though), so I am satisfied that it should be ok.

I would like you to try applying your nvenc changes on top of that. Yes, add the packages to the base docker file.

LRTNZ commented 4 years ago

Ok, so the first issue that I came across, is that the removal of the --enable-nonfree flag caused an issue when adding in the nvenc flags etc: https://github.com/CasparCG/server/commit/7d258d6ff2b2dbbdb0da94efebd669d843d43cf0

I am also having an issue where it is throwing "Failing to check for nvcc" when building using the additions I had made to the Dockerfile. Unfortunately, I am not to experienced with Docker, so I am not to sure quite how everything is processed.
What I did was to add in the Nvenc header install line from the dockerfile I had attached earlier, and added in the required flags - including adding back the --enable-nonfree flag.

Julusian commented 4 years ago

Unfortunately removing --enable-nonfree is something that we had to do, and it cannot be added back. Under the license of FFmpeg, any builds produced with that flag are not allowed to be redistributed.

Even without that flag, at least some of the nvenc components will work. The zeranoe windows ffmpeg builds have the following enabled in their builds. So perhaps lets start with those and you can see if there is anything you want missing after that

  --enable-ffnvcodec
  --enable-cuvid
  --enable-nvenc
  --enable-nvdec

I shall try and give this a go myself soon, depending on when I have an evening with some free time

Julusian commented 4 years ago

You're in luck, I was curious enough to take a look already.

Could you give https://github.com/CasparCG/server/compare/feat/linux-ffmpeg-matching-zeranoe a try? This branch will make the ffmpeg build match what we use on windows as closely as possible, and that includes some of the nvidia options. There is also some other general build tidying, but that shouldn't affect anything.

I haven't tested on an nvidia gpu yet, and dont really know where to begin with it really. Could you give it a quick test to see if it works properly?

LRTNZ commented 4 years ago

Ok, so I have just tried building it up - and it all appears to be working. Running build-in-docker from scratch did not work (As to be expected), but by running build-base-images first, then running build-in-docker it seems to have compiled correctly. Then running the resulting build, once I extracted it from docker, I was able to successfully create a multicast stream with the following ffmpeg consumer channel: 

<channel>
     <video-mode>1080p3000</video-mode>
     <consumers>
         <ffmpeg>
             <path>udp://239.1.1.5:1234?pkt_size=1316</path>
            <args>-codec:v h264_nvenc -profile:v high -cq:v 10 -aud:v 1 -g:v 25 -keyint_min:v 25 -filter:v format=yuv420p -format mpegts</args>
        </ffmpeg>
    </consumers>
</channel>

This is using nvenc to encode, and I don't really have the setup right now to test ndvec decoding.

Julusian commented 4 years ago

@LRTNZ are you happy to close this? I merged in the changes from that branch above, so nvenc and some other bits are now included by default

LRTNZ commented 4 years ago

Yes, thank you for this! Hopefully this update will open the door for more people to start more fully utilising the hardware they have in their servers!