Video Transcoding with HEVC 10 bit and nvenc

[Knoedelauflauf]

Hello there,

to be honest I am not sure if this issue belongs here or the the go-transcoder project.

I am running nextcloud memories with enabled nvenc option, most of the time the transcoding functions correctly, but with some videos hevc (Main 10) streams the transcoding fails.

go-vod.log:

2023/01/18 16:55:12ffmpeg-error: [h264_nvenc @ 0x55906b2c8ac0] 10 bit encode not supported
2023/01/18 16:55:12 ffmpeg-error: [h264_nvenc @ 0x55906b2c8ac0] Provided device doesn't support required NVENC features

The issue seems to be the wrong NVIDIA hardware acceleration, the transcoder uses h264_nvenc which does not support the HEVC 10 bit mode. According to the NVIDIA documentation[1] there exist three different nvenc: h264_nvenc - H264 Encoder hevc_nvenc - HEVC Encoder av1_nvenc - AV1 Encoder in case of the 10 bit video the HEVC encoder seems to be the correct choice.

I hope this is the correct project to submit the issue, if not please correct me and I will open the issue in the other project.

Kind regards knoedelauflauf

PS: thanks for the support and the nextcloud app.

[1] https://docs.nvidia.com/video-technologies/video-codec-sdk/ffmpeg-with-nvidia-gpu/#basic-testing

[pulsejet]

The problem is that it's trying to encode back to 10-bit. Maybe there's a flag to force 8-bit. I don't have an NVIDIA card so someone else needs to investigate this.

[Knoedelauflauf]

Starting with ffmpeg 5.0 there is an scale_cuda=format=nv12 option. I have tested the command manually and there was no error, but I have not tested if there is any effect on the quality of the stream, nor do I know how I could test the whole thing in memories.

[pulsejet]

I'd be careful. I think most distributions still ship with 4.x

You can try it by compiling go-vod yourself. Should be very straightforward.

[MB-Finski]

Oh, this is my bad. I was under the impression that "format=nv12" should already enforce converting to 8bit. When adding this to go-vod, I specifically tried 10bit source material with this in mind and it worked flawlessly. Although it wasn't HEVC 10bit but VP2 profile 2, which also gets software decoded, so maybe that's the issue.

[Knoedelauflauf]

I did some digging. If I understand the format=nv12|cuda,hwupload option correctly: the decoder will either output cuda or software frames, depending if hwaccel is usable. If cuda is the format, hwupload does nothing, otherwise hwupload converts the software frames and uploads them.

With ffmpeg4 I've tested a few options:

1. hwaccel_output_format nv12 but this results in videos with wrong colors.
2. format=cuda,hwdownload,format=p010le,format=nv12,hwupload,scale_cuda=w=640:h=360:force_original_aspect_ratio=decrease:passthrough=0 this worked for the hevc video but not for any other video, furthermore this was kind of slow compared to other videos with cuda, but still a little faster than the cpu encoding and the cpu usage was way less.

As for ffmpeg5, the scale_cuda=format=nv12 seems to be the most promising option, but I did only some basic testing, the video looked similar to the cpu encoded video, and the performance seemed to be better. But to be honest I would also rather stick with ffmpeg4, as ffmpeg5 is still only in debian testing and compiling from source and maintaining the system is not an real option.