Nvidia Hardware Transcoding HLS Issue + Fix

The bug NVENC accelerated transcoding succeeds, but the client's video will not play since the files are not sliced at all. When a video is requested from the client, transcoding begins, however it creates a single .ts file. Due to the size of transcoding an entire video, the browser appears to timeout, and sends a request for a lower resolution, repeating the process.

To Reproduce Enable NVENC accelerated encoding. Temporal AQ is disabled, using the CUDA scaler. Play a video through memories with a quality setting making use of transcoding. Check the size of the .ts file created (/tmp/go-vod//.ts).

To Fix After pulling the command from the log file (/tmp/go-vod/*.log), I was able to recreate the issue (successfully transcode the file, but only output a single .ts file. FFmpeg reported EXT-X-TARGETDURATION:246 once complete). I added -hls_flags split_by_time to the command, and it correctly created multiple .ts files in the output directory, and FFmpeg logged EXT-X-TARGETDURATION:3. CPU transcoding worked from the beginning, and I had no issue transcoding files with libx264 and the split_by_time flag.

Platform:

Browser: Chrome
Memories Version: 5.4.1
Nextcloud Version: 27.0.2
PHP Version: 8.1.22
GPU: Nvidia GTX 1070
GPU Driver Version: 535.104.05

Additional info Unsure if this is a separate bug, but the NPP scaler throws the following error:

Starting VOD server
2rtzftsh43n0: new manager for /storage/144959.mp4
2rtzftsh43n0-max: stopping stream
2rtzftsh43n0-max: /usr/local/bin/ffmpeg -loglevel warning -hwaccel cuda -i /storage/144959.mp4 -copyts -vf format=nv12|cuda,hwupload,scale_npp=force_original_aspect_ratio=decrease:passthrough=0 -profile:v main -map 0:v:0 -c:v h264_nvenc -preset p6 -tune ll -rc vbr -rc-lookahead 30 -cq 24 -map 0:a:0? -c:a aac -ac 1 -avoid_negative_ts disabled -f hls -hls_time 3 -force_key_frames expr:gte(t,n_forced*3) -hls_segment_type mpegts -start_number 0 -hls_segment_filename /tmp/go-vod/f7sd834jksdf/2rtzftsh43n0-783131264/max-%06d.ts -
ffmpeg-error: Error applying option 'passthrough' to filter 'scale_npp': Option not found
ffmpeg-error: [vost#0:0/h264_nvenc @ 0x5636b308ee00] Error initializing a simple filtergraph
ffmpeg-error: Error opening output file -.
ffmpeg-error: Error opening output files: Option not found
2rtzftsh43n0-max: ffmpeg exited with status: 8

Removing the :passthrough=0 from the command appears to let it run. Again, this results in a single large .ts file unless -hls_flags split_by_time is added. Please let me know if this NPP filter is due to a forgotten flag when building FFmpeg.

Additionally, do we want the browser making requests to the server for lower resolution videos, even if the end user has selected a specific resolution? As discovered, if a user requests a 1080p video, but it is not supplied, why does it request for the 720p video?

Thanks for the detailed report! These should be fixed now.

added -hls_flags split_by_time to the command, and it correctly created multiple .ts files in the output directory

https://github.com/pulsejet/go-vod/commit/58ac632839537f5c8b16124c7ab5c81a52869f76

Removing the :passthrough=0 from the command appears to let it run.

https://github.com/pulsejet/go-vod/commit/8d46582c11e3f3598ebf0ad89cb8e79169a842bf

Additionally, do we want the browser making requests to the server for lower resolution videos, even if the end user has selected a specific resolution? As discovered, if a user requests a 1080p video, but it is not supplied, why does it request for the 720p video?

On "Auto", videojs often requests different streams to check the bandwidth for adaptive streaming. But if a single resolution is selected, it should generally only request the video file for the selected resolution, not the others. It will often request the playlists for the other resolutions; this is not an issue since we don't invoke ffmpeg to generate the playlists.

So I built Memories from git to try out your changes, and they appear to all be functional. CUDA & NPP scalers are creating segments, and the folder scrollbar works lovely on desktop and android. 👍

Despite the HW accel transcoding pumping out segments faster, I'm still having issues with playback.

You say it shouldn't be requesting the other playlists, but it seems it still is. The video is set to only 720p, and the Memories default playback setting is direct.

VIDEOJS: WARN: Problem encountered with playlist 2-720p.m3u8. Excessive main segment downloading detected. Switching to playlist 0-360p.m3u8.
(anonymous) @ video.es.js:218
log.warn @ video.es.js:391
excludePlaylist @ video.es.js:47380
checkSegmentDownloads_ @ video.es.js:48279
updateend @ video.es.js:48125
data.dispatcher @ video.es.js:2144
trigger @ video.es.js:2275
trigger @ video.es.js:2621
handleAppendsDone_ @ video.es.js:42802
checkAppendsDone_ @ video.es.js:42707
(anonymous) @ video.es.js:43144
shiftQueue @ video.es.js:43082
(anonymous) @ video.es.js:43254

And FFmpeg is definitely being invoked to render all these other resolutions, as seen in nvtop in Ubuntu, and the fact all the browser requests have response data.

For some reason, it appears the video will play for the first few seconds, but then freeze & buffer, despite the console showing 200 response status for all the following *.ts files of the requested resolution... This does not happen with CPU transcoding.

Edit: If a shorter video is played (29s), the buffer bar will fully fill once all the .ts files arrive. This appears to happen because all segments of the m3u8 file are fulfilled, however it still doesn't play the video fully through. It also looks like short videos don't surpass the 'excessive main segment downloading' threshold, so it doesn't invoke the playlist-switch. Because of this, it is impossible to fully view short videos. However, longer videos eventually play because the playlist reverts to direct.

Larger log of all attempts

``` video.es.js:218 VIDEOJS: WARN: The element supplied is not included in the DOM (anonymous) @ video.es.js:218 log.warn @ video.es.js:391 videojs @ video.es.js:25342 initVideo @ PsVideo.ts:175 onContentActivate @ PsVideo.ts:491 (anonymous) @ photoswipe.esm.js:4431 dispatch @ photoswipe.esm.js:4430 activate @ photoswipe.esm.js:4938 activate @ photoswipe.esm.js:855 append @ photoswipe.esm.js:808 setContent @ photoswipe.esm.js:6416 init @ photoswipe.esm.js:6221 open @ Viewer.vue:582 await in open (async) routeChange @ Timeline.vue:194 await in routeChange (async) $route @ Timeline.vue:96 invokeWithErrorHandling @ vue.runtime.esm.js:3017 Watcher.run @ vue.runtime.esm.js:3534 flushSchedulerQueue @ vue.runtime.esm.js:4121 (anonymous) @ vue.runtime.esm.js:3143 flushCallbacks @ vue.runtime.esm.js:3065 Promise.then (async) timerFunc @ vue.runtime.esm.js:3090 nextTick @ vue.runtime.esm.js:3155 queueWatcher @ vue.runtime.esm.js:4207 Watcher.update @ vue.runtime.esm.js:3513 Dep.notify @ vue.runtime.esm.js:720 reactiveSetter @ vue.runtime.esm.js:954 (anonymous) @ vue-router.esm.js:3005 (anonymous) @ vue-router.esm.js:3004 updateRoute @ vue-router.esm.js:2414 (anonymous) @ vue-router.esm.js:2263 (anonymous) @ vue-router.esm.js:2402 step @ vue-router.esm.js:2084 runQueue @ vue-router.esm.js:2095 (anonymous) @ vue-router.esm.js:2397 step @ vue-router.esm.js:2084 (anonymous) @ vue-router.esm.js:2088 (anonymous) @ vue-router.esm.js:2384 (anonymous) @ vue-router.esm.js:2162 iterator @ vue-router.esm.js:2362 step @ vue-router.esm.js:2087 step @ vue-router.esm.js:2091 runQueue @ vue-router.esm.js:2095 confirmTransition @ vue-router.esm.js:2392 transitionTo @ vue-router.esm.js:2260 push @ vue-router.esm.js:2606 (anonymous) @ vue-router.esm.js:3036 push @ vue-router.esm.js:3035 openViewer @ SelectionManager.vue:883 clickPhoto @ SelectionManager.vue:334 pointerdown @ Timeline.vue:93 invokeWithErrorHandling @ vue.runtime.esm.js:3017 invoker @ vue.runtime.esm.js:1815 invokeWithErrorHandling @ vue.runtime.esm.js:3017 Vue.$emit @ vue.runtime.esm.js:3716 &pointerdown @ Photo.vue:84 invokeWithErrorHandling @ vue.runtime.esm.js:3017 invoker @ vue.runtime.esm.js:1815 original_1._wrapper @ vue.runtime.esm.js:7473 blob:https://cloud./4bb76a83-35b9-4485-9525-d9b8c29924d7:1 GET blob:https://cloud./4bb76a83-35b9-4485-9525-d9b8c29924d7 net::ERR_FILE_NOT_FOUND video.es.js:218 VIDEOJS: WARN: Problem encountered with playlist 6-max.m3u8. Excessive main segment downloading detected. Switching to playlist 0-360p.m3u8. (anonymous) @ video.es.js:218 log.warn @ video.es.js:391 excludePlaylist @ video.es.js:47380 checkSegmentDownloads_ @ video.es.js:48279 updateend @ video.es.js:48125 data.dispatcher @ video.es.js:2144 trigger @ video.es.js:2275 trigger @ video.es.js:2621 handleAppendsDone_ @ video.es.js:42802 checkAppendsDone_ @ video.es.js:42707 (anonymous) @ video.es.js:43144 shiftQueue @ video.es.js:43082 (anonymous) @ video.es.js:43254 video.es.js:218 VIDEOJS: WARN: Problem encountered with playlist 0-360p.m3u8. Excessive main segment downloading detected. Switching to playlist 1-480p.m3u8. video.es.js:218 VIDEOJS: WARN: Problem encountered with playlist 1-480p.m3u8. Excessive main segment downloading detected. Switching to playlist 2-720p.m3u8. video.es.js:218 VIDEOJS: WARN: Problem encountered with playlist 2-720p.m3u8. Excessive main segment downloading detected. Switching to playlist 3-1080p.m3u8. video.es.js:218 VIDEOJS: WARN: Problem encountered with playlist 3-1080p.m3u8. Excessive main segment downloading detected. Switching to playlist 4-1440p.m3u8. video.es.js:218 VIDEOJS: WARN: Problem encountered with playlist 4-1440p.m3u8. Excessive main segment downloading detected. Switching to playlist 5-2160p.m3u8. video.es.js:218 VIDEOJS: ERROR: (CODE:3 MEDIA_ERR_DECODE) Playback cannot continue. No available working or supported playlists. MediaError {code: 3, message: 'Playback cannot continue. No available working or supported playlists.'} (anonymous) @ video.es.js:218 log.error @ video.es.js:383 error @ video.es.js:23417 (anonymous) @ video.es.js:49290 data.dispatcher @ video.es.js:2144 trigger @ video.es.js:2275 trigger @ video.es.js:2621 excludePlaylist @ video.es.js:47375 checkSegmentDownloads_ @ video.es.js:48279 updateend @ video.es.js:48125 data.dispatcher @ video.es.js:2144 trigger @ video.es.js:2275 trigger @ video.es.js:2621 handleAppendsDone_ @ video.es.js:42802 checkAppendsDone_ @ video.es.js:42707 (anonymous) @ video.es.js:43144 shiftQueue @ video.es.js:43082 (anonymous) @ video.es.js:43254 PsVideo.ts:203 PsVideo: HLS stream could not be opened. (anonymous) @ PsVideo.ts:203 data.dispatcher @ video.es.js:2144 trigger @ video.es.js:2275 trigger @ video.es.js:3142 error @ video.es.js:23423 (anonymous) @ video.es.js:49290 data.dispatcher @ video.es.js:2144 trigger @ video.es.js:2275 trigger @ video.es.js:2621 excludePlaylist @ video.es.js:47375 checkSegmentDownloads_ @ video.es.js:48279 updateend @ video.es.js:48125 data.dispatcher @ video.es.js:2144 trigger @ video.es.js:2275 trigger @ video.es.js:2621 handleAppendsDone_ @ video.es.js:42802 checkAppendsDone_ @ video.es.js:42707 (anonymous) @ video.es.js:43144 shiftQueue @ video.es.js:43082 (anonymous) @ video.es.js:43254 PsVideo.ts:210 PsVideo: Trying direct video stream (anonymous) @ PsVideo.ts:210 data.dispatcher @ video.es.js:2144 trigger @ video.es.js:2275 trigger @ video.es.js:3142 error @ video.es.js:23423 (anonymous) @ video.es.js:49290 data.dispatcher @ video.es.js:2144 trigger @ video.es.js:2275 trigger @ video.es.js:2621 excludePlaylist @ video.es.js:47375 checkSegmentDownloads_ @ video.es.js:48279 updateend @ video.es.js:48125 data.dispatcher @ video.es.js:2144 trigger @ video.es.js:2275 trigger @ video.es.js:2621 handleAppendsDone_ @ video.es.js:42802 checkAppendsDone_ @ video.es.js:42707 (anonymous) @ video.es.js:43144 shiftQueue @ video.es.js:43082 (anonymous) @ video.es.js:43254 ```

If a 3s video is played, only one segment needs to be sent, the playlist is fulfilled and everything plays smoothly. On a 5s video, the first playthrough only plays the first segment (3s of the video). However clicking play again will play the entire 5s from the beginning. On an 11s video, the same effect is seen as the above 5s video, however the second play doesn't show the entire video, instead will only play 6 seconds of the video, aka only two segments, despite successfully receiving 3 segments. A 24s video has the same outcome as the 11s video, but of course receives more segments in total.

Bottom line is that the video freezes after 2 segments, but audio is heard through the entire clip.

Disclaimer: I don't have a NVIDIA card so can't really test anything.

Guessing from having to specify split_by_time, I think forcing keyframes isn't working as it should be. Can you try this patch on go-vod and see what happens?

diff --git a/stream.go b/stream.go
index 5c09ee6..42efc14 100644
--- a/stream.go
+++ b/stream.go
@@ -519,7 +519,8 @@ func (s *Stream) transcode(startId int) {
        "-hls_time", fmt.Sprintf("%d", s.c.ChunkSize),
        "-hls_segment_type", "mpegts",
        "-hls_segment_filename", s.getTsPath(-1),
-       "-force_key_frames", fmt.Sprintf("expr:gte(t,n_forced*%d)", s.c.ChunkSize),
+       "-g:v:0", fmt.Sprintf("%d", s.c.ChunkSize*s.m.probe.FrameRate),
+       "-keyint_min:v:0", fmt.Sprintf("%d", s.c.ChunkSize*s.m.probe.FrameRate),
        "-start_number", fmt.Sprintf("%d", startId),
        "-",
    }...)

BTW can you share the 720p.m3u8 being generated (see the preview tab) and the sizes of the ts files being downloaded (collapse the side bar and possibly need to add columns)

Took me a minute to figure it out, but I was able to clone go-vod, manually make the edits, and build it. Once I directed Memories to use my custom library, videos are successfully loading on the client. 👍👍

Here's the 720p.m3u8:

#EXTM3U
#EXT-X-VERSION:4
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-TARGETDURATION:3
#EXTINF:3.000, nodesc
720p-000000.ts
#EXTINF:3.000, nodesc
720p-000001.ts
#EXTINF:3.000, nodesc
720p-000002.ts
#EXTINF:3.000, nodesc
720p-000003.ts
#EXTINF:3.000, nodesc
720p-000004.ts
#EXTINF:3.000, nodesc
720p-000005.ts
#EXTINF:3.000, nodesc
720p-000006.ts
#EXTINF:3.000, nodesc
720p-000007.ts
#EXTINF:3.000, nodesc
720p-000008.ts
#EXTINF:3.000, nodesc
720p-000009.ts
#EXTINF:3.000, nodesc
720p-000010.ts
#EXTINF:3.000, nodesc
720p-000011.ts
#EXTINF:3.000, nodesc
720p-000012.ts
#EXTINF:3.000, nodesc
720p-000013.ts
#EXTINF:3.000, nodesc
720p-000014.ts
#EXTINF:3.000, nodesc
720p-000015.ts
#EXTINF:1.758, nodesc
720p-000016.ts
#EXT-X-ENDLIST

So can you confirm,

force_key_frames works on CPU transcoding
force_key_frames does NOT work correctly with NVENC
Specifying -g and -keyint_min works correctly with NVENC

Yes, force_key_frames works on CPU, but causes issues with NVENC. Replacing force_key_frames with '-g' and '-keyint_min' allows the client to play the video.

I built an additional copy of go-vod without any changes, just to confirm enabling external transcoder doesn't impact the outcome. My compiled go-vod has the exact same functionality as the included go-vod-amd64 when placed in the exiftool-bin folder and updated in the memories settings page. The modified go-vod bin then works for both NVENC and CPU transcoding. 🎉

Hopefully my experience applies to other CPU/GPU hardware as well. My configuration is: Intel Xeon E5-2470v2 (x2) Nvidia GTX 1070

Is there any other information you would like?

pulsejet / memories

Nvidia Hardware Transcoding HLS Issue + Fix #800