search

savonet / liquidsoap

Liquidsoap is a statically typed scripting general-purpose language with dedicated operators and backend for all thing media, streaming, file generation, automation, HTTP backend and more.
http://liquidsoap.info
GNU General Public License v2.0
1.4k stars 130 forks source link

setting video frame width/height causes memory leak, locks up system. #1740

Closed dikkedimi closed 2 years ago

dikkedimi commented 3 years ago

Describe the bug cpu usage goes from 100% to 140-180% when using the options below. Even though the static frame used is the same resolution.

video.frame.width.set(1920)
video.frame.height.set(1080)
video.frame.rate.set(24)

cpu load goes to 100% or more, and memory usage (I have 10GB, which should be plenty, right?), goes up to 95%, locking the entire system. Happens pretty soon, probably 5 minutes in. I think the load is more around 140%-180% when scaling to 1280x720.

I also noticed some audio crackle

Youtube also complains that "Please check the video resolution. The current resolution is (65535x65535), which is not optimal." I'm not sure where I'm outputting that kind of resolution.

Log

./sharedencoding-failover-harbor.liq 
[libx264 @ 0x560127700940] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[libx264 @ 0x560127700940] profile High, level 4.0
2021/07/16 20:04:40 >>> LOG START
2021/07/16 20:04:40 [main:3] Liquidsoap 2.0.0-beta2
2021/07/16 20:04:40 [main:3] Using: bytes=[distributed with OCaml 4.02 or above] pcre=7.4.6 sedlex=2.3 menhirLib=20210419 dtools=0.4.4 duppy=0.9.2 cry=0.6.5 mm=0.7.1 ogg=0.7.0 ogg.decoder=0.7.0 vorbis=0.8.0 vorbis.decoder=0.8.0 mad=0.5.0 dynlink=[distributed with Ocaml] lame=0.3.4 gstreamer=0.3.1 fdkaac=0.3.2 ffmpeg-avutil=1.0.0-beta1 ffmpeg-avcodec=1.0.0-beta1 ffmpeg-avdevice=1.0.0-beta1 ffmpeg-av=1.0.0-beta1 ffmpeg-avfilter=1.0.0-beta1 ffmpeg-swresample=1.0.0-beta1 ffmpeg-swscale=1.0.0-beta1 samplerate=0.1.5 taglib=0.3.6 magic=0.7.3 camomile=1.0.2 tsdl=v0.9.8 tsdl_ttf=0 tsdl_image=0
2021/07/16 20:04:40 [gstreamer.loader:3] Loaded GStreamer 1.16.2 0
2021/07/16 20:04:40 [clock:3] Using builtin (low-precision) implementation for latency control
2021/07/16 20:04:40 [frame:3] Using 44100Hz audio, 24Hz video, 88200Hz main.
2021/07/16 20:04:40 [frame:3] Frame size must be a multiple of 7350 ticks = 3675 audio samples = 2 video samples.
2021/07/16 20:04:40 [frame:3] Targeting 'frame.duration': 0.04s = 1764 audio samples = 3528 ticks.
2021/07/16 20:04:40 [frame:3] Frames last 0.08s = 3675 audio samples = 2 video samples = 7350 ticks.
2021/07/16 20:04:40 [sandbox:3] Sandboxing disabled
2021/07/16 20:04:40 [video.converter:3] Using preferred video converter: ffmpeg.
2021/07/16 20:04:40 [audio.converter:3] Using samplerate converter: ffmpeg.
2021/07/16 20:04:40 [single_0:3] "/home/user/assets/music/DJ-sets/set.mp3" is static, resolving once for all...
2021/07/16 20:04:40 [single_0:3] Prepared "/home/user/assets/music/DJ-sets/set.mp3" (RID 0).
2021/07/16 20:04:40 [/stream:3] Connecting mount /stream for source@10.0.0.3...
2021/07/16 20:04:40 [/stream:3] Connection setup was successful.
2021/07/16 20:04:41 [clock.main:3] Streaming loop starts in auto-sync mode
2021/07/16 20:04:41 [clock.main:3] Delegating synchronisation to CPU clock
2021/07/16 20:04:41 [switch_0:3] Switch to single_0.
2021/07/16 20:04:41 [decoder:3] Method "SDL/IMAGE" accepted "/home/user/assets/graphics/splash.jpg".
[mp3float @ 0x7fc47c009f00] Could not update timestamps for skipped samples.
2021/07/16 20:04:41 [playlist_0:3] Prepared "/home/user/assets/music/unreleased/tune.wav" (RID 1).
2021/07/16 20:04:41 [switch_0:3] Switch to crossfade_0 with transition.
2021/07/16 20:04:41 [clock.ffmpeg.encode.audio.producer_0.child:2] Source ffmpeg.encode.audio.consumer_0 failed while streaming: Invalid_argument("index out of bounds")!
2021/07/16 20:04:41 [main:3] Shutdown started!
2021/07/16 20:04:41 [main:3] Waiting for main threads to terminate...
2021/07/16 20:04:42 [decoder:3] Method "SDL/IMAGE" accepted "/home/user/assets/graphics/splash.jpg".
^C^C^C^C^C^C^C^CConnection to 10.0.0.2 closed by remote host.
Connection to 10.0.0.2 closed.

while commenting out video width/height settings, produces the following log output, but frames my static image like it's in 1280x720. Memory usage stay steady at about 8%

./sharedencoding-failover-harbor.liq 
[libx264 @ 0x55f7a2294040] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[libx264 @ 0x55f7a2294040] profile High, level 3.1
2021/07/16 20:16:04 >>> LOG START
2021/07/16 20:16:04 [main:3] Liquidsoap 2.0.0-beta2
2021/07/16 20:16:04 [main:3] Using: bytes=[distributed with OCaml 4.02 or above] pcre=7.4.6 sedlex=2.3 menhirLib=20210419 dtools=0.4.4 duppy=0.9.2 cry=0.6.5 mm=0.7.1 ogg=0.7.0 ogg.decoder=0.7.0 vorbis=0.8.0 vorbis.decoder=0.8.0 mad=0.5.0 dynlink=[distributed with Ocaml] lame=0.3.4 gstreamer=0.3.1 fdkaac=0.3.2 ffmpeg-avutil=1.0.0-beta1 ffmpeg-avcodec=1.0.0-beta1 ffmpeg-avdevice=1.0.0-beta1 ffmpeg-av=1.0.0-beta1 ffmpeg-avfilter=1.0.0-beta1 ffmpeg-swresample=1.0.0-beta1 ffmpeg-swscale=1.0.0-beta1 samplerate=0.1.5 taglib=0.3.6 magic=0.7.3 camomile=1.0.2 tsdl=v0.9.8 tsdl_ttf=0 tsdl_image=0
2021/07/16 20:16:04 [gstreamer.loader:3] Loaded GStreamer 1.16.2 0
2021/07/16 20:16:04 [clock:3] Using builtin (low-precision) implementation for latency control
2021/07/16 20:16:04 [frame:3] Using 44100Hz audio, 25Hz video, 44100Hz main.
2021/07/16 20:16:04 [frame:3] Frame size must be a multiple of 1764 ticks = 1764 audio samples = 1 video samples.
2021/07/16 20:16:04 [frame:3] Targeting 'frame.duration': 0.04s = 1764 audio samples = 1764 ticks.
2021/07/16 20:16:04 [frame:3] Frames last 0.04s = 1764 audio samples = 1 video samples = 1764 ticks.
2021/07/16 20:16:04 [sandbox:3] Sandboxing disabled
2021/07/16 20:16:04 [video.converter:3] Using preferred video converter: ffmpeg.
2021/07/16 20:16:04 [audio.converter:3] Using samplerate converter: ffmpeg.
2021/07/16 20:16:04 [single_0:3] "/home/user/assets/music/DJ-sets/set.mp3" is static, resolving once for all...
2021/07/16 20:16:04 [single_0:3] Prepared "/home/user/assets/music/DJ-sets/set.mp3" (RID 0).
2021/07/16 20:16:04 [/stream:3] Connecting mount /stream for source@10.0.0.3...
2021/07/16 20:16:04 [/stream:3] Connection setup was successful.
2021/07/16 20:16:08 [clock.main:3] Streaming loop starts in auto-sync mode
2021/07/16 20:16:08 [clock.main:3] Delegating synchronisation to CPU clock
2021/07/16 20:16:08 [switch_0:3] Switch to single_0.
2021/07/16 20:16:08 [decoder:3] Method "SDL/IMAGE" accepted "/home/user/assets/graphics/splash.jpg".
[mp3float @ 0x7f0490006340] Could not update timestamps for skipped samples.
2021/07/16 20:16:08 [playlist_0:3] Prepared "/home/user/assets/music/unreleased/tune.wav" (RID 1).
2021/07/16 20:16:08 [switch_0:3] Switch to crossfade_0 with transition.
2021/07/16 20:16:08 [decoder:3] Method "SDL/IMAGE" accepted "/home/user/assets/graphics/splash.jpg".
2021/07/16 20:16:08 [single_1:3] Prepared "/home/user/assets/graphics/splash.jpg" (RID 2).
2021/07/16 20:16:08 [decoder:3] Method "SDL/IMAGE" accepted "/home/user/assets/graphics/splash.jpg".

Youtube still complains about this weird resolution (65535x65535), but the stream works, audio is good, To Reproduce

#!/usr/bin/env liquidsoap

 video.frame.width.set(1920)
 video.frame.height.set(1080)
 video.frame.rate.set(24)

%include "youtube.liq"

YTurl = "rtmp://a.rtmp.youtube.com/live2/#{YTKey}"
live    =   input.http("10.0.0.3:8000")

#sources

#jukebox directory
jukebox =   playlist("/home/user/assets/music/unreleased")

#juicebox crossfade
jukebox =   crossfade(fade_out=3., fade_in=3., duration=6., jukebox)

# emergency fallback
emergency   =   single("/home/user/assets/music/DJ-sets/set.mp3")

# set up fallability
stream  =   fallback(track_sensitive=false, [live, jukebox, emergency, blank()])

# Encode it in mp3
EncodedAudio = ffmpeg.encode.audio(
  %ffmpeg(%audio(codec="libmp3lame")),
  stream
)

# Send it to icecast
output.icecast(
  %ffmpeg(format="mp3", %audio.copy),
  fallible=true,
  host = "10.0.0.3",
  port = 8443,
  password = "12Receive",
  mount = "/stream",
  EncodedAudio
)

# Splash injection
# liquidsoap  installed with tsdl-image magic tsdl tsdl-ttf

StaticFrame = video.add_image(
       width=1920, height=1080,
       x=0, y=0,
       file="/home/user/assets/graphics/splash.jpg", blank())

# Encode video in h264 format
StaticFrame = ffmpeg.encode.video(
  %ffmpeg(%video(codec="libx264", g=48)),
  StaticFrame)

# Mux it with the audio into stream
MuxedStream = mux_video(video=StaticFrame, EncodedAudio)
# mksafe(stream)

# Copy encoder settings(?) or coded data(?) for the rtmp stream
enc = %ffmpeg(
  format="flv",
  %audio.copy,
  %video.copy
)

# Send to YouTube
output.url(url=YTurl, enc, MuxedStream)

Expected behavior I would expect some increase in cpu usage from scaling an image,, but not 10GB of memory used to 95%. the 8% used without the options seems more within expectations

Version details

Install method installed 2.0.0 beta 2

Common issues ehhh, not sure. sorry.

nigeyuk commented 3 years ago

I set it mine like this

# Video Settings
set("frame.video.width",1280)
set("frame.video.height",720)

and i get no mem leak, maybe worth a try?

dikkedimi commented 3 years ago

I set it mine like this

Video Settings

set("frame.video.width",1280) set("frame.video.height",720)

and i get no mem leak, maybe worth a try?

This works! However when I try that syntax on the frame rate:

~/assets/recipes$ ./sharedencoding-failover.liq 
At line 9, char 21:
Error 7: Invalid value:
there is no configuration key named "frame.video.rate"!

Youtube however does no longer complain about the 65535x65535 resolution, so I guess that issue is solved, at the least :) Hopefully my splash image suffers less from degradation.

Can I ask where you got this syntax from? I checked the 2.0 docs but I guess they're not up to date yet. Maybe we could contribute. I got my syntax from this page: https://www.liquidsoap.info/doc-dev/video-static.html

Oh, never mind;

Found out the 1.4.4 syntax still works https://www.liquidsoap.info/doc-1.4.4/settings.html

set("frame.video.width", 1920)
set("frame.video.height", 1080)
set("frame.video.samplerate",24)

but:

[lang.deprecated:2] WARNING: "frame.video.samplerate" is deprecated and will be removed in future version. Please use "frame.video.framerate" instead.

so

set("frame.video.width", 1920)
set("frame.video.height", 1080)
# set("frame.video.samplerate",24)
set("frame.video.framerate", 24)

but

./sharedencoding-failover.liq 
[libx264 @ 0x563429fa3640] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[libx264 @ 0x563429fa3640] profile High, level 4.0
2021/07/18 13:32:48 >>> LOG START
2021/07/18 13:32:47 [main:3] Liquidsoap 2.0.0-beta2
2021/07/18 13:32:47 [main:3] Using: bytes=[distributed with OCaml 4.02 or above] pcre=7.4.6 sedlex=2.3 menhirLib=20210419 dtools=0.4.4 duppy=0.9.2 cry=0.6.5 mm=0.7.1 ogg=0.7.0 ogg.decoder=0.7.0 vorbis=0.8.0 vorbis.decoder=0.8.0 mad=0.5.0 dynlink=[distributed with Ocaml] lame=0.3.4 gstreamer=0.3.1 fdkaac=0.3.2 ffmpeg-avutil=1.0.0-beta1 ffmpeg-avcodec=1.0.0-beta1 ffmpeg-avdevice=1.0.0-beta1 ffmpeg-av=1.0.0-beta1 ffmpeg-avfilter=1.0.0-beta1 ffmpeg-swresample=1.0.0-beta1 ffmpeg-swscale=1.0.0-beta1 samplerate=0.1.5 taglib=0.3.6 magic=0.7.3 camomile=1.0.2 tsdl=v0.9.8 tsdl_ttf=0 tsdl_image=0
2021/07/18 13:32:47 [gstreamer.loader:3] Loaded GStreamer 1.16.2 0
2021/07/18 13:32:47 [clock:3] Using builtin (low-precision) implementation for latency control
2021/07/18 13:32:48 [frame:3] Using 44100Hz audio, 24Hz video, 88200Hz main.
2021/07/18 13:32:48 [frame:3] Frame size must be a multiple of 7350 ticks = 3675 audio samples = 2 video samples.
2021/07/18 13:32:48 [frame:3] Targeting 'frame.duration': 0.04s = 1764 audio samples = 3528 ticks.
2021/07/18 13:32:48 [frame:3] Frames last 0.08s = 3675 audio samples = 2 video samples = 7350 ticks.
2021/07/18 13:32:48 [sandbox:3] Sandboxing disabled
2021/07/18 13:32:48 [video.converter:3] Using preferred video converter: ffmpeg.
2021/07/18 13:32:48 [audio.converter:3] Using samplerate converter: ffmpeg.
2021/07/18 13:32:48 [single_0:3] "/home/user/assets/music/dj-sets/set.mp3" is static, resolving once for all...
2021/07/18 13:32:48 [single_0:3] Prepared "/home/user/assets/music/dj-sets/set.mp3" (RID 0).
2021/07/18 13:32:48 [/stream:3] Connecting mount /stream for source@10.0.0.3...
2021/07/18 13:32:48 [/stream:3] Connection setup was successful.
2021/07/18 13:32:48 [single_1:3] "/home/user/assets/graphics/splash.jpg" is static, resolving once for all...
2021/07/18 13:32:48 [decoder:3] Method "SDL/IMAGE" accepted "/home/user/assets/graphics/splash.jpg".
2021/07/18 13:32:48 [decoder:3] Method "SDL/IMAGE" accepted "/home/user/assets/graphics/splash.jpg".
2021/07/18 13:32:48 [single_1:3] Prepared "/home/user/assets/graphics/splash.jpg" (RID 1).
2021/07/18 13:32:51 [clock.main:3] Streaming loop starts in auto-sync mode
2021/07/18 13:32:51 [clock.main:3] Delegating synchronisation to CPU clock
2021/07/18 13:32:52 [switch_0:3] Switch to single_0.
[mp3float @ 0x7f00d8009f00] Could not update timestamps for skipped samples.
2021/07/18 13:32:52 [playlist_0:3] Prepared "/home/user/assets/music/tracks/releases/Flarking Breed/Feelz EP/Masters/Flarking Breed Feelz EP/Wave/02 Feelz (Original Mix).wav" (RID 2).
2021/07/18 13:32:52 [switch_0:3] Switch to crossfade_0 with transition.
2021/07/18 13:32:52 [clock.ffmpeg.encode.audio.producer_0.child:2] Source ffmpeg.encode.audio.consumer_0 failed while streaming: Invalid_argument("index out of bounds")!
2021/07/18 13:32:52 [main:3] Shutdown started!
2021/07/18 13:32:52 [main:3] Waiting for main threads to terminate...

and the memory leak is back..

So I guess I'll leave that option out for now. If I need to try stuff, let me know, I'll happpily post the results

toots commented 3 years ago

Hi all! This looks like a bug with the recent settings changes, I'm looking at it. Thanks for reporting!

toots commented 3 years ago

Ok, so far everything points to add() being way too inefficient as the source of the issue.

Any chance you could try the original script with this instead of video.add_image:

StaticFrame = mksafe(single("annotate:width=1920,height=1080:/home/user/assets/graphics/splash.jpg"))
toots commented 3 years ago

Also, if you are only sending to a single rtmp output, I'd recommend to remove the inline encoding here:

StaticFrame = ffmpeg.encode.video(
  %ffmpeg(%video(codec="libx264", g=48)),
  StaticFrame)

And simply do:

enc = %ffmpeg(
  format="flv",
  %audio.copy,
  %video(codec="libx264", g=48)
)

For some reason, this seems to be much more efficient on my end (about half CPU usage).

Lastly, for a static image, you could also pre-encode a video track with just that image and use the copy decoder. In this case, no video encoding is done at all!

StaticFrame = merge_tracks(single("/tmp/static.mp4"))

# And no ffmpeg.encode.video
toots commented 3 years ago

Also, if you are only sending to a single rtmp output, I'd recommend to remove the inline encoding here:

StaticFrame = ffmpeg.encode.video(
  %ffmpeg(%video(codec="libx264", g=48)),
  StaticFrame)

And simply do:

enc = %ffmpeg(
  format="flv",
  %audio.copy,
  %video(codec="libx264", g=48)
)

For some reason, this seems to be much more efficient on my end (about half CPU usage).

This has been fixed in the latest main!

dikkedimi commented 3 years ago

Ok, so far everything points to add() being way too inefficient as the source of the issue.

Any chance you could try the original script with this instead of video.add_image:

StaticFrame = mksafe(single("annotate:width=1920,height=1080:/home/user/assets/graphics/splash.jpg"))

took a few days off to do some other activities, sorry for the delay in response

This runs steadily on 140% cpu load but with ever so slightly increasing memory load. I think if I run this for 12 hours we'll get a similar result.

#!/usr/bin/env liquidsoap
 #
 # video.frame.width.set(1920)
 # video.frame.height.set(1080)
 # video.frame.rate.set(24)
# set("video.converter.preferred","ffmpeg")
# set("decoder.file_extensions.ffmpeg",["mp3","mp4","m4a","wav","flac","ogg","wma","webm","osb"])

set("frame.video.width", 1920)
set("frame.video.height", 1080)
# set("frame.video.samplerate",24)
# set("frame.video.framerate", 24)
# set("frame.video.rate", 24)
%include "/home/user/assets/recipes/youtube.liq"

YTurl = "rtmp://a.rtmp.youtube.com/live2/#{YTKey}"
live    =   input.http("10.0.0.3:8000")

#sources

#jukebox directory
jukebox =   playlist("/home/user/assets/music/tracks")

#juicebox crossfade
jukebox =   crossfade(fade_out=3., fade_in=3., duration=3., jukebox)

# emergency fallback
emergency   =   single("/home/user/assets/music/dj-sets/set.mp3")

# set up fallability
stream  =   fallback(track_sensitive=false, [live, jukebox, emergency, blank()])

# Encode it in mp3
EncodedAudio = ffmpeg.encode.audio(
  %ffmpeg(%audio(codec="libmp3lame", b="128k")),
  stream
)

# Send it to icecast
output.icecast(
  %ffmpeg(format="mp3", %audio.copy),
  fallible=true,
  host = "10.0.0.3",
  port = 8443,
  password = "12Receive",
  mount = "/stream",
  EncodedAudio
)

# Splash injection
# liquidsoap  installed with tsdl-image magic tsdl tsdl-ttf
#StaticFrame = single("/home/user/assets/graphics/splash.jpg")
#toots suggested change
StaticFrame = mksafe(single("annotate:width=1920,height=1080:/home/user/assets/graphics/splash.jpg"))
# StaticFrame = video.add_image(
#        width=80, height=80,
#        x=1820, y=980,
#        file="/home/user/assets/graphics/logo-transparent.gif", blank())

# Encode video in h264 format
EncodedVideo = ffmpeg.encode.video(
  %ffmpeg(%video(codec="libx264"
  #, g=48, threads=2
  )),
  StaticFrame)

# Mux it with the audio into stream
MuxedStream = mux_video(video=EncodedVideo, EncodedAudio)
# mksafe(stream)

# Copy encoder settings(?) or coded data(?) for the rtmp stream
enc = %ffmpeg(
  format="flv",
  %audio.copy,
  %video.copy
)

# Send to YouTube
output.url(url=YTurl, enc, MuxedStream)
dikkedimi commented 3 years ago

Also, if you are only sending to a single rtmp output, I'd recommend to remove the inline encoding here:

StaticFrame = ffmpeg.encode.video(
  %ffmpeg(%video(codec="libx264", g=48)),
  StaticFrame)

And simply do:

enc = %ffmpeg(
  format="flv",
  %audio.copy,
  %video(codec="libx264", g=48)
)

For some reason, this seems to be much more efficient on my end (about half CPU usage).

This works, but with higher CPU load (~150%) and substantially more memory usage (36%), which also climbs steadily.

Lastly, for a static image, you could also pre-encode a video track with just that image and use the copy decoder. In this case, no video encoding is done at all!

StaticFrame = merge_tracks(single("/tmp/static.mp4"))

# And no ffmpeg.encode.video

A friend recommended this to me and it's probably the best way to go. Makes sense to prepare static sources, as decoding (and thus injecting) a PNG/GIF (transparent image) doesn't seem to work at all.


#!/usr/bin/env liquidsoap
 #
 # video.frame.width.set(1920)
 # video.frame.height.set(1080)
 # video.frame.rate.set(24)
# set("video.converter.preferred","ffmpeg")
# set("decoder.file_extensions.ffmpeg",["mp3","mp4","m4a","wav","flac","ogg","wma","webm","osb"])

set("frame.video.width", 1920)
set("frame.video.height", 1080)
# set("frame.video.samplerate",24)
# set("frame.video.framerate", 24)
# set("frame.video.rate", 24)
%include "/home/user/assets/recipes/youtube.liq"

YTurl = "rtmp://a.rtmp.youtube.com/live2/#{YTKey}"
live    =   input.http("10.0.0.3:8000")

#sources

#jukebox directory
jukebox =   playlist("/home/user/assets/music/tracks")

#juicebox crossfade
jukebox =   crossfade(fade_out=3., fade_in=3., duration=3., jukebox)

# emergency fallback
emergency   =   single("/home/user/assets/music/dj-sets/set.mp3")

# set up fallability
stream  =   fallback(track_sensitive=false, [live, jukebox, emergency, blank()])

# Encode it in mp3
EncodedAudio = ffmpeg.encode.audio(
  %ffmpeg(%audio(codec="libmp3lame", b="128k")),
  stream
)

# Send it to icecast
output.icecast(
  %ffmpeg(format="mp3", %audio.copy),
  fallible=true,
  host = "10.0.0.3",
  port = 8443,
  password = "12Receive",
  mount = "/stream",
  EncodedAudio
)

# Splash injection
# liquidsoap  installed with tsdl-image magic tsdl tsdl-ttf
#StaticFrame = single("/home/user/assets/graphics/splash.jpg")

#toots suggested change 1
StaticFrame = mksafe(single("annotate:width=1920,height=1080:/home/user/assets/graphics/splash.jpg"))

# StaticFrame = video.add_image(
#        width=80, height=80,
#        x=1820, y=980,
#        file="/home/user/assets/graphics/logo-transparent.gif", blank())

# Encode video in h264 format
# EncodedVideo = ffmpeg.encode.video(
#   %ffmpeg(%video(codec="libx264"
#   #, g=48, threads=2
#   )),
#   StaticFrame)

# Mux it with the audio into stream
MuxedStream = mux_video(video=StaticFrame, EncodedAudio)
# mksafe(stream)

# Copy encoder settings(?) or coded data(?) for the rtmp stream
enc = %ffmpeg(
  format="flv",
  %audio.copy,
  %video(codec="libx264", g=48)
)

# Send to YouTube
output.url(url=YTurl, enc, MuxedStream)
dikkedimi commented 3 years ago

Also, if you are only sending to a single rtmp output, I'd recommend to remove the inline encoding here:

StaticFrame = ffmpeg.encode.video(
  %ffmpeg(%video(codec="libx264", g=48)),
  StaticFrame)

And simply do:

enc = %ffmpeg(
  format="flv",
  %audio.copy,
  %video(codec="libx264", g=48)
)

For some reason, this seems to be much more efficient on my end (about half CPU usage).

This has been fixed in the latest main!

Not sure how to test this in my 2.0.0b2 installed from tar.gz Should we close for now?

toots commented 3 years ago

Hi!

The best is indeed a pre-encoded short video, with no re-encoding.

Transparency does not work at the moment on video. It's a pretty niche support, most video codecs do not support it. As far as I have been able to test, we do carry the alpha channel but it does not seem to work with ffmpeg.

Let's keep this open for a minute to see if we can do anything about the performances of add_video.

toots commented 3 years ago

Any update with this issue? Have y'all been able to find a workaround?

dikkedimi commented 3 years ago

Hi @toots ! Sorry for the long wait. tumultuous times. Anyhow, I got around to testing with a 1 second video. It is encoded for youtube, by Davinci Resolve, so that should be optimal. (quicktime says it's H.264, 1920x1080, AAC, 48000hz, Stereo (L R) @ 60FPS (2,44Mbit/s).

seems to work at first, but it makes liquidsoap throw up once again. Googling the error given gave me some results, but nothing conclusive (at least to me 😅)

The only thing changed from previous attempts is the file path.

here's my code

#!/usr/bin/env liquidsoap
set("frame.video.width", 1920)
set("frame.video.height", 1080)
%include "/home/user/assets/recipes/youtube.liq"

YTurl = "rtmp://a.rtmp.youtube.com/live2/#{YTKey}"
live    =       input.http("10.0.0.3:8000")

#sources
#jukebox directory
jukebox =       playlist("/home/user/assets/music/tracks")

#jukebox crossfade
jukebox =       crossfade(fade_out=3., fade_in=3., duration=3., jukebox)

# emergency fallback
emergency       =       single("/home/user/assets/music/dj-sets/loot.cache.mp3")

# set up fallability
stream  =       fallback(track_sensitive=false, [live, jukebox, emergency, blank()])

# Encode it in mp3
EncodedAudio = ffmpeg.encode.audio(
  %ffmpeg(%audio(codec="libmp3lame", b="128k")),
  stream
)

# Send it to icecast
output.icecast(
  %ffmpeg(format="mp3", %audio.copy),
  fallible=true,
  host = "10.0.0.3",
  port = 8443,
  password = "12Receive",
  mount = "/stream",
  EncodedAudio
)

# Splash injection
StaticFrame = mksafe(single("annotate:width=1920,height=1080:/home/user/assets/graphics/CFL_Sign.mp4"))

# Mux it with the audio into stream
MuxedStream = mux_video(video=StaticFrame, EncodedAudio)
# mksafe(stream)

# Copy encoder settings(?) or coded data(?) for the rtmp stream
enc = %ffmpeg(
  format="flv",
  %audio.copy,
  %video(codec="libx264", g=48)
)

# Send to YouTube
output.url(url=YTurl, enc, MuxedStream)

The result

./sharedencoding-failover.liq 
2021/10/06 19:39:57 >>> LOG START
2021/10/06 19:39:56 [main:3] Liquidsoap 2.0.0-beta2
2021/10/06 19:39:56 [main:3] Using: bytes=[distributed with OCaml 4.02 or above] pcre=7.4.6 sedlex=2.3 menhirLib=20210419 dtools=0.4.4 duppy=0.9.2 cry=0.6.5 mm=0.7.1 ogg=0.7.0 ogg.decoder=0.7.0 vorbis=0.8.0 vorbis.decoder=0.8.0 mad=0.5.0 dynlink=[distributed with Ocaml] lame=0.3.4 gstreamer=0.3.1 fdkaac=0.3.2 ffmpeg-avutil=1.0.0-beta1 ffmpeg-avcodec=1.0.0-beta1 ffmpeg-avdevice=1.0.0-beta1 ffmpeg-av=1.0.0-beta1 ffmpeg-avfilter=1.0.0-beta1 ffmpeg-swresample=1.0.0-beta1 ffmpeg-swscale=1.0.0-beta1 samplerate=0.1.5 taglib=0.3.6 magic=0.7.3 camomile=1.0.2 tsdl=v0.9.8 tsdl_ttf=0 tsdl_image=0
2021/10/06 19:39:56 [gstreamer.loader:3] Loaded GStreamer 1.16.2 0
2021/10/06 19:39:56 [clock:3] Using builtin (low-precision) implementation for latency control
2021/10/06 19:39:57 [frame:3] Using 44100Hz audio, 25Hz video, 44100Hz main.
2021/10/06 19:39:57 [frame:3] Frame size must be a multiple of 1764 ticks = 1764 audio samples = 1 video samples.
2021/10/06 19:39:57 [frame:3] Targeting 'frame.duration': 0.04s = 1764 audio samples = 1764 ticks.
2021/10/06 19:39:57 [frame:3] Frames last 0.04s = 1764 audio samples = 1 video samples = 1764 ticks.
2021/10/06 19:39:57 [sandbox:3] Sandboxing disabled
2021/10/06 19:39:57 [video.converter:3] Using preferred video converter: ffmpeg.
2021/10/06 19:39:57 [audio.converter:3] Using samplerate converter: ffmpeg.
2021/10/06 19:39:57 [single_0:3] "/home/user/assets/music/dj-sets/simmer.session.berlin.loot.cache.mp3" is static, resolving once for all...
2021/10/06 19:39:57 [single_0:3] Prepared "/home/user/assets/music/dj-sets/simmer.session.berlin.loot.cache.mp3" (RID 0).
[libx264 @ 0x7f5324038d00] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[libx264 @ 0x7f5324038d00] profile High, level 4.0
[libx264 @ 0x7f5324038d00] 264 - core 155 r2917 0a84d98 - H.264/MPEG-4 AVC codec - Copyleft 2003-2018 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=48 keyint_min=4 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
2021/10/06 19:39:57 [clock.main:3] Streaming loop starts in auto-sync mode
2021/10/06 19:39:57 [clock.main:3] Delegating synchronisation to CPU clock
2021/10/06 19:39:57 [/stream:3] Connecting mount /stream for source@10.0.0.3...
2021/10/06 19:39:57 [/stream:2] Connection failed: bad answer: scanf: bad input at char number 0: looking for 'H', found '\021'
2021/10/06 19:39:57 [/stream:3] Will try again in 3.00 sec.
2021/10/06 19:39:57 [switch_0:3] Switch to single_0.
[mp3float @ 0x7f52fc2fd9c0] Could not update timestamps for skipped samples.
2021/10/06 19:39:57 [mksafe:3] Switch to safe_blank.
2021/10/06 19:39:57 [playlist_0:3] Prepared "/home/user/assets/music/tracks/unreleased/Romancer - Fuck You Good (Long version with proper guitar version).wav" (RID 1).
2021/10/06 19:39:57 [switch_0:3] Switch to crossfade_0 with transition.
2021/10/06 19:39:57 [single_1:3] Prepared "/home/user/assets/graphics/CFL_Sign.mp4" (RID 2).
2021/10/06 19:39:57 [mksafe:3] Switch to single_1 with transition.

(Liquidsoap:4342): GStreamer-CRITICAL **: 19:39:57.891: 
Trying to dispose element fakesink0, but it is in READY instead of the NULL state.
You need to explicitly set elements to the NULL state before
dropping the final reference, to allow them to clean up.
This problem may also be caused by a refcounting bug in the
application or some element.

EDIT: Note that the last error repeats until liquidsoap crashes

dikkedimi commented 3 years ago

I found this topic https://github.com/savonet/liquidsoap/issues/692 but I couldn't really follow what I was supposed to do. So I decided to review what I had. Apparently doing things in the morning DOES make a difference 😝

I think I found what caused the issue.

changed StaticFrame = mksafe(single("annotate:width=1920,height=1080:/home/user/assets/graphics/frame.mp4")) to StaticFrame = mksafe(single("/home/user/assets/graphics/frame.mp4")) seems to perform way better. will leave it on for a while and see if it makes it past 24h

and if it's stable we'll try to fix this:

2021/10/19 09:24:48 [/stream:3] Connecting mount /stream for source@10.0.0.3...
2021/10/19 09:24:48 [/stream:2] Connection failed: bad answer: scanf: bad input at char number 0: looking for 'H', found '\021'
2021/10/19 09:24:48 [/stream:3] Will try again in 3.00 sec.

it seems CPU usage is still pretty high

  24048 user    20   0 5686420   4.1g  56048 S 153.3  42.2  28:27.29 liquidsoap                      
  24048 user    20   0 5751956   4.1g  56048 S 187.7  42.5  28:32.94 liquidsoap                      
  24048 user    20   0 5751956   4.1g  56048 S 200.3  42.3  28:38.97 liquidsoap                      
  24048 user    20   0 5784740   4.1g  56048 S 219.7  42.6  28:45.56 liquidsoap                      
  24048 user    20   0 5784740   4.1g  56048 S 193.0  42.7  28:51.37 liquidsoap

And youtube also still whines about the stream's bitrate being too low (2500mbps), it would prefer 4500 mpbs

dikkedimi commented 3 years ago

Since it's still encoding (should it?) for youtube live stream I thought I'd try to adjust my encoder settings to just slip in a copy of the video file, however this example found on the cookbook under shared encoding

enc = %ffmpeg(
  format="flv",
  %audio.copy,
  %video.copy)

causes error:

/sharedencoding-failover.liq 
At line 61, char 27-38:
Error 5: this value has type
  source(, video=?internal(?A),...) (inferred at line 46, char 14-56)
but it should be a subtype of
  source(, video=ffmpeg.video.copy,...)
dikkedimi commented 3 years ago

After an hour of testing liquidsoap starts lagging and the stream starts skipping. It starts saying we must catch up an increasing amount of seconds. CPU usage remains between 100% and 180% I don't hear the computer fan so it's not being pegged, unless linux doesn't know how to cool a mac mini

toots commented 2 years ago

Hi @dikkedimi !

There are pretty high chances that this issue was linked to https://github.com/savonet/liquidsoap/issues/2054, which is fixed now.

Any chance that you could try with the latest v2.0.1-preview? The latest build should be coming up here: https://github.com/savonet/liquidsoap/actions/runs/1496688104

toots commented 2 years ago

I'm gonna go ahead and close this as I believe it is very likely to be the same issue as https://github.com/savonet/liquidsoap/issues/1740. Please re-open if needed. Thks!