search

savonet / liquidsoap

Liquidsoap is a statically typed scripting general-purpose language with dedicated operators and backend for all thing media, streaming, file generation, automation, HTTP backend and more.
http://liquidsoap.info
GNU General Public License v2.0
1.39k stars 126 forks source link

Dynamic Text Issue with video.add_text.ffmpeg.raw.filter() #3550

Closed bradenfallon closed 9 months ago

bradenfallon commented 9 months ago

Describe the bug There appears to be a bug in the video.add_text.ffmpeg.raw.filter() function, which correctly processes static text but generates incorrect output if the input is dynamic (such as changing metadata or a counter). I've included two lines of codes in one of my functions: 

print("The current song is: " ^ "\n" ^ "#{nowplaying()}")
print("#{format_time(source.remaining(radio))}")

These output in the Liquidsoap log and show that the functions intially generating this data are running correctly. However, the issue arises after they are passed into video.add_text.ffmpeg.raw.filter().

To Reproduce

# Define Time Format and Percent
def format_time(time)
    seconds = int(time mod 60.)
    seconds_string =
        if seconds < 10 then
            "0" ^ string(seconds)
        else
            string(seconds)
        end
    minutes = int(time / 60.)
    minutes_string =
        if minutes < 10 then
            "0" ^ string(minutes)
        else
            string(minutes)
        end
    minutes_string ^ ":" ^ seconds_string
end

# Now Playing align text and define font size
np_size = 55
np_x = 75
np_y = 875

# Song Timer align text and define font size
time_size = 55
time_x = 75
time_y = 75

# Add Now Playing metadata
nowplaying = ref("")
def update_nowplaying(m)
    nowplaying := m["artist"] ^ "\n" ^ "------------------------" ^ "\n" ^ "\"" ^ m["title"] ^ "\""
    print("The current song is: " ^ "\n" ^ "#{nowplaying()}")
    print("#{format_time(source.remaining(radio))}")
end
radio.on_metadata(update_nowplaying)

def prepare_source(~now_playing, ~logo, ~audio, background_vid) =
  def mkfilter(graph)
    play_text = ffmpeg.filter.video.input(graph, source.tracks(background_vid).video)
    play_text = video.add_text.ffmpeg.raw.filter(
      font=font_file,
      size=np_size,
      speed=0,
      x=np_x,
      y=np_y,
      graph=graph,
      now_playing,
      play_text
    )

    time_add = video.add_text.ffmpeg.raw.filter(
      font=font_file,
      speed=0,
      size=time_size, 
      x=time_x, 
      y=time_y, 
      graph=graph,
      {"Next in " ^ "#{format_time(source.remaining(radio))}"}, 
      play_text
    ) 

    logo = ffmpeg.filter.video.input(graph, source.tracks(logo).video)
    logo = ffmpeg.filter.scale(graph, logo, w="-1", h="178")

    text_logo = ffmpeg.filter.overlay(graph, time_add, logo,  y="0", x="1780")
    video_out = ffmpeg.filter.video.output(graph, text_logo)

    source({
      video=video_out,
      audio=source.tracks(audio).audio
    })
  end
  ffmpeg.filter.create(mkfilter)
end

# Muxing the text and video streams
logo = single(logo_file)
videostream = single(video_file)
videostream = prepare_source(now_playing={nowplaying()}, logo=logo, audio=radio, videostream)

Logs 

2023/11/21 13:42:44 [local_2:3] Connecting mount /radio-256.mp3 for source@127.0.0.1...
2023/11/21 13:42:44 [ffmpeg_filter_video_input.2:3] Content type is {video=ffmpeg.video.raw}.
2023/11/21 13:42:44 [single.2:3] "/var/azuracast/stations/chillradio/media/videostream/Hipster-Cat-Banner2.png" is static, resolving once for all...
2023/11/21 13:42:44 [decoder.ffmpeg:3] Requested content-type for "/var/azuracast/stations/chillradio/media/videostream/Hipster-Cat-Banner2.png": {video=ffmpeg.video.raw}
2023/11/21 13:42:44 [decoder.ffmpeg:3] FFmpeg recognizes "/var/azuracast/stations/chillradio/media/videostream/Hipster-Cat-Banner2.png" as video: {codec: png, 200x356, pal8}
2023/11/21 13:42:44 [decoder.ffmpeg:3] Decoded content-type for "/var/azuracast/stations/chillradio/media/videostream/Hipster-Cat-Banner2.png": {video=ffmpeg.video.raw(pixel_format=pal8,height=356,width=200)}
2023/11/21 13:42:44 [video.converter:3] Using preferred video converter: ffmpeg.
2023/11/21 13:42:44 [audio.converter:3] Using samplerate converter: libsamplerate.
2023/11/21 13:42:44 [video.text:3] Using sdl implementation
2023/11/21 13:42:44 [lang:3] API nextsong - Sending POST request to 'http://127.0.0.1:6010/api/internal/1/liquidsoap/nextsong' with body: 
2023/11/21 13:42:44 [decoder.ffmpeg:3] Requested content-type for "/var/azuracast/stations/chillradio/media/Chilled Cat - Lofi Non-Copyright/kretzschクレツ,_chilled_cat_-_the_first_winter_of_a_terraformed_mars.mp3": {audio=pcm(stereo)}
2023/11/21 13:42:44 [decoder.ffmpeg:3] FFmpeg recognizes "/var/azuracast/stations/chillradio/media/Chilled Cat - Lofi Non-Copyright/kretzschクレツ,_chilled_cat_-_the_first_winter_of_a_terraformed_mars.mp3" as audio: {codec: mp3, 48000Hz, 2 channel(s)}, video: {codec: mjpeg, 640x640, yuvj444p}
2023/11/21 13:42:44 [decoder.ffmpeg:3] Decoded content-type for "/var/azuracast/stations/chillradio/media/Chilled Cat - Lofi Non-Copyright/kretzschクレツ,_chilled_cat_-_the_first_winter_of_a_terraformed_mars.mp3": {audio=pcm(stereo)}
2023/11/21 13:42:44 [lang:3] API nextsong - Response (200): annotate:title="A passion for relaxation",artist="PGN Music",duration="146.00",song_id="71af43a96ebe83fbc1e1aecf5961f3a1",media_id="531",playlist_id="2":media:PGN Music/a_passion_for_relaxation.mp3
2023/11/21 13:42:44 [decoder.ffmpeg:3] Requested content-type for "/var/azuracast/stations/chillradio/media/PGN Music/a_passion_for_relaxation.mp3": {audio=pcm(stereo)}
2023/11/21 13:42:44 [decoder.ffmpeg:3] FFmpeg recognizes "/var/azuracast/stations/chillradio/media/PGN Music/a_passion_for_relaxation.mp3" as audio: {codec: mp3, 44100Hz, 2 channel(s)}, video: {codec: mjpeg, 640x640, yuvj420p}
2023/11/21 13:42:44 [decoder.ffmpeg:3] Decoded content-type for "/var/azuracast/stations/chillradio/media/PGN Music/a_passion_for_relaxation.mp3": {audio=pcm(stereo)}
2023/11/21 13:42:44 [local_2:3] Connection setup was successful.
[mpegts @ 0xffffadda2400] frame size not set
[mpegts @ 0xffffadda3000] frame size not set
[mpegts @ 0xffffadda3c00] frame size not set
[mpegts @ 0xffffadda4800] frame size not set
2023/11/21 13:42:44 [single.3:3] Prepared "/var/azuracast/stations/chillradio/media/videostream/Hipster-Cat.mp4" (RID 4).
2023/11/21 13:42:44 [next_song:3] Prepared "/var/azuracast/stations/chillradio/media/PGN Music/a_passion_for_relaxation.mp3" (RID 7).
2023/11/21 13:42:44 [lang:3] AutoDJ is ready!
2023/11/21 13:42:44 [single.2:3] Prepared "/var/azuracast/stations/chillradio/media/videostream/Hipster-Cat-Banner2.png" (RID 3).
2023/11/21 13:42:44 [switch.8:3] Switch to source.6.
2023/11/21 13:42:44 [safe_fallback:3] Switch to source.3 with transition.
2023/11/21 13:42:44 [interrupting_fallback:3] Switch to requests_fallback.
2023/11/21 13:42:44 [requests_fallback:3] Switch to autodj_fallback.
2023/11/21 13:42:44 [autodj_fallback:3] Switch to dynamic_startup.
2023/11/21 13:42:44 [dynamic_startup:3] Switch to cue_next_song.
[mp3float @ 0xffffae8c6300] Could not update timestamps for skipped samples.
The current song is: 
PGN Music
------------------------
"A passion for relaxation"
02:26
2023/11/21 13:42:44 [cue_next_song:3] Cueing in...
2023/11/21 13:42:44 [lang:3] API feedback - Sending POST request to 'http://127.0.0.1:6010/api/internal/1/liquidsoap/feedback' with body: {
2023/11/21 13:42:44 [lang:3]   "song_id": "71af43a96ebe83fbc1e1aecf5961f3a1",
2023/11/21 13:42:44 [lang:3]   "playlist_id": "2",
2023/11/21 13:42:44 [lang:3]   "media_id": "531"
2023/11/21 13:42:44 [lang:3] }

Expected behavior This code should generate a video stream that shows the currently playing song metadata, a countdown until the next song is played, and a logo overlay. Instead, the code outputs no metadata, the number 00 instead of a countdown, but the logo is displayed correctly. You can access the livestream to see the output at https://www.youtube.com/watch?v=HPhxFR1Y4MY

Version details Liquidsoap 2.2.2

Install method AzuraCast - v0.19.3 Stable - Docker - PHP 8.2

vitoyucepi commented 9 months ago

Hi @bradenfallon, This is indeed a problem. I've found that the text in the video.add_text.ffmpeg.raw.filter is only evaluated during filter initialization. Do you have any preference against using the video.add_text function?

Also savonet/liquidsoap:v2.2.2 has problems with the video.add_text.ffmpeg.raw.filter function. So I used an ubuntu:22.04 based container.

[drawtext_0 @ 0x7fbc4645dc80] Both text and text file provided. Please provide only one
Unknown position:

Error -1: Exception raised: Avutil.Error(Invalid argument)
Raised by primitive operation at Avfilter.attach in file "avfilter/avfilter.ml", line 247, characters 4-49 
Called from Builtins_ffmpeg_filters.apply_filter in file "src/core/builtins/builtins_ffmpeg_filters.ml", line 353, characters 17-68
Called from Liquidsoap_lang__Evaluation.apply.f in file "src/lang/evaluation.ml", line 174, characters 8-12 
Called from Liquidsoap_lang__Evaluation.eval_term in file "src/lang/evaluation.ml", line 341, characters 10-43
Called from Liquidsoap_lang__Evaluation.eval in file "src/lang/evaluation.ml", line 353, characters 10-38
Called from Liquidsoap_lang__Evaluation.eval_base_term in file "src/lang/evaluation.ml", line 272, characters 16-38
Called from Liquidsoap_lang__Evaluation.eval_term in file "src/lang/evaluation.ml", line 341, characters 10-43
Called from Liquidsoap_lang__Evaluation.eval in file "src/lang/evaluation.ml", line 353, characters 10-38
Called from Liquidsoap_lang__Evaluation.eval_term in file "src/lang/evaluation.ml", line 341, characters 10-43
Called from Liquidsoap_lang__Evaluation.eval in file "src/lang/evaluation.ml", line 353, characters 10-38
Called from Liquidsoap_lang__Evaluation.eval_term in file "src/lang/evaluation.ml", line 341, characters 10-43
Called from Liquidsoap_lang__Evaluation.eval in file "src/lang/evaluation.ml", line 353, characters 10-38
Called from Liquidsoap_lang__Evaluation.apply.f in file "src/lang/evaluation.ml", line 174, characters 8-12 
Called from Liquidsoap_lang__Evaluation.eval_term in file "src/lang/evaluation.ml", line 341, characters 10-43
Called from Liquidsoap_lang__Evaluation.eval in file "src/lang/evaluation.ml", line 353, characters 10-38
Called from Liquidsoap_lang__Evaluation.eval_base_term in file "src/lang/evaluation.ml", line 272, characters 16-38
Called from Liquidsoap_lang__Evaluation.eval_term in file "src/lang/evaluation.ml", line 341, characters 10-43
Called from Liquidsoap_lang__Evaluation.eval in file "src/lang/evaluation.ml", line 353, characters 10-38
Called from Liquidsoap_lang__Evaluation.eval_term in file "src/lang/evaluation.ml", line 341, characters 10-43
Called from Liquidsoap_lang__Evaluation.eval in file "src/lang/evaluation.ml", line 353, characters 10-38
Called from Liquidsoap_lang__Evaluation.apply.f in file "src/lang/evaluation.ml", line 174, characters 8-12 
Called from Stdlib__Fun.protect in file "fun.ml", line 33, characters 8-15 
Re-raised at Stdlib__Fun.protect in file "fun.ml", line 38, characters 6-52 
Called from Builtins_ffmpeg_filters.(fun) in file "src/core/builtins/builtins_ffmpeg_filters.ml", line 879, characters 16-58
Called from Liquidsoap_lang__Evaluation.apply.f in file "src/lang/evaluation.ml", line 174, characters 8-12 
Called from Liquidsoap_lang__Evaluation.eval_term in file "src/lang/evaluation.ml", line 341, characters 10-43
Called from Liquidsoap_lang__Evaluation.eval in file "src/lang/evaluation.ml", line 353, characters 10-38
Called from Liquidsoap_lang__Evaluation.eval_term in file "src/lang/evaluation.ml", line 341, characters 10-43
Called from Liquidsoap_lang__Evaluation.eval in file "src/lang/evaluation.ml", line 353, characters 10-38
Called from Liquidsoap_lang__Evaluation.apply.f in file "src/lang/evaluation.ml", line 174, characters 8-12 
Called from Liquidsoap_lang__Evaluation.eval_term in file "src/lang/evaluation.ml", line 341, characters 10-43
Called from Liquidsoap_lang__Evaluation.eval in file "src/lang/evaluation.ml", line 353, characters 10-38
Called from Liquidsoap_lang__Evaluation.eval_toplevel in file "src/lang/evaluation.ml", line 488, characters 38-46
Called from Liquidsoap_lang__Startup.time in file "src/lang/startup.ml", line 30, characters 12-16
Called from Stdlib__Fun.protect in file "fun.ml", line 33, characters 8-15 
Re-raised at Stdlib__Fun.protect in file "fun.ml", line 38, characters 6-52 
Called from Liquidsoap_lang__Runtime.type_and_run in file "src/lang/runtime.ml", line 30, characters 4-612
Called from Liquidsoap_lang__Runtime.report in file "src/lang/runtime.ml" (inlined), line 215, characters 12-23
Called from Liquidsoap_lang__Runtime.from_lexbuf in file "src/lang/runtime.ml", line 229, characters 2-159
toots commented 9 months ago

@vitoyucepi I'm a bit surprised by this error, we never use this parameter:

          filter=
            ffmpeg.filter.drawtext.create(
              fontfile=font,
              fontsize="#{size}",
              x="#{getter.get(x)}",
              y="#{getter.get(y)}",
              fontcolor=color,
              text=getter.get(d),
              graph
            )

@bradenfallon the reload mechanism is tricky with fffmpeg filters. You need to trigger the reload manually to for a reload of the filter's paramers. If you look at the filter's doc, this is explained:

Display a text. Use this operator inside ffmpeg filters with a ffmpeg video input Returns a ffmpeg video output with on_change and on_metadata methods to be used to update the output text.

You can either call on_change when the text needs to be refreshed or use on_metadat with a specific metadata.

If you don't know when to change, you can hook up the on_change to on_frame as is done in our own use of use it:

  s =
    source.on_frame(
      s,
      fun () ->
        begin
          fn = on_frame()
          fn()
        end
    )

  s = ffmpeg.filter.video.input(graph, source.tracks(s).video)
  v =
    video.add_text.ffmpeg.raw.filter(
      color=color,
      cycle=cycle,
      font=font,
      metadata=metadata,
      size=size,
      speed=speed,
      x=x,
      y=y,
      graph=graph,
      d,
      s
    )

  on_frame := v.on_change