search

savonet / liquidsoap

Liquidsoap is a statically typed scripting general-purpose language with dedicated operators and backend for all thing media, streaming, file generation, automation, HTTP backend and more.
http://liquidsoap.info
GNU General Public License v2.0
1.41k stars 130 forks source link

Upgrade to 2.1.0 fails to resolve external URLs (m3u, mp3) #2557

Open frytg opened 2 years ago

frytg commented 2 years ago

Describe the bug When upgrading from v2.0.7 to v2.1.0 loading mp3 urls from a https://.../something.m3u playlist (audio/x-mpegurl) no longer works and results in a loop of revolving assets / [procol.external:3] Error while fetching http data: 522 - Connection timed out

To Reproduce We're using this to load an m3u playlist from an external URL (via https://)

base = playlist(id="primary",
  mode="normal",
  reload_mode="seconds",
  reload=600,
  mime_type="audio/x-mpegurl",
  native=true,
  timeout=600.0,
  getenv("PLAYLIST_URL"))

One error we witnessed was this:

[request:5] Resolving request [[https://redacted.de/playlist10.m3u]].
[request:5] Resolved to [[/tmp/liq-processbb038d.osb], [https://redacted.de/playlist10.m3u]].
[request:5] Resolving request [[https://storage.googleapis.com/redacted.mp3?signed-hashes]].
[request:4] Failed to resolve "https://storage.googleapis.com/redacted.mp3?signed-hashes"! For more info, see server command `request.trace 2`.
[request:5] Resolved to [].

or 

[request:5] Resolving request [[https://redacted.de/playlist10.m3u]].
[request:5] Resolving request [[???failed???]].
[request:3] Nonexistent file or ill-formed URI "???failed???"!
[request:5] Resolved to [].

Downgrading back to v2.0.7 with the same settings and debug level showed the correct workflow:

[request:5] Resolving request [[https://redacted.de/playlist10.m3u]].
[request:5] Resolved to [[/tmp/liq-process1e3102.osb], [https://redacted.de/playlist10.m3u]].
[request:5] Resolving request [[https://storage.googleapis.com/redacted.mp3?signed-hashes]].
[request:5] Resolved to [[/tmp/liq-processe46557.mp3], [https://storage.googleapis.com/redacted.mp3?signed-hashes]].

Do you have any suggestions why this happens? The mp3 files are fairly large (~ 1-2 GBs), but we rarely had any issues and the 522 errors get thrown every ~30sec, so it doesn't even wait the full 600 sec timeout. We tried both native=true, and false. Could this due to the internal changes from request.dynamic.list to request.dynamic?

Version details v2.0.7/ v2.1.0 from Dockerhub

frytg commented 2 years ago

I spent a lot of time debugging and wrote a custom https resolver. This one shows, that the initial playlist10.m3u request has a timeout that corresponds to our config. The next call for the mp3 file only has ~30sec.

Could this be due to this (which begs the question how it ever worked for us)? https://github.com/savonet/liquidsoap/blob/5edc4d24627fcb9c6c68f427db67c9154bf53807/src/core/sources/harbor_input.ml#L315-L318

https://github.com/savonet/liquidsoap/blob/a2d40a89d3a13fe8a9a594040dfb17c919e0e0d1/src/core/builtins/builtins_request.ml#L72-L75

Shouldn't this one be overwrite-able with settings?

settings.harbor.timeout.set(600.)
toots commented 2 years ago

The timeout for playlists is set via its own timeout parameter. There's been some change in libcurl recently that seem to have broken some backward compatibility, see: https://github.com/savonet/liquidsoap/issues/2551 it's possible the default timeout for also was changed there.

toots commented 2 years ago

Hey! If you could send my the playlist I could try to reproduce locally.. email: toots@rastageeks.org