search

savonet / liquidsoap

Liquidsoap is a statically typed scripting general-purpose language with dedicated operators and backend for all thing media, streaming, file generation, automation, HTTP backend and more.
http://liquidsoap.info
GNU General Public License v2.0
1.4k stars 129 forks source link

Memory leak on windows using telnet commands #1122

Open ignaciovenezia opened 4 years ago

ignaciovenezia commented 4 years ago

Hello! I've had this issue of liquidsoap memoryleaking for a while and tried to fix it to no avail. Whilst trying to pinpoint why was I having memory issues, I've come to the conclusion that using telnet commands, on windows, is the cause.

Why is this I'm not sure, I'm quite new to liquidsoap so know for sure.

Testing script to replicate: https://pastebin.com/VhdjAqus

I interact with liquidsoap through telnet from my own application. I have a service that sends the tcp commands "decoder.status" and the custom command "lm" every second to check the status of the decoder and the latest title provided by the notify_metadata procedure. I then manage a secondary process based off the result of these tcp commands.

If my approach is very wrong, I ask of you to guide me in a better direction.

Anyway, whilst searching for an answer I stumbled upon issue #622 in which in a response @toots asked this very question, so I guess there must be something to it.

Some more information:

It can get very wild

If this is entirely my fault, don't hesitate to call me on it. Tough love works.

Thanks.

toots commented 4 years ago

Thanks for this report. I'll have another try at it.

ignaciovenezia commented 4 years ago

Thanks for this report. I'll have another try at it.

Thanks for looking into it. Whilst looking for a non-memoryleak way of interacting with the server I found out that using http harbor also leaks in the same way.

def get_meta(~protocol,~data,~headers,uri) =
  m = "asd"
  http_response(
    protocol=protocol,
    code=200,
    headers=[("Content-Type","application/json; charset=utf-8")],
    data=json_of(m)
  )
end

harbor.http.register(port=7000,method="GET","/getmeta",get_meta)

Using this, which is pretty much the code on the docs, every time I do an http request to localhost:7000/getmeta memory usage increases and it never goes down.

I tried calling 'garbage_collect()' in the script in many different locations with no effect whatsoever in the memory usage.

edit: Somewhere in the docs (i forgot where) says that the server stores the response in memory before sending it.
My guess is that memory is never being freed.

pranza commented 4 years ago

hi! opening a telnet session is a guaranteed way to start a memleak, but even without that it will start leaking. for example i use a script which kills liquidsoap every morning (it can't exit gracefully sometimes due to internal corruption), and even then I have around 1,5GB of leaked memory per day per instance, even a simplest radio. Romain has suspected .flac support being the culprit and has made some builds without it, yet i don't remember how it ended as for me liquidsoap without flac is useless. anyway, this doesn't really explain the telnet memleak... but you can try toying around with some liquidsoap build without flac support - maybe it doesn't start leaking if you don't touch telnet.

we tried debugging a year or two ago but it didn't go far - the dump files didn't help.

toots commented 4 years ago

I just committed a fix for a memleak with flac. It seems far-fetched but I'll publish a new build of liquidsoap when the bugfix release 1.4.2 is out and we'll see from there..

pranza commented 4 years ago

happy easter!

cool, looking forward to it!! an easy check would be opening a telnet connection and seeing if mem usage starts to grow from there.. without that it takes some time to start, has to run for a while..

regards, p

toots commented 4 years ago

Hey there! We just published liquidsoap 1.4.2 here with updated windows build: https://github.com/savonet/liquidsoap/releases/tag/v1.4.2

There were some changes in this release addressing memory leaks with ocaml-flac. Not sure if that would apply to y'all issues but it would be nice if y'all could test it! Make sure to check the release notes for the breaking change with cross/crossfade..

pranza commented 4 years ago

hi Romain!

wow, thanks! running already. no changes to my configs were needed, weeee! for now i've disabled automatic killing every morning - let's see if this build behaves!

regards, Pranas

-----Original Message----- From: Romain Beauxis Date: 2020 m. gegužės 3 d. 22:29 To: savonet/liquidsoap Cc: pranza ; Comment Subject: Re: [savonet/liquidsoap] Memory leak on windows using telnet commands (#1122)

Hey there! We just published liquidsoap 1.4.2 here with updated windows build: https://github.com/savonet/liquidsoap/releases/tag/v1.4.2

There were some changes in this release addressing memory leaks with ocaml-flac. Not sure if that would apply to y'all issues but it would be nice if y'all could test it! Make sure to check the release notes for the breaking change with cross/crossfade..

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

pranza commented 4 years ago

hey ho! so the first impressions... in the morning one of two radios were already dead, one of processes showing impossible to end "suspended" status and 20 KB of memory. the other one was bloated till 800 MB but still working and growing.

after restarting processes few hours ago, both are still working, one having eaten 240MB of ram already, and the other at 85MB (95MB already by the end of writing this email hehe).

looks like things are still the same.. telnet is not even needed to start a memleak.

regards, p

toots commented 4 years ago

Well, I'm really sorry to hear that. I'll resume the work on this as soon as time allows.

toots commented 4 years ago

TODO on this issue: rebuild with only: ogg flac, ogg vorbis, mp3 (mad & lame) and send back to the reported for testing.

toots commented 4 years ago

Note: the build at https://github.com/savonet/liquidsoap/actions/runs/180097192 didn't fix the issue.

pranza commented 4 years ago

yes, i didn't even have to try accessing it via telnet, mem usage grew to ~400 MB the same evening for both radios.

p

On 24 Jul 2020, at 15:14, Romain Beauxis notifications@github.com wrote:

Note: the build at https://github.com/savonet/liquidsoap/actions/runs/180097192 didn't fix the issue.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

toots commented 4 years ago

Thanks for the script @ignaciovenezia that's very useful. I'm trying to reproduce both on linux and windows but haven't been able so far.

Couple of questions:

So far, I've been running a version of the script with output.ao instead of output.pulseaudio and also the input.http isn't working since the localhost isn't setup. I kept connecting via telnet and typing commands but the memory usage doesn't seem to be blowing up.

toots commented 4 years ago

Actually it also doesn't seem to blow up the memory when using ouput.pulseaudio so probably unrelated.

pranza commented 4 years ago

hi Romain! have you tried any of my scripts?

p

toots commented 4 years ago

I've been experimenting with @ignaciovenezia's script first. What I can report:

That gives me a pretty good idea about the part of the code to look at. Let's see if we can make more progress on this one!

pranza commented 4 years ago

hello!

hmm, my issue is a bit different - the mem never goes down and telnet is not necessary to start the leak, it's just a quick way to introduce it after a fresh start...

that's the stats for today since killed and restarted at 8AM as every day: first radio - 100 megs used so far second radio - 500 megs used so far and at least one of them will be around two gigs in the evening

regards, p

toots commented 4 years ago

I hear you. The two issues may or may not be related but I'd like to tackle them one by one to separate my concerns. I'll fix the one I can reproduce first, then we'll see if it fixes yours and, if not, back at the drawing board!

toots commented 3 years ago

Hi again! I realized that the build image we use for windows binaries was ~2 years old. I've just finished a rebuild using the latest image. Any chance you could try and see if the bug is still present in that build? I'm attaching it to this comment. Please note that opus was disabled in this build as it failed to link for some reason that I need to investigate..

Build is here: liquidsoap-1.4.3-win64.zip

pranza commented 3 years ago

hi Romain! oh, the .exe size increase is huge :) just tried running it.

it doesn't run properly... first it complained not having rights to access the log (although it would actually write to it).. when i took ownership of the log, it started complaining that 'grep' doesn't work:

liquidsoap.exe muilas.liq 'grep' is not recognized as an internal or external command, operable program or batch file. init: exception encountered during main phase: Invalid_argument("Sys.signal: unavailable signal") exception: Invalid_argument("Sys.signal: unavailable signal") Raised by primitive operation at file "stdlib/sys.mlp", line 81, characters 39-6 3 Called from file "main.ml", line 625, characters 8-52 Called from file "dtools.ml", line 556, characters 16-20

also it put some interesting lines in the log: 2020/11/07 11:18:18 [sandbox:3] Could not find binary bwrap, disabling sandboxing.. 2020/11/07 11:18:18 [video.converter:3] Couldn't find preferred video converter: gavl.

that's my experience so far :) going back to the old leaky build for time being.

regards, p

-----Original Message----- From: Romain Beauxis Date: 2020 m. lapkričio 6 d. 23:58 To: savonet/liquidsoap Cc: pranza ; Comment Subject: Re: [savonet/liquidsoap] Memory leak on windows using telnet commands (#1122)

Hi again! I realized that the build image we use for windows binaries was ~2 years old. I've just finished a rebuild using the latest image. Any chance you could try and see if the bug is still present in that build? I'm attaching it to this comment. Please note that opus was disabled in this build as it failed to link for some reason that I need to investigate..

Build is here: liquidsoap-1.4.3-win64.zip

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

toots commented 3 years ago

My bad, there was an issue with the previous build I'm attaching a new one here. Would you mind testing again? Thks! liquidsoap-1.4.3-windows64.zip

pranza commented 3 years ago

Thanks Romain!! Now this one starts and runs. I've been able to kickstart memleak by opening and closing telnet to the process, but i'll also try to see how it behaves if i don't touch telnet at all...:)

regards, p

-----Original Message----- From: Romain Beauxis Date: 2020 m. lapkričio 9 d. 20:21 To: savonet/liquidsoap Cc: pranza ; Comment Subject: Re: [savonet/liquidsoap] Memory leak on windows using telnet commands (#1122)

My bad, there was an issue with the previous build I'm attaching a new one here. Would you mind testing again? Thks! liquidsoap-1.4.3-windows64.zip

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

toots commented 3 years ago

Ha darn. I'm working on a port of OCaml 4.11.1 for windows cross-compilation. Maybe that could have fixed the issue?

pranza commented 3 years ago

hmm i don't know but i'm happy to run the binary if you made one :)

i also have cygwin environment on that same machine if that helps... to test some half-baked versions maybe?

regards, p

On 10 Nov 2020, at 16:11, Romain Beauxis notifications@github.com wrote:

Ha darn. I'm working on a port of OCaml 4.11.1 for windows cross-compilation. Maybe that could have fixed the issue?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

toots commented 3 years ago

Hi! I made a build of 1.4.3 with OCaml any chance you could have a second to try it as well? It's attached here. liquidsoap-v1.4.3-windows-ocaml-4.11.1-win64.zip

pranza commented 3 years ago

hi Romain and thanks for yet another one!

ran this since morning - still leaks after a while. opening telnet triggers a leak instantly...

regards, p

-----Original Message----- From: Romain Beauxis Date: 2020 m. lapkričio 17 d. 03:04 To: savonet/liquidsoap Cc: pranza ; Comment Subject: Re: [savonet/liquidsoap] Memory leak on windows using telnet commands (#1122)

Hi! I made a build of 1.4.3 with OCaml any chance you could have a second to try it as well? It's attached here. liquidsoap-v1.4.3-windows-ocaml-4.11.1-win64.zip

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.