`./livepeer -orchestrator -transcoder` process dies, reporting `Killed` (possible memory leak)

[chrishobcroft]

Bug Description

When operating livepeer in -orchestrator -transcoder mode, with a B sending it a single stream, the process consumes all RAM on the host, reporting Killed as the process fails.

Observed Behaviour

Steps to reproduce the behavior:

1. Download version 0.5.27 of Livepeer wget https://github.com/livepeer/go-livepeer/releases/download/v0.5.27/livepeer-linux-amd64.tar.gz
2. Unzip tar -xzf livepeer-linux-amd64.tar.gz
3. start O/T ./livepeer -orchestrator -transcoder -serviceAddr 127.0.0.1:8935 -v 99
4. start B ./livepeer -broadcaster -httpAddr :8936 -cliAddr :7936 -rtmpAddr :1936 -orchAddr 127.0.0.1:8935 -transcodingOptions P720p30fps16x9,P576p30fps16x9,P360p30fps16x9,P240p30fps16x9,P144p30fps16x9 -v 99
5. start ffmpeg to publish into the B ffmpeg -re -f lavfi -i testsrc=size=1920x1080:rate=30,format=yuv420p -f lavfi -i sine -c:v libx264 -b:v 10000k -x264-params keyint=60 -c:a aac -f flv rtmp://127.0.0.1:1936/test_source
6. run the following to confirm that transcoding is happening: ffplay http://localhost:8936/stream/test_source/P144p30fps16x9.m3u8
7. observe multiple renditions being served by the B
8. Observe the Memory consumption of the O process start increasing at a rate of ~300Mb every 2 seconds.
9. Observe the O process terminating ungracefully, reporting "Killed".

Expected Behavior

I expect the O process to release Memory once it has transcoded and served content back to the B.

Screenshots

Screenshot from System Monitor application on Ubuntu:

Environment

• Livepeer Version: 0.5.27

Additional Comments

Splitting the O/T into O and T

When running with a split O and T setup, the issue also manifests, with the T process being the one Killed.

B: ./livepeer -broadcaster -httpAddr :8936 -cliAddr :7936 -rtmpAddr :1936 -orchAddr 127.0.0.1:8935 -transcodingOptions P720p30fps16x9,P576p30fps16x9,P360p30fps16x9,P240p30fps16x9,P144p30fps16x9 O: ./livepeer -orchestrator -orchSecret hello -serviceAddr 127.0.0.1:8935 T: ./livepeer -transcoder -orchAddr 127.0.0.1:8935 -orchSecret hello -cliAddr :7937 P: ffmpeg -re -f lavfi -i testsrc=size=1920x1080:rate=30,format=yuv420p -f lavfi -i sine -c:v libx264 -b:v 10000k -x264-params keyint=60 -c:a aac -f flv rtmp://127.0.0.1:1936/test_source

Previous versions

Reproduced on the following versions of Livepeer:

0.5.26 0.5.25 0.5.24 0.5.23 0.5.22 [stopped testing]

System Monitor charts running 0.5.22:

[cyberj0g]

I confirm that there seem to be a leak - checked with current master of go-livepeer and lpms. RAM usage rises fast in first 30 sec and then growth slows down at about 4 Gb, though not stops.