I'm running on my Raspberry Pi 2 eight ices0 streams of 320kbps mp3 without reecoding the files on the fly (they have all been reencoded in this format . With this setup all the streams together use only 43MB of RAM and 2% of CPU.
As soon as I realized that the set of ices0 streams sometimes increases its RAM usage until the machine becomes unresponsive, I wrote a script that periodically checks how much memory it is using, and if it exceeds a threshold, it kills and restarts all the streams, which consume again 43MB together.
I obtain the value in MB from this command ps axo rss,comm | awk -v CONVFMT='%.0f' '/ices0/ {$1/=1024;sum+=$1} END {print sum""}'.
I've also tried to see what changes in memory usage when there is a normal and an excessive RAM usage by comparing the smaps, but they only differ in 4 lines for few kB. I read per-stream smaps with sudo cat /proc/$pid/smaps.
Do you know some other places where I can look the composition of the memory usage when it is excessive? Can this behaviour be a consequence of a memory leak? Why does it only start after some hour the stream have been restarted and not immediatly after?
Could you add a timestamp to each line of the ices0 log, both verbose and not?
EDIT
I was able to find out that only a stream is guily of today fail. It disappeared from the list of Icecast streams in status-json.xsl and its mountpoint returned 404, but according to the logs and the time it took, the stream waited for the song to finish before serving the following track.
Screenshot from htop:
With sudo pmap $pid I found a difference between the normal and the excessive usage at the beginning of the file:
normal
Hexdumping the last line outputs hundreds of MB of junk.
The stream log file looks like this:
[...]
# Start of the strange behaviour
DEBUG: Builtin playlist handler serving: /path/to/Artist - Title.mp3
DEBUG: Filename cleaned up from [/path/to/Artist - Title.mp3] to [Artist - Title]
DEBUG: Trimmed short frame (301 bytes missing) at offset 109247768
DEBUG: MPEG-1 layer III, 320 kbps, 44100 Hz, j-stereo
DEBUG: Ext: 0 Mode_Ext: 0 Copyright: 0 Original: 1
DEBUG: Error Protection: 0 Emphasis: 0 Padding: 0
Playing /path/to/Artist - Title.mp3
Updating metadata on /mountpoint failed.
DEBUG: Done sending
# Other tracks show a log like this until the script kills the source
DEBUG: Builtin playlist handler serving: /path/to/Artist2 - Title2.mp3
DEBUG: Filename cleaned up from [/path/to/Artist2 - Title2.mp3] to [Artist2 - Title2]
DEBUG: ID3v2: version 4.0. Tag size is 332806 bytes.
DEBUG: ID3v2: Title found: Title2
DEBUG: ID3v2: Artist found: Artist2
DEBUG: MPEG-1 layer III, 320 kbps, 44100 Hz, j-stereo
DEBUG: Ext: 0 Mode_Ext: 2 Copyright: 0 Original: 1
DEBUG: Error Protection: 0 Emphasis: 0 Padding: 0
Playing /path/to/Artist2 - Title2.mp3
Updating metadata on /mountpoint failed.
DEBUG: Done sending
[...]
* Stream is automatically stopped and manually restared with a new playlists that begins with the new "faulty" track*
# Begin of streaming
DEBUG: Builtin playlist handler serving: /path/to/Artist - Title.mp3
DEBUG: Filename cleaned up from [/path/to/Artist - Title.mp3] to [Artist - Title]
DEBUG: Trimmed short frame (301 bytes missing) at offset 109247768
DEBUG: MPEG-1 layer III, 320 kbps, 44100 Hz, j-stereo
DEBUG: Ext: 0 Mode_Ext: 0 Copyright: 0 Original: 1
DEBUG: Error Protection: 0 Emphasis: 0 Padding: 0
Playing /path/to/Artist - Title.mp3
Mounted on http://127.0.0.1:4040/mountpoint
DEBUG: Delaying metadata update...
DEBUG: Updated metadata on /mountpoint to: Artist - Title
DEBUG: Updated metadata on /mountpoint to: Artist - Title
[...]
The .mp3 files are fine, I can't understand why it can now mount and update the metadata if it failed some minutes ago. The timestamps would confirm if this strange behaviour started with the "Artist - Title" track, or if it had already begun.
I'm running on my Raspberry Pi 2 eight ices0 streams of 320kbps mp3 without reecoding the files on the fly (they have all been reencoded in this format . With this setup all the streams together use only 43MB of RAM and 2% of CPU.
As soon as I realized that the set of ices0 streams sometimes increases its RAM usage until the machine becomes unresponsive, I wrote a script that periodically checks how much memory it is using, and if it exceeds a threshold, it kills and restarts all the streams, which consume again 43MB together.
I obtain the value in MB from this command
ps axo rss,comm | awk -v CONVFMT='%.0f' '/ices0/ {$1/=1024;sum+=$1} END {print sum""}'
.I've also tried to see what changes in memory usage when there is a normal and an excessive RAM usage by comparing the smaps, but they only differ in 4 lines for few kB. I read per-stream smaps with
sudo cat /proc/$pid/smaps
.Do you know some other places where I can look the composition of the memory usage when it is excessive? Can this behaviour be a consequence of a memory leak? Why does it only start after some hour the stream have been restarted and not immediatly after?
Could you add a timestamp to each line of the ices0 log, both verbose and not?
EDIT I was able to find out that only a stream is guily of today fail. It disappeared from the list of Icecast streams in
status-json.xsl
and its mountpoint returned 404, but according to the logs and the time it took, the stream waited for the song to finish before serving the following track.Screenshot from
htop
:With
sudo pmap $pid
I found a difference between the normal and the excessive usage at the beginning of the file:normal
excessive
This is the output of
sudo cat /proc/$pid/maps
:Hexdumping the last line outputs hundreds of MB of junk.
The stream log file looks like this:
The .mp3 files are fine, I can't understand why it can now mount and update the metadata if it failed some minutes ago. The timestamps would confirm if this strange behaviour started with the "Artist - Title" track, or if it had already begun.