Possible Memory Leak / server crash

[lbenicio]

Describe the bug 3 times a day the server start increasing memory used until the server crash, when I start it manually the server uses 10% of my ram (1GB) but after a few hours it starts eating memory. (i have 2 instances with +/- 30 players each)

To Reproduce I could not found any

Screenshots https://imgur.com/a/FuumL52

Logs https://codeshare.io/Gq3V9R (15mb file)

Environment information: Operating system: ubuntu x64 TeaSpeak version: TeaSpeak-1.2.9-beta1

edit: server version

Extra info: I have a bot running on the server making queries each minute.

[mrc4tt]

1.1.40-beta1 ? - do you have not updated to 1.2.9-beta1 (i just asking) ;)

[lbenicio]

i updated, forgot to edit this part

[mrc4tt]

I can't not 100% confirm about the RAM, since i'm running (webserver/mailserver) on the same server. but i'm using: 1.61GB out of 30GB (VPS of course)

[lbenicio]

TeaSpeak was using 48% of my RAM with a 1gb VPS from DigitalOcean, it keeps increasing until i receive a 80% alert and then a few minutes later server crashs

[mrc4tt]

But i can confirm. the original TeamSpeak don't use so much CPU (of %) (like 1% but only with: 40 users) Dunno about it's a problem with TeaSpeak. (my is using: 15-40% CPU with 22 users). but i don't know with RAM. maybe 512MB. but it's most weird with CPU using.

[WolverinDEV]

Yeah you're right there was a huge memory leak in there :) Its fixed within the next version :)

[lbenicio]

CPU utilization is low, like 1-2%, just RAM got very high. as of now TeaSpeak is using 4.36% of CPU and 47,27% of RAM, with a 1gb vps with 1 core

[lbenicio]

maybe it stil happening in the last version, memory utilization still increase over time: https://imgur.com/a/QjPocFE

i'm using 1.2.10-beta-1

edit: screenshot url

[dretax]

thts | 5.11 GB | ./TeaSpeakServer

xD. Im also checking it out with the update.

[WolverinDEV]

Hmm well CPU seems to be okey. The memory not really. I'll look at this may later but I thing a critical memory bug is already out so it should be definitely not grow so fast anymore :)

[lbenicio]

still happening on 1.2.10-beta-5

https://imgur.com/a/Uz1avPq

see in screenshot how the RAM graph only increase over time

[WolverinDEV]

Lovely... (ironic) I'll look at that

[dretax]

Also would like to confirm.

Atleast It's not 5gb.

[Erza]

Can confirm, there's still a leak in 1.2.10-beta-5, been steadily climbing over a 24h period till it runs out of memory, at which point the CPU usage goes to 100% and the servers lag because it spams this error [2018-08-20 22:04:26] [DEBUG] 36 | Cant decrypt packet of type Command. Dropping packet. Error: memory verify failed! I run TeaSpeak on a Debian VM with 2GB memory assigned, so it takes less than 24h to fill up and shit itself <_>

https://pastebin.com/jTim2NTS happened when I tried to stop the server.

[dretax]

Size is on 3.90GB now. It increases slowly.

[lbenicio]

Use a script to restart the server each 24h, on a time that nobody is using, or fewer people are using. i'm doing that, like 5:30 AM, and manage to avoid crashes

[dretax]

This will be my last report about the 1.2.10 testing. The size is on 4.45 GB now. Seems like 10 hours = 0.55 GB, which is 550mb / 10 hours. So every 1 hour = 55MB + usage based on my server. This might help.

[Erza]

The memory increase depends on activity on the servers, I had a server idling with no users on another machine and one with 19 running virtual servers with user activity, and the idling one didn't suffer from memory leakage at all while the active one does alot.

[dretax]

I don't know if that's true. I have 3 virtual servers running with maximum 1-2 guys on 1 virtual server or 0. The other 2 is not being used. The reason of this is because I am currently also at an AHTP, and I wanted to see how the server goes with 0 usage.

[mrc4tt]

On the teamspeak3. - on the latest system requirements for TS3 server: Total memory usage depends on number and size of hosted virtual servers. (Maybe it's same with TeaSpeak?)

[Erza]

Got a dump that's probably related to this https://transfer.sh/XyWdW/crash_dump_bbd503d2-be48-4c6c-6a54bfad-53323223.dmp

@WolverinDEV

[Erza]

Got 2 more crashes shortly after each other, dunno if it's related to this mem leak:

https://transfer.sh/g78nP/crash_dump_211a08d5-a9d7-45a3-d6fafbb8-9a8fccfa.dmp https://transfer.sh/FjCH3/crash_dump_dcb9bd5c-1430-419f-e885c48e-206e4167.dmp

[WolverinDEV]

All three crashes had the same reason. Should be fixed within the next version :)

[lbenicio]

Still happening on 1.2.12-beta

See how a.usage increase over time, at the moment of the screen shot, using 250mb

https://imgur.com/gallery/mhRNoks

[WolverinDEV]

Hey, I'lve never said that the "memory leak" got better. Just the crashes :)

[mrc4tt]

Btw. your TeaSpeakServer is only crashing. because:

Low MEMORY or less cores on server (as example 1GB RAM or 2 cores) - You need to know. = If there are more users and more servers, then it requires more RAM/cores.

I'm using a 6cores with 32GB RAM, and got no issues! :)

[WolverinDEV]

Well i have a short question for all users which executes TeaSpeak with a large amount of RAM. Do you see any kind of limit? Or does the server grows continuously? (Keep in mind when you're searching for a kind of limit that this is strong connected with the amount of users)

I guess its not a memory leak. It seems to be more something like a memory fragmentation.

[lbenicio]

But it is strange, I have 2 servers, one has like 20 users and the other 50. So the worst case it would be 70 users connected. I have watched the growths graphs and they start at 6% (when the server starts) of RAM and keep growing, independent of the number of users connected. At the moment of this comment, I have 28 connected users and servers is using 350mb RAM

[Erza]

I increased the memeory of my VM from 2GB to 8GB now, will report back.

[WolverinDEV]

Yeah it was a memory fragmentation bug. I changed the memory allocator to jemalloc and it should work now :) (1.2.13b)

[Erza]

Testing on 1.2.13 now, will monitor usage and report back.

[dretax]

+1

[Erza]

Seems to have gotten alot better with 1.2.13, good job.

[dretax]

Ram usage seems to start at 685 MB, after 15 hours It is using 1.39GB.

[WolverinDEV]

Well start with 685MB is huge. I guess you have a lot of virtual servers. And 1.4GB is still acceptable when you have a lot of servers and users. The basic reason why this spreads a little bit is memory fragmentation, but the new memory allocator handle that much better.

[mrc4tt]

No issues here. ;) it's works pretty cool.

I only using 2GB out of 32GB because webserver/mailserver is running too. :) . if both was offline - the memory will be: 1.50GB out of 32GB. :)

[lbenicio]

Much better now but are this numbers normal? i got like 900mb ram with 30 clients connected to 1 server. After 1.12.13 it starts increasing but much slower. What is the recommended setting for like 100 client, did it stop increasing after a certain amount (like if i upgrade to a 2gb server)?

[dretax]

1.84GB now. @WolverinDEV And no. I have 3 virtual servers, which of 2 running empty. The first one has like 4-5 people on. The memory still increases, slower, and the huge memory leak is gone, but not fully. I will report back once more tomorrow. https://i.imgur.com/oDew3aZ.png

[lbenicio]

https://imgur.com/a/kOOt4Ub

here a chart of memory usage, currently using 240mb with 2 servers, no clients connected. apparently the RAM usage has nothing to do with the number of client connections. it spiked to 850mb of RAM at 12:39 AM (which had only 10 users connected to only 1 server, the other was without users.

[WolverinDEV]

1.84GB for just one server is intense. You re sure that you're not compeering VIRT to RES?

[lbenicio]

Still leaking on 1.2.13-beta-4 | ubuntu x64. it started leaking 3mb/hours. After 5 hours, 10mb/hours of leaking, after 6 hours, 15mb leaking. It is slightly increasing over time and increase the increase rate over time started leaking 3mb/hours and now it on 15mb/hours. The last 3 hours had constant 18 users connected and the memory usage increase in 25mb.

if it is interesting I can plot a graph of CPU, Memory and users online and all correlation between CPU x memory and users online.

here is the data in a table:

time	RAM (mb)	CPU (%)	users
hour 1	44,31	1,19	10
hour 2	47,77	1,61	10
hour 3	49,12	1,1	9
hour 4	51,40	1,29	11
hour 5	56,32	2,08	17
hour 6	63,87	2,36	18
hour 7	79,38	4,00	18
hour 8	66,26	4,59	19
hour 9	103,95	2,68	19
hour 10	128,11	2,98	20
hour 11	155,96	7,74	22

see? it is slowly increainsg the usage and as the time pass it increases faster

after 24hours i submit a new review with a chart

[lbenicio]

After 24h server was using 450mb RAM without users connected to it.

here is the charts: https://imgur.com/a/HpoSdVV

[lbenicio]

did it stop for you guys? what you hvae done? i'm on 1.2.15-beta-6 and still leaking 500mb/day. i upgraded my server to 4gb RAM but it wont least long, at the moment, it is using 2437mb.

[WolverinDEV]

I recommand you to use beta 8 (There is may a slight enchantment :))

[dretax]

The actual ram usage stayed on 2gb or so, but why does the "size" of the process keep growing? xd

https://i.imgur.com/y9os14h.png