search

evilnet / nefarious2

Nefarious IRCu (Based on IRCu 2.10.12)
Other
38 stars 24 forks source link

Server RAM usage increasing #76

Open ZioNeL opened 4 months ago

ZioNeL commented 4 months ago

HEllo, After 5 days of server start the ircd process is consuming 1.8 GB of RAM. High peak was 5 GB after 30 days. What can be the issue? Running Debian 12 64 bit and Nefarious2 git version. Thank you.

Alpha4615 commented 4 months ago

I have observed this too on a similar set up. I'm trying to isolate what may be triggering this so I wrote a node script that will trigger periodic core dumps once ram usage or rate of increase hits a certain threshold to see if we can figure what is possibly being leaked.

Right now I'm awaiting for the mem runaway to trigger, and I'm afraid that'll be a waiting game since I don't know what might be triggering it.

Is it always reproducible for you? Are you on a network or a single server? Have you noticed when it seems to trigger; perhaps after a user connects and sets a certain user mode?

When the runaway happens, I'll notice the leak is pretty rapid, 10 percentage points of increase in an hour so until I hit the machine cap

On Sat, Feb 24, 2024, 16:28 ZioNeL @.***> wrote:

HEllo, After 5 days of server stats the ircd process is consuming 1.8 GB of RAM. High peak was 5 GB after 30 days. What can be the issue? Running Debian 12 64 bit and Nefarious2 git version. Thank you.

— Reply to this email directly, view it on GitHub https://github.com/evilnet/nefarious2/issues/76, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA252YMJN5IUDC3LZ2MRX5LYVJLQDAVCNFSM6AAAAABDYJOOC6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGE2TENBYHEZDQMQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

ZioNeL commented 4 months ago

We have a small network, 2 main servers and also 3 servers for services. Not so much of activity. What i can tell is that with each day that passes the RAM consumed is more and more high.

jobe1986 commented 4 months ago

Out of curiosity (and in an attempt to rule something out), what distro are you using? and what openssl library version do you have installed?

ZioNeL commented 4 months ago

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm

OpenSSL 3.0.11 19 Sep 2023 (Library: OpenSSL 3.0.11 19 Sep 2023 3.0.11-1~deb12u2

jobe1986 commented 4 months ago

Thanks, that ruled out the wild goose I was chasing.

rubinlinux commented 4 months ago

Does the increase in memory use happen immediately after you restart, or does it run for a while with very little, and then start to get bigger after some triggering event?

ZioNeL commented 4 months ago

is growing little by little, for example 3 days ago RAM usage was 1.84 GB, now is 2.74GB

ZioNeL commented 3 months ago

I`ve noticed that RAM increase is only on running servers where services are linked. Due to lots of net splits. Everytime a netsplit is taking place also ram increase is happening.

Alpha4615 commented 1 month ago

@ZioNeL Just to update you, I'm still waiting for my server to go into a mem-leak state. I have a system that monitors the RAM usage, and if it exceeds a certain threshold or increases too quickly, it generates memory dumps.

Sadly, since we don't know what triggers it, it's a waiting game.