search

matrix-org / matrix-ircd

An IRCd implementation backed by Matrix.
Apache License 2.0
224 stars 41 forks source link

*Way* too much opening/closing of /etc/hosts and ca-certificates.crt #45

Open madduck opened 7 years ago

madduck commented 7 years ago

I just noticed that my matrix-ircd, which I run from HEAD (release build) is consistently using up 100% of the CPU. With strace, I found that it seems to be in an endless loop reading /etc/ssl/certs/ca-certificates.crt and /etc/hosts, or maybe it's opening and closing these files for every single data packet, or somesuch. Check out for example this strace output of 10 seconds, while the IRC client reports almost no traffic: strace-open+close.gz

madduck commented 7 years ago

I just ran Irssi rawlog next to strace -fe open,close and it seems that matrix-ircd is going crazy on these files, while there are no data packets reaching the client. tcpdump suggests a continuous stream of packets between matrix-ircd and the homeserver, but the homeserver logs do not indicate nearly enough traffic to warrant this chatty behaviour.

Something's not right…

madduck commented 7 years ago

Oh dear, and now I noticed that there are literally thousands of TCP connections between the homeserver and matrix-ircd in TIME_WAIT state. It seems to me that matrix-ircd just keeps making new connections to the homeserver at a rate of several per second!

madduck commented 7 years ago

Another datapoint: I just had to restart matrix-ircd after it crashed (see below), and now it's running just fine, i.e. not hogging the CPU and not going crazy on connections. Wondering what to do to identify whatever triggers this…

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Ssl(ErrorStack([Error { code: 336224321, library: "SSL routines", function: "SSL_CREATE_CIPHER_LIST", reason: "malloc failure" }]))', /checkout/src/libcore/result.rs:860
note: Run with `RUST_BACKTRACE=1` for a backtrace.
cargo run --release -- --url https://matrix.madduck.net --bind [::]:6697      94336,65s user 7325,60s system 42% cpu 66:39:49,06 total