search

karlheyes / icecast-kh

KH branch of icecast
GNU General Public License v2.0
298 stars 106 forks source link

icecast_kh CLOSE_WAIT issues #349

Open AxisNL opened 3 years ago

AxisNL commented 3 years ago

I'm hosting dozens of icecast_kh servers, all without a problem. (Running Icecast 2.4.0-kh15-20210202233630, installed by my control panel).

However, on one server, for one particular client, I see problems where icecast_kh is not closing it's sockets, and this is causing the amount of connections icecast is reporting to go up until the server is full (we limit connections). Part of the output for netstat:

tcp 32 4548 178.132.1.50:8604 173.252.79.24:48152 CLOSE_WAIT tcp 32 2209 178.132.1.50:8604 173.252.107.3:54350 CLOSE_WAIT tcp 32 2153 178.132.1.50:8604 31.13.115.4:45382 CLOSE_WAIT tcp 32 2311 178.132.1.50:8604 173.252.111.24:57788 CLOSE_WAIT tcp 32 1810 178.132.1.50:8604 173.252.95.12:53604 CLOSE_WAIT tcp 32 1713 178.132.1.50:8604 31.13.115.15:38028 CLOSE_WAIT tcp 32 2017 178.132.1.50:8604 173.252.83.5:61214 CLOSE_WAIT tcp 32 2398 178.132.1.50:8604 31.13.115.10:49448 CLOSE_WAIT tcp 32 219 178.132.1.50:8604 31.13.103.19:60726 CLOSE_WAIT tcp 32 2072 178.132.1.50:8604 31.13.115.23:43516 CLOSE_WAIT

Why would icecast_kh keep the sockets open? Now that I look at the ip addresses and do a few whois queries, I see these ip's are almost all hosted by facebook.

Configuration of icecast is written out by control panel, so basically the same across other customers.

Could be related to https://github.com/karlheyes/icecast-kh/issues/152 ?

snahor commented 3 years ago

If the IP addresses are from Facebook, they are probably crawlers and they can stay connected for a long time, just like a regular listener. You may want to limit the time a client stays connected and/or block all those addresses.

For more info about FB addresses check https://developers.facebook.com/docs/sharing/webmasters/crawler/.

AxisNL commented 3 years ago

Even if they are crawlers, the fact that they are in the CLOSE_WAIT state, means icecast_kh isn't cleaning up the sockets, or so I think I should interpret these stats.

mooseh commented 2 years ago

We can confirm this issue, we have our clients sometimes interrogating our servers for certain pages like the status.xsl (which we have disabled) and we hit over 500K open sockets today causing our process to lock up.