search

karlheyes / icecast-kh

KH branch of icecast
GNU General Public License v2.0
300 stars 107 forks source link

Client limit reached on low listener count #389

Closed glitch59 closed 1 year ago

glitch59 commented 1 year ago

Hi Karl,

Been testing in production and after some period of time, the server is not accepting connections anymore. It responds with: 403 - Too many clients connected

The max clients is set to 1200 in the xml file.

After temporally updating the max clients to a higher number and reloading the config, without restarting, the admin stats are back online and there are 9 listeners registered. So it looks like the client counter is not registering the correct amount of listeners.

The following data is logged:

[2023-01-27  21:55:01] WARN connection/_handle_get_request server client limit reached (1200/1203) for xx.xxx.xx.x
[2023-01-27  21:55:01] WARN connection/_handle_get_request server client limit reached (1200/1203) for yy.yy.yy.yyyy
[2023-01-27  21:55:02] WARN connection/_handle_get_request server client limit reached (1200/1203) for xxx.x.x.x
[2023-01-27  21:55:02] WARN connection/_handle_get_request server client limit reached (1200/1203) for 123.45.67.8
[2023-01-27  21:55:02] WARN connection/_handle_get_request server client limit reached (1200/1203) for 11.22.333.4

Could this be a bug, or any other idea's of what causes this?

Best regards, Sven

glitch59 commented 1 year ago

I see this in the logs every few seconds:

123.456.67.88 - - [27/Jan/2023:22:23:03 +0100] "GET /status-json.xsl HTTP/1.1" 200 13459 "-" "Mozilla" 0
123.456.67.88 - - [27/Jan/2023:22:23:06 +0100] "GET /status-json.xsl HTTP/1.1" 200 13459 "-" "Mozilla" 0
123.456.67.88 - - [27/Jan/2023:22:23:09 +0100] "GET /status-json.xsl HTTP/1.1" 200 13459 "-" "Mozilla" 1
123.456.67.88 - - [27/Jan/2023:22:23:12 +0100] "GET /status-json.xsl HTTP/1.1" 200 13459 "-" "Mozilla" 0
123.456.67.88 - - [27/Jan/2023:22:23:15 +0100] "GET /status-json.xsl HTTP/1.1" 200 13459 "-" "Mozilla" 1
123.456.67.88 - - [27/Jan/2023:22:23:18 +0100] "GET /status-json.xsl HTTP/1.1" 200 13459 "-" "Mozilla" 0
123.456.67.88 - - [27/Jan/2023:22:23:21 +0100] "GET /status-json.xsl HTTP/1.1" 200 13459 "-" "Mozilla" 1
123.456.67.88 - - [27/Jan/2023:22:23:24 +0100] "GET /status-json.xsl HTTP/1.1" 200 13459 "-" "Mozilla" 0
123.456.67.88 - - [27/Jan/2023:22:23:27 +0100] "GET /status-json.xsl HTTP/1.1" 200 13403 "-" "Mozilla" 0
123.456.67.88 - - [27/Jan/2023:22:23:30 +0100] "GET /status-json.xsl HTTP/1.1" 200 13403 "-" "Mozilla" 0
123.456.67.88 - - [27/Jan/2023:22:23:33 +0100] "GET /status-json.xsl HTTP/1.1" 200 13403 "-" "Mozilla" 1
123.456.67.88 - - [27/Jan/2023:22:23:36 +0100] "GET /status-json.xsl HTTP/1.1" 200 13402 "-" "Mozilla" 0
123.456.67.88 - - [27/Jan/2023:22:23:39 +0100] "GET /status-json.xsl HTTP/1.1" 200 13403 "-" "Mozilla" 1
123.456.67.88 - - [27/Jan/2023:22:23:42 +0100] "GET /status-json.xsl HTTP/1.1" 200 13403 "-" "Mozilla" 0
123.456.67.88 - - [27/Jan/2023:22:23:45 +0100] "GET /status-json.xsl HTTP/1.1" 200 13403 "-" "Mozilla" 1
123.456.67.88 - - [27/Jan/2023:22:23:48 +0100] "GET /status-json.xsl HTTP/1.1" 200 13375 "-" "Mozilla" 0
123.456.67.88 - - [27/Jan/2023:22:23:51 +0100] "GET /status-json.xsl HTTP/1.1" 200 13375 "-" "Mozilla" 1
123.456.67.88 - - [27/Jan/2023:22:23:54 +0100] "GET /status-json.xsl HTTP/1.1" 200 13375 "-" "Mozilla"

Someone is requesting this file very often.

Is it possible this interacts with the client counter? And if so, how can this be prevented from happening?

glitch59 commented 1 year ago

This is happening on the version committed at 2023-01-18.

Client counter stays around 1215 now, even after almost 1 hour, it does not go lower. Something is keeping those connections alive, or the counting is wrong.

Edit: After some more testing it seems clear now, clients only increasing and never decreasing. When I stop playing a stream and play another one, the counter increases, but it never decreases. So it is only a matter of time until the next limit is reached.

karlheyes commented 1 year ago

I have cut a 17.2 release (https://github.com/karlheyes/icecast-kh/tags) if you can verify this. I did try some stream connections and drops and the counter looked ok but you may be tripping something I'm not. Check the same sort of setup with kh17.2 to get that increase going and if so, send me the level 4 log.

karl.

glitch59 commented 1 year ago

Thanks for the fast response!

I have installed the new version in production this morning, and the client counter stays normal now. Will check a few times extra today and tomorrow and then report back here, but the issue seems resolved.

Best regards, Sven

glitch59 commented 1 year ago

Few day later, and the counter is still correct. This issue is resolved and can be closed. Thanks again for the support!