search

karlheyes / icecast-kh

KH branch of icecast
GNU General Public License v2.0
298 stars 106 forks source link

Icecast fails when reaching file descriptor limit #232

Closed evgenmedia closed 5 years ago

evgenmedia commented 5 years ago

Hello!

I have a problem about 2 weeks My icecast kh-7 server fails, when reaching file descriptor limit In average we have 2000 listeners

This information in log:

[2018-09-17  12:29:36] WARN client/worker_wait worker 0x8a7940 has been stuck for 1126 ms
[2018-09-17  12:30:10] WARN source/source_read Nothing received on /pionerfm for 3 seconds
[2018-09-17  12:30:10] WARN source/source_read Nothing received on /pionerfm for 3 seconds
[2018-09-17  12:30:10] WARN source/source_read Nothing received on /pionerfm for 3 seconds
[2018-09-17  12:30:18] WARN source/source_read Disconnecting /pionerfm due to socket timeout
[2018-09-17  12:30:27] WARN client/worker_wait worker 0x8a7940 has been stuck for 5856 ms
[2018-09-17  12:31:04] WARN client/worker_wait worker 0x8a7940 has been stuck for 1187 ms
[2018-09-17  12:31:21] WARN client/worker_wait worker 0x8a7940 has been stuck for 1121 ms
[2018-09-17  12:31:36] WARN client/worker_wait worker 0x8a7940 has been stuck for 3746 ms
[2018-09-17  12:31:51] WARN client/worker_wait worker 0x8a7940 has been stuck for 2154 ms
[2018-09-17  12:32:11] WARN client/worker_wait worker 0x8a7940 has been stuck for 2279 ms
[2018-09-17  12:40:48] WARN slave/slave_startup process has 65536 max file descriptor limit

How can i fix it?

karlheyes commented 5 years ago

you should update first, kh7 has been out for some time now. Do you have a <workers> setting in <limits>, if not, try setting it to the number of processors you have or want to allocate. If you still have an FD leak then let me know.

karl.

evgenmedia commented 5 years ago

Thank you! Now my version is kh10, waiting for downtime ...

While waiting: how i can setup the number of processors? No manual found ;(

karlheyes commented 5 years ago

sorry about that, my tags were filtered. Yes I need to add those tags into the docs.

karl.

evgenmedia commented 5 years ago

News from error.log:

[2018-10-03  10:59:30] WARN client/worker_check_time_ms worker 0x8aae30 has been stuck for 1861 ms
[2018-10-03  10:59:30] WARN client/worker_check_time_ms worker 0x8abca0 has been stuck for 1878 ms
[2018-10-03  10:59:30] WARN client/worker_check_time_ms worker 0x8ac4e0 has been stuck for 1833 ms
[2018-10-03  10:59:30] WARN client/worker_check_time_ms worker 0x8c18c0 has been stuck for 1808 ms
[2018-10-03  10:59:30] WARN client/worker_check_time_ms worker 0x8ab040 has been stuck for 1869 ms
[2018-10-03  10:59:30] WARN client/worker_check_time_ms worker 0x8c2850 has been stuck for 1854 ms
[2018-10-03  10:59:30] WARN client/worker_check_time_ms worker 0x8aac20 has been stuck for 1852 ms
[2018-10-03  10:59:30] WARN client/worker_check_time_ms worker 0x8ad560 has been stuck for 1809 ms
[2018-10-03  10:59:30] WARN client/worker_check_time_ms worker 0x8c1f80 has been stuck for 1900 ms
[2018-10-03  10:59:30] WARN client/worker_check_time_ms worker 0x8ad350 has been stuck for 1881 ms
[2018-10-03  10:59:30] WARN client/worker_check_time_ms worker 0x8ac2d0 has been stuck for 1907 ms
[2018-10-03  10:59:30] WARN client/worker_check_time_ms worker 0x8ae3d0 has been stuck for 2026 ms
[2018-10-03  10:59:30] WARN client/worker_check_time_ms worker 0x8ae1c0 has been stuck for 1934 ms
018-10-03  10:59:56] WARN client/worker_check_time_ms worker 0x8acd20 has been stuck for 2193 ms
[2018-10-03  10:59:56] WARN client/worker_check_time_ms worker 0x8ab880 has been stuck for 2253 ms
[2018-10-03  10:59:56] WARN client/worker_check_time_ms worker 0x8ab460 has been stuck for 2224 ms
[2018-10-03  10:59:56] WARN format-mp3/validate_mpeg source /powerhit128, len 739, unprocessed 788
[2018-10-03  10:59:56] WARN client/worker_check_time_ms worker 0x8c0d80 has been stuck for 2260 ms
[2018-10-03  10:59:57] WARN client/worker_check_time_ms worker 0x8ae3d0 has been stuck for 2930 ms
[2018-10-03  10:59:57] WARN format-mp3/validate_mpeg source /powerhit128, len 717, unprocessed 730
[2018-10-03  10:59:57] WARN format-mp3/validate_mpeg source /powerhit128, len 739, unprocessed 739
[2018-10-03  10:59:57] WARN client/worker_check_time_ms worker 0x8c0d80 has been stuck for 1071 ms
[2018-10-03  10:59:59] WARN client/worker_check_time_ms worker 0x8adfb0 has been stuck for 1576 ms
[2018-10-03  10:59:59] WARN client/worker_check_time_ms worker 0x8ab880 has been stuck for 1635 ms
[2018-10-03  10:59:59] WARN client/worker_check_time_ms worker 0x8ad140 has been stuck for 1671 ms
[2018-10-03  10:59:59] WARN client/worker_check_time_ms worker 0x8ac2d0 has been stuck for 1681 ms
[2018-10-03  10:59:59] WARN client/worker_check_time_ms worker 0x8abeb0 has been stuck for 1665 ms

[2018-10-03  11:00:21] WARN format-mp3/validate_mpeg source /powerhit128, len 738, unprocessed 716
[2018-10-03  11:00:21] WARN format-mp3/validate_mpeg source /powerhit128, len 716, unprocessed 740
[2018-10-03  11:00:21] WARN format-mp3/validate_mpeg source /powerhit128, len 740, unprocessed 732

When workers stucks, the server was very slow... ssh connect a long time

I restart the server

Server: https://radio.dline-media.com

evgenmedia commented 5 years ago

The situation repeated, some logs from monitoring

Look at network graph 2018-10-03 13 16 29

evgenmedia commented 5 years ago

Server continues to fall ... (

Downtime between 19:26:54 and 22:01:50, here is my logs & config

error.log daemon.log access.log config.xml.txt

Virtual server params: 2 CPU 4 GB RAM SSD drive, free disk space is about 80%

evgenmedia commented 5 years ago

Test server, 3500 listeners is ok for him, not fails But fails from DF leak ...

evgenmedia commented 5 years ago

Problem is solved

Edit in config timeout for source/clients & add some RAM & 2 additional CPU, no fails still this fixes