search

karlheyes / icecast-kh

KH branch of icecast
GNU General Public License v2.0
298 stars 106 forks source link

Disconnecting Sources #206

Closed mrigdon-emf closed 6 years ago

mrigdon-emf commented 6 years ago

I am seeing some strange disconnects, I am not 100% certain what is causing this and I wanted to reach out and see if you have heard of or experienced these types of issues before. I have a cluster of encoding servers 4 primary & 4 backup. These servers are configured to combined generate approximately 200 mount points with various formats and bitrates (AAC-HE 64, AAC-HE 24, MP3 128 & MP3 64). The Backup encoding servers are configured with the same connection details to the primary and secondary icecast servers and are configured with encoding enabled but since the primary servers are already connected to icecast on the same mountpoints the backup servers cannot connect which is what we want. However, gradually over time the backup encoding servers take over encoding to the different icecast servers. We have a monitoring system in place and our CDN relays and we do see that the relays switch sources when these disconnects happen.

My question for you is have you heard of this? Do you have any suggestions on what I might look for in the debug logs to help with this. I am also exploring this as a possible issue with the encoding servers. I will say that I had run testing with one of the encoding servers into icecast that ran for over a month and never saw these issues which is leading me to an issue with our specific os layer, however I am hoping perhaps you have insight to assist with the root cause of this issue.

A notes we are running 2.4.0-kh8 on Windows Server 2016 x64. I also theorize that it is an issue with the windows implementation. I may do an experiment to move over to the linux branch as the windows implementation was inherited.

Thanks for any help you may be able to provide.

mrigdon-emf commented 6 years ago

I seem to have been able to stabilize, my numbers for active client & source connections didn't appear to be over my set <limits> of 2,000 but after changing those <limits> to 10,000 that seemed to stabilize things. I am going to be keeping an eye on things for a while. I will close for now and can always open a new issue if I find a cause. Thanks.