alba check: too many client connections

jeroenmaelbrancke commented 7 years ago

Ticket: https://github.com/openvstorage/alba/issues/461

Check with the healthcheck if alba throws too many client connections.

Nov 25 10:15:50 ftcmp02 alba[22625]: 2016-11-25 10:15:50 104537 +0100 - ftcmp02 - 22625/0 - alba/proxy - 442600 - info - server: new client connection from (172.22.186.32,40850)
Nov 25 10:15:50 ftcmp02 alba[22625]: 2016-11-25 10:15:50 105027 +0100 - ftcmp02 - 22625/0 - alba/proxy - 442601 - info - server: new client connection from (172.22.186.32,40852)
Nov 25 10:16:02 ftcmp02 alba[22625]: 2016-11-25 10:16:02 400797 +0100 - ftcmp02 - 22625/0 - alba/proxy - 442602 - warning - Denying connection from (172.22.186.32,43148), too many client connections 128
Nov 25 10:16:03 ftcmp02 alba[22625]: 2016-11-25 10:16:03 223398 +0100 - ftcmp02 - 22625/0 - alba/proxy - 442603 - warning - Denying connection from (172.22.186.32,43448), too many client connections 128
Nov 25 10:16:04 ftcmp02 alba[22625]: 2016-11-25 10:16:04 401951 +0100 - ftcmp02 - 22625/0 - alba/proxy - 442604 - warning - Denying connection from (172.22.186.32,43556), too many client connections 128

wimpers commented 7 years ago

@jeroenmaelbrancke Isn't this something which should better be detected by the log monitoring?

jeroenmaelbrancke commented 7 years ago

Ops is not working on the healthcheck. Reassigned ticket to Jeffrey. @JeffreyDevloo you can use your logparser script to check this output.

wimpers commented 7 years ago

@jeroenmaelbrancke I was talking about the Kibina log gathering. We aggregate the logs in Kibana so there must be a way to raise a ticket/alert in case we detect such an issue there.

jeroenmaelbrancke commented 7 years ago

@wimpers i'm still not convinced to use Kibana for this. What with an environment without Kibana? We want to know this on environments without Kibana also. The healthcheck can use the logparser from @JeffreyDevloo to get this information.

wimpers commented 7 years ago

I believe there are multiple reasons for doing it it on the Kibana side.

doing it on the Kibana side doesn't impact the storage performance. Every check in health check adds load.
The health check only runs on scheduled intervals so it might take an hour/day for it to be picked up. This is a serious issue with downtime so the sooner we can detect it, the sooner we can act upon it.
Doing it in the health check poses some questions: which logs will you can? All logs? Also the rotated ones? How will you make a distinction between a new occurrence and something that happened an hour ago?
The monitoring rule will require finetuning (# occurrences before escalating), so hardcoding it isn't a good idea.

'What with an environment without Kibana?' is a follow up question. We should put it in the Kibana monitoring and later ask the question if it is worthwhile to also add it to the health check. If we have the Kibana monitoring, I don't see a strict need to add it to the health check, except for a second set of eyes. Keep in mind that the clusters we monitor, will always have Kibana.

jeroenmaelbrancke commented 7 years ago

Closing ticket on the healthcheck repo. New ticket https://github.com/openvstorage/openvstorage-monitoring/issues/104

openvstorage / openvstorage-health-check

alba check: too many client connections #191