DHCP server error count high

[juddin927]

User Story

    Labels:

• alertname = DHCP KEA Server Alert
• account_id = 037161842252
• instance = cloudwatch-exporter:5000
• job = cloudwatch
• name = Kea DHCP
• namespace = monitoring
• prometheus = monitoring/kube-prometheus-stack-prometheus
• region = eu-west-2
• service = DNS DHCP
• severity = critical Annotations:
• description = The server alert ERROR or FATAL is greater than 10. The current value is 111
• grafana_dashboard_url = https://monitoring-alerting.staff.service.justice.gov.uk/d/cEwjsH1Gk/kea-dhcp-metrics
• summary = DHCP KEA Server Alert ERROR or FATAL > 10 Source: http://kube-prometheus-stack-prometheus.monitoring:9090/graph?g0.expr=aws_kea_dhcp_error_sum%7Baccount_id%3D%22037161842252%22%7D+%3E+100+or+aws_kea_dhcp_fatal_sum%7Baccount_id%3D%22037161842252%22%7D+%3E+100&g0.tab=1

Value / Purpose

these alert has been triggered multiple times today

Useful Contacts

No response

Additional Information

No response

Definition of Done

• [ ] Investigate the logs
• [ ] document the findings
• [ ] Generate recommendations if any
• [ ] Another team member has reviewed
• [ ] Tests are green

[juddin927]

alarm was generated due to the sum of server error count was greater than 100 for 7 minute timeframe. Whilst investigating the logs it was generated due to a standby server failing to sync as well as host that did not have a scope setup on our DHCP server. since redeployment we got those error cleared however we still have couple of noisy host kept on sending us DHCPDISCOVER packet. 10.150.211.1 and 10.151.88.1. Colleague from lan and wifi reached out to BT to remove those devices

[juddin927]

we could also potentially increase the generic error count threshold.

[juddin927]

to address the critical error alerts issue we have temporarily changed the DHCP server error count sum from 100 to 150.