DNS/DHCP Portal Issue -- Understand how DHCP Admins invokes Kea agent running on DHCP Server

[satishgummadellimoj]

Suggestion from Mitch to revert any changes prior to when we were seeing the errors to investigate further.

Database error DHCP ADMIN LOGS

(UTC+1:00) [22a912f-0e4f-40fe-877e-b1196541d8a7] Gateways::KeaControlAgent::InternalError (unable to execute for <SELECT address, hwaddr, client_id, valid_lifetime, expire, subnet_id, fqdn_fwd, fqdn_rev, hostname, state, user_context FROM lease4 WHERE subnet_id = ?>, reason: Commands out of sync; you can't run this command now (error code 2014)):

https://app.zenhub.com/workspaces/nvvs-devops-622a0b371800e400133bb924/issues/gh/ministryofjustice/nvvs-devops/414

Useful Contact: Mitch Dawson

[tomwells98]

This ticket is for developing a better understanding on the topic as supposed to any specific work.

[tomwells98]

Ticket to be pulled into a sprint when James is back from AL

[tomwells98]

From Mitch: I have been speaking with the rest of the guys about this issue. Our suggestion would be to revert any changes from the known working period up until this materialised. If you can supply some detailed examples of the investigation, troubleshooting and log analysis that has been performed by the team then we might be able to suggest some other potential causes.

[satishgummadellimoj]

We have seen same error yesterdayYou are receiving this email because your Amazon CloudWatch Alarm "Commands_Out_Of_Sync" in the EU (London) region has entered the ALARM state, because "Threshold Crossed: 1 out of the last 1 datapoints [4.0 (09/10/23 10:31:00)] was greater than the threshold (2.0) (minimum 1 datapoint for OK -> ALARM transition)." at "Monday 09 October, 2023 10:32:42 UTC".

View this alarm in the AWS Management Console: https://eu-west-2.console.aws.amazon.com/cloudwatch/deeplink.js?region=eu-west-2#alarmsV2:alarm/Commands_Out_Of_Sync

Alarm Details:

• Name: Commands_Out_Of_Sync
• Description: Cloud watch alarm triggered for Commands_Out_Of_Sync~view~'timeSeries~stacked~false~region~'eu-west-2~stat~'Sum~period~300~start~'-PT3H~end~'P0D)&query=~'7bDHCP_Admin7d20MetricName3d22Commands_Out_Of_Sync22) this is likely a database query error, previously resolved by redeploying DHCP server cluster.
• State Change: OK -> ALARM
• Reason for State Change: Threshold Crossed: 1 out of the last 1 datapoints [4.0 (09/10/23 10:31:00)] was greater than the threshold (2.0) (minimum 1 datapoint for OK -> ALARM transition).
• Timestamp: Monday 09 October, 2023 10:32:42 UTC
• AWS Account: 037161842252
• Alarm Arn: arn:aws:cloudwatch:eu-west-2:037161842252:alarm:Commands_Out_Of_Sync

Threshold:

• The alarm is in the ALARM state when the metric is GreaterThanThreshold 2.0 for at least 1 of the last 1 period(s) of 60 seconds.

Monitored Metric:

• MetricNamespace: DHCP_Admin
• MetricName: Commands_Out_Of_Sync
• Dimensions:
• Period: 60 seconds
• Statistic: Sum
• Unit: not specified
• TreatMissingData: missing

State Change Actions:

• OK:
• ALARM: [arn:aws:sns:eu-west-2:037161842252:DHCP_Admin_Alarm]
• INSUFFICIENT_DATA:

-- If you wish to stop receiving notifications from this topic, please click or visit the link below to unsubscribe: https://sns.eu-west-2.amazonaws.com/unsubscribe.html?SubscriptionArn=arn:aws:sns:eu-west-2:037161842252:DHCP_Admin_Alarm:5fd4e0f9-efae-4f83-9128-3f0cee059f03&Endpoint=nvvs-devops-dhcp-admi-aaaakrpoeh3cbhb7vaa4vcjfke@moj.org.slack.com

Please do not reply directly to this email. If you have any questions or comments regarding this email, please contact us at https://aws.amazon.com/support

[satishgummadellimoj]

https://dsdmoj.atlassian.net/wiki/spaces/MOJO/pages/4550066177/DNS+DHCP+Portal+Working

[satishgummadellimoj]

new ticket to resolve mysql errors https://app.zenhub.com/workspaces/nvvs-devops-622a0b371800e400133bb924/issues/gh/ministryofjustice/nvvs-devops/485

[satishgummadellimoj]

https://dsdmoj.atlassian.net/wiki/spaces/MOJO/pages/4550066177/DNS+DHCP+Portal+Working

[juddin927]

@satishgummadellimoj I think your update gone in to wrong page!