beyhan
commented
4 months ago @max-soe is there a bosh director version since which you observe this behaviour?
Closed max-soe closed 3 months ago
beyhan
commented
4 months ago @max-soe is there a bosh director version since which you observe this behaviour?
max-soe
commented
4 months ago No, as it is only a false-positive alert, nobody looked into that. But in the last month the number of alerts increased. My assumption is that we meanwhile use faster and faster vm types and with that this happens more often.
jpalermo
commented
4 months ago I think the health monitor logs all the VM NATS updates. So you should be able to reconstruct a history of what happened and how it ended up in this state. Are the updates not being received/processed by the health monitor? Is it a timing issue?
max-soe
commented
4 months ago Yes i have the full log, but it's to big for this thread... But i can summaries it.
Start of health-monitoring after the bosh director is recreated:
I, [2024-06-04T00:28:59.173218 #7] INFO : Connected to NATS at 'nats://bosh.cf-eu10-003.internal:4222'
I, [2024-06-04T00:28:59.173585 #7] INFO : Logging delivery agent is running...
I, [2024-06-04T00:28:59.173828 #7] INFO : Event logger is running...
I, [2024-06-04T00:28:59.174100 #7] INFO : Resurrector is running...
I, [2024-06-04T00:28:59.175149 #7] INFO : HTTP server is starting on port 25923...
I, [2024-06-04T00:28:59.201964 #7] INFO : BOSH HealthMonitor 0.0.0 is running...
I, [2024-06-04T00:28:59.202087 #7] INFO : connection.graphite-connected
I, [2024-06-04T00:28:59.761777 #7] INFO : Resurrection config remains the sameFound deployments and adding agents:
I, [2024-06-04T00:28:59.802495 #7] INFO : Found deployment 'app-autoscaler'
I, [2024-06-04T00:28:59.881586 #7] INFO : Adding agent e35a914e-233a-44c2-a01a-bb52bdd7faa3 (metricsforwarder/0d04ccc0-7a15-4e9d-9ec1-0f9c6eb7897d) to app-autoscaler...Start analysing and reporting some alerts:
I, [2024-06-04T00:30:29.179033 #7] INFO : Analyzing agents...
I, [2024-06-04T00:30:29.179254 #7] INFO : [ALERT] Alert @ 2024-06-04 00:30:29 UTC, severity 2: e35a914e-233a-44c2-a01a-bb52bdd7faa3 has timed out
I, [2024-06-04T00:30:29.251761 #7] INFO : (Event logger) notifying director about event: Alert @ 2024-06-04 00:30:29 UTC, severity 2: e35a914e-233a-44c2-a01a-bb52bdd7faa3 has timed outMeltdown alerts:
I, [2024-06-04T00:30:29.256503 #7] INFO : [ALERT] Alert @ 2024-06-04 00:30:29 UTC, severity 2: app-autoscaler has instances with timed out agents
I, [2024-06-04T00:30:29.256565 #7] INFO : (Event logger) notifying director about event: Alert @ 2024-06-04 00:30:29 UTC, severity 2: app-autoscaler has instances with timed out agents
I, [2024-06-04T00:30:29.308410 #7] INFO : [ALERT] Alert @ 2024-06-04 00:30:29 UTC, severity 1: Skipping resurrection for instances: metricsforwarder/0d04ccc0-7a15-4e9d-9ec1-0f9c6eb7897d, metricsforwarder/fb8e7a9b-d7d0-47cd-bfd4-f1e59b880f35, metricsserver/fe5527ab-57fa-4c24-a457-acebe8e65edb, metricsgateway/bd0b06a0-180e-4a1b-8d3e-0588c5eeefe7, operator/d199c4db-5d05-456b-91c0-ed634d40bd07, operator/f40b3de1-6e01-4e8a-8a42-ab99336536f1, eventgenerator/7bb09c5b-a81f-47ac-bc3c-57ce20e25c54, scalingengine/dda1e1f7-3290-43eb-b605-1e040614966f; deployment: 'app-autoscaler'; 8 of 23 agents are unhealthy (34.8%)If the issue not occur, the reports are similar except the meltdown alerts. We getting timeout alerts on all landscapes, but they not hit everytime the threshold. As the other deployments like here autoscaler are not updated, i think this is a timing issue. But how can we fix that to avoid this false positives?
jpalermo
commented
3 months ago Working backward. To get the meltdown alert, the agents need to be timed out.
Agent objects, when they are created, are seeded with the current time as their updated_at value. Which is updated to the current time any time the agent heartbeats.
So why didn't the health monitor get a heartbeat from the agent before it was timed out? You can check the health monitor logs to see when it got heartbeats from the agent. Agents send heartbeats every 30 seconds, so you should be able to work forward or backward from other heartbeats in the logs to see what the health monitor was doing when it should have received it.
Might also check the agent logs in this situation to see if there was any sort of error when sending the heartbeat.
max-soe
commented
3 months ago We cannot reproduce this issue, after we migrated from the bosh version 280.0.20 to 280.0.25. Maybe this was fixed by the metric server improvements. I will close this ticket for now.
Hey everyone,
Describe the bug From time to time we see meltdown alerts in our dashboards. They only occur during the updates of bosh and and then are sent out for nearly all deployments. But only for some of our our bosh directors and not in every update. After some research we think that this depends on the restart time. After a restart our dashboard shows that bosh report a lot of unresponsive agents. So our assumption here is that this is the default state for all instances if a new bosh director vm is deployed. Is this correct? If the health-monitoring job now is started fast enough, it getting the unresponsive agent state for more and more instances and then sent a meltdown alert. What i currently unsure is, is bosh sending this deployments into meltdown? Or is it only sending the alert?
Expected behavior Is it correct that instances have the unresponsive agent state after startup of bosh? And if we cannot change that, is it possible to start the health monitoring later to avoid such alerts?
Logs