Health checks and dashboard for connectivity between zones

jakubdyszkiewicz commented 3 years ago

Summary

Right now we have pretty good observability of a connection from Remote to Global. You can see easily see in Global that Remote is online or not.

The problem is that once everything zone is connected to global, we don't provide any tools to check connectivity from one zone to another.

Observability

I'd like to have a dashboard in the GUI to see connectivity between zones. Something like this

where there is a separate Zone on each end. If the connection is up, the line is green, if not the line is red.

(It also can be just a simple table with every zone in column and every zone in row)

Traffic reliability

If Global <-> Remote communication works, but Zone <-> Zone does not we still include Ingress endpoint into EDS. Of course, Envoy can exclude this endpoint with Health Checks / Circuit Breaker + Retries, but we should do this beforehand.

If Remote A knows that it cannot connect to Ingress of Remote B, it should not include this in the EDS.

Overview of potential implementation

Build a component that runs on Remote CP (leader) that healthchecks (just open TCP connection) Ingress from other zones and save this information in Config called zone-healthcheck
When building XDS config, check zone-healthcheck Config and exclude ingress if we see that HC failed
Propagate zone-healthcheck from every Remote CP to Global CP
Build an API based on all zone-healthcheck from every zone
Build a dashboard based on this API

Considerations

Zone can have 2 ingresses, one can fail, the other don't. Probably it would be good to have "partially degrated" status just to follow the same pattern as with DP health checks.

github-actions[bot] commented 2 years ago

This issue was inactive for 30 days it will be reviewed in the next triage meeting and might be closed. If you think this issue is still relevant please comment on it promptly or attend the next triage meeting.

lahabana commented 2 years ago

This ticket will need to be split into a milestone and this only tracks the design of it. The proposed design/idea from @jakubdyszkiewicz probably needs to be adapted with the creation of ZoneEgress.

github-actions[bot] commented 2 years ago

This issue was inactive for 30 days it will be reviewed in the next triage meeting and might be closed. If you think this issue is still relevant please comment on it promptly or attend the next triage meeting.

github-actions[bot] commented 2 years ago

This issue was inactive for 30 days it will be reviewed in the next triage meeting and might be closed. If you think this issue is still relevant please comment on it promptly or attend the next triage meeting.

github-actions[bot] commented 2 years ago

This issue was inactive for 30 days it will be reviewed in the next triage meeting and might be closed. If you think this issue is still relevant please comment on it promptly or attend the next triage meeting.

github-actions[bot] commented 2 years ago

This issue was inactive for 30 days it will be reviewed in the next triage meeting and might be closed. If you think this issue is still relevant please comment on it promptly or attend the next triage meeting.

github-actions[bot] commented 1 year ago