Review split brain mechanism for Route53 and future external-dns based EdgeDNS providers

ytsarev commented 4 years ago

In case of Infoblox we are heavily relying on TXT records which hold heartbeat timestamps of each k8gb controller instance. k8gb then mutual cross-check each other TXT records and if record is stale - remove the 'dead' k8gb from zone delegation configuration. This approach works perfect in on-prem Infoblox scenario but in case of external-dns / Route53 we are hitting two limitations:

AWS Route 53 API throttling of 5 request per second, we might easily hit it with multiple Gslb objects and frequent reconciliation loop
External-dns treating TXT records as own way to hold metadata and ownership and it conceptually conflicts with custom TXT records creation in a current state of codebase

We might redesign split brain mechanism and/or add custom TXT support to external-dns upstream

ytsarev commented 3 years ago

We also got a report from Infoblox admin that current hearbeat implementation is eating too much CPU, we need to optimize it apparently and/or make it optional

somaritane commented 2 years ago

@ytsarev @kuritka is this still relevant?

ytsarev commented 2 years ago

Depends if we still want to invest into any for of split brain handling. If not, we should deterministically prove that it is not required.

ytsarev commented 1 year ago

As discussed in the community meeting today we should

Instead of the complex logic, be verbose in case of horizontal network partitioning and split-brain (thanks @infbase )
- Clarify logging message
- Potentially propagate to Gslb status
Test failure scenarios to deeply understand what will happen ( thanks @jkremser )

k8gb-io / k8gb

Review split brain mechanism for Route53 and future external-dns based EdgeDNS providers #175