The current implementation assumes that the kube-state-rs service is either always up, or that losing label changes is not particularly important. Introduce fault tolerance to the service.
Some fault scenarios to consider and potential solutions:
The Service is down and nodes enter the cluster while it is down. We can iterate through all cluster nodes on startup and add labels as necessary.
The Service is down and nodes leave the cluster while it is down. We really want to avoid this scenario. Two strategies: 1) Run multiple instances of the Service while avoiding duplicated work. 2) Store labels when nodes are modified rather than when they're deleted. Need to think through performance considerations of this.
Multiple instances of the Service attempt to write different information at the same time. We can store a timestamp and keep only the latest.
Our storage mechanism crashes or a disk dies. ConfigMaps are stored in etcd. We can achieve our fault tolerance goals here if we have backups enabled on our Kubernetes cluster's etcd, and we have multiple etcd replicas.
The current implementation assumes that the kube-state-rs service is either always up, or that losing label changes is not particularly important. Introduce fault tolerance to the service.
Some fault scenarios to consider and potential solutions: