search

clusterpedia-io / clusterpedia

The Encyclopedia of Kubernetes clusters
https://clusterpedia.io
Apache License 2.0
787 stars 122 forks source link

Implementations of Watch that can be used for most storage layers #453

Open Iceber opened 1 year ago

Iceber commented 1 year ago

What would you like to be added?

We currently have Watch capability in memory storage, but it requires the binding-apiserver command to run both Clusterpedia APIServer and Clusterpedia Clustersynchro Manager, which prevents us from scaling the service horizontally

We wanted to find a generic Watch mechanism that could be reused on most storage tiers and that would not affect the horizontal scaling of the Clusterpedia APIServer.

For the Watch feature, there are roughly three ways to implement it:

  1. Customized to a specific storage layer, using a storage layer with its own event notifications, such as memory, etcd-based Watch, or redis publish-and-subscribe for Watch But these specific storage layers may not be good for resource retrieval, and we will lose some performance or functionality for complex resource retrieval

  2. A middleware-based subscription publishing for messages, decoupling Query/Watch(APIServer) and collecting resources through a message middleware. But the introduction of middleware will increase the complexity of the architecture and may be troublesome for Watch handling when the middleware is down

  3. Forwarding Watch requests to the collection of resources -- ClusterSynchroManager But the ClusterSynchroManager may need to increase the cache of some events, which inadvertently brings uncontrollable memory pressure In addition, the future Agent mode will add a resource collector component that can be scaled horizontally, which will not be able to forward Watch requests correctly

Why is this needed?

https://github.com/clusterpedia-io/clusterpedia/issues/452

clusterpedia-bot commented 1 year ago

Hi @Iceber, Thanks for opening an issue! We will look into it as soon as possible.

Details Instructions for interacting with me using comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the [gh-ci-bot](https://github.com/clusterpedia-io/gh-ci-bot) repository.
xyz2277 commented 1 year ago
new-watcher

hi, iceber, this is my solution for the new Watch feature.

xyz2277 commented 1 year ago
  1. A message queue is used for transporting events from ClusterSynchroManager to ApiServer, before transporting, events must be persisted to the storage layer, and rv of events must be converted to multi-clusters-rv.
  2. EventCache is a buffer for events, when watch request is arrived, initialization events will be found in it, and the following events will be coming from the Message queue.
  3. Multi-clusters-rv will be returned in a list request.
xyz2277 commented 1 year ago

What do you think of it. @Iceber

Iceber commented 1 year ago

With the addition of the new component, I think one issue we can't ignore is the behavior of the Synchor and Apiserver after the message queue hangs