search

containerd / nri

Node Resource Interface
Apache License 2.0
238 stars 62 forks source link

How can I ensure that containerd cannot create a container if the NRI plugin is unavailable? #85

Open t33m opened 4 months ago

t33m commented 4 months ago

Is there a way to configure containerd so that starting containers is impossible if the NRI plugin is not working or hasn't registered yet?

kad commented 4 months ago

It is impossible now, and not sure that this is going to be good idea. Some of NRI plugins are going to be deployed as containers, and infrastructure components are also in containers (e.g. kubelet's static pod manifests), so not starting until NRI plugin registers will lead to non-functional node. Another thing is the crashes of the plugins and re-connects: we shouldn't have scenarios where crash of the plugin will render node even temporary to non-usable state.

For all of those the sync-on-connect calls were implemented: the plugin during start can inspect and adjust running containers which were started before NRI plugin registers. As well, if there is something that can't be modified via Update cals, there is always possibility for NRI plugin to trigger stop of the existing container, and eventual re-creation of it by orchestration layer above.

mikebrow commented 4 months ago

Some discussion here .. wip https://github.com/containerd/nri/pull/43#discussion_r1213237527

mikebrow commented 4 months ago

@kad good insight

Should probably develop some use cases .. an integration bucket and a e2e bucket that exercises the supported use cases.

klihub commented 4 months ago

It is impossible now, and not sure that this is going to be good idea. Some of NRI plugins are going to be deployed as containers, and infrastructure components are also in containers (e.g. kubelet's static pod manifests), so not starting until NRI plugin registers will lead to non-functional node. Another thing is the crashes of the plugins and re-connects: we shouldn't have scenarios where crash of the plugin will render node even temporary to non-usable state.

For all of those the sync-on-connect calls were implemented: the plugin during start can inspect and adjust running containers which were started before NRI plugin registers. As well, if there is something that can't be modified via Update cals, there is always possibility for NRI plugin to trigger stop of the existing container, and eventual re-creation of it by orchestration layer above.

I agree with @kad that it's probably not a good idea to bake such logic into the runtime itself.

If I had to do something to this effect, my first idea would probably be to roll some extra tooling for it. Run the critical plugin(s) as DaemonSets, monitor whether they are ready/live (or have them refresh a CRD or a label on their node periodically and monitor that), then taint and if necessary drain/cordon/uncordon the nodes as necessary by extra tooling. If useful, maybe also label workloads that need the critical plugins as such, and have a mutating webhook to inject tolerations for unlabeled workloads to tolerate nodes without the critical plugins being up and running.

zhaodiaoer commented 2 months ago

In the scenarios where I work, i also encountered similar problems... I'm not sure if it's a good idea for us to provide the ability to categorize all plugins by role and make corresponding distinctions during the plugin registration/deregistration process. such as give a flag used to identify what the critical level for plugin to notify NRI when Initiate registration, and when deal with different level plugin's deregistration use different policy, like only when plugin explicitly initiates one rpc request can the deregistration be completed. (just an example and hasn't been carefully considered)