Use native sidecar containers

nais / naiserator

Naiserator creates a full set of Kubernetes application infrastructure based on a single application spec.

MIT License

43 stars 12 forks source link

Use native sidecar containers #568

Open tronghn opened 1 week ago

tronghn commented 1 week ago

Kubernetes natively supports sidecar containers, which has been available in beta since Kubernetes 1.29. Slated for GA in 1.33. Upstream issue tracking: https://github.com/kubernetes/enhancements/issues/753

Depending on an Application's configuration, Naiserator may inject long-lived containers that run alongside the main application container.

Examples include:

Cloud SQL proxy - auth proxy for Cloud SQL
Elector - provides an API for leader election
Fluentbit - ships secure logs (already uses native sidecar)
Wonderwall - reverse proxy for user authentication

We should update the remaining containers to also use the new spec.

Seems like there are still some issues to be resolved. Keep these in consideration if we decide to update the remaining containers before GA.

kimtore commented 1 week ago

Do we want to wait for GA in 1.33 or try to integrate changes now?

@tronghn, what do you think after reading that issue?

tronghn commented 1 week ago

https://github.com/kubernetes/kubernetes/issues/127958 would affect Wonderwall at least, and it was only recently fixed. We could work around this by using numeric ports instead, though I'm not sure how this plays out with our logging and prometheus stack.

On the other hand, we already have one sidecar using this feature and it seems to be stable enough for that use case.

We could try moving these one by one and iron out any problems on the way, though I think we'd benefit from just waiting until 1.32 or GA. The current set up with "normal" containers has been working fine without any major issues.

sklirg commented 1 week ago

kubernetes/kubernetes#127958 would affect Wonderwall at least, and it was only recently fixed. We could work around this by using numeric ports instead, though I'm not sure how this plays out with our logging and prometheus stack.

It's also possible to work around it by injecting the (named) port in "one of the main containers" instead of in the sidecar/initContainer, as workaround number 2 demonstrates. Then, in the future, when this is fixed (it's targeting Kuberetes v1.32), the injection can move to the initContainer instead. More hacky, but retains the named port functionality.