Inject the Agent as a native sidecar

[busser]

Hi 👋

Is your feature request related to a problem? Please describe.

Running the Vault Agent as a sidecar container is extremely useful. However, like with most sidecars, we quickly run into issues that require custom solutions. For example:

• the Agent needs to run as both an init-container and a sidecar container to ensure secrets are available to the main container when it starts
• the init-container must persist leases for the sidecar container, to avoid double-fetching secrets
• the sidecar container must not be injected into short-lived pods like those created by Jobs

Kubernetes native sidecars aim to solve these issues for all sidecars, including the Vault Agent.

Describe the solution you'd like

I would like the Agent Injector to support injecting the Agent as a native sidecar.

Concretely, this means that the Injector should:

• add only the init-container
• set the init-container's restartPolicy to Always
• configure the Agent to run forever and not fetch secrets only once
• configure a startupProbe so that Kubernetes knows when secrets have been fetched and templates rendered at least once

Implementing the startup probe in the Agent may require further discussion. This may require adding a new endpoint to the Agent.

We could start by adding a boolean annotation that enables this behaviour on annotated pods. The annotation could be named agent-native-sidecar for instance. I think that eventually this behaviour could become the default on clusters where the native sidecars feature is enabled.

Additional context

I am more than happy to help implement this feature.

[tvoran]

Thanks for raising this too! The lack of a long-lived sidecar option is precisely why we ended up adding persistent caching to Vault agent, but it would be great to remove the need for that complexity. I've been meaning to try this out for a while, but contributions welcome!

[komapa]

I would like to see this functionality implemented as well now that native sidecars are beta in 1.29 and we have 1.30 already released. I can help test any implementation with real use cases.