search

google / k8s-digester

Add digests to container and init container images in Kubernetes pod and pod template specs. Use either as a mutating admission webhook, or as a client-side KRM function with kpt or kustomize.
Apache License 2.0
122 stars 20 forks source link

Support for path string substitutions #35

Open iamasmith opened 2 weeks ago

iamasmith commented 2 weeks ago

I'll start with a scenario.

Suppose we have multiple regional registries in a GCP although this works for a number of other scanarios.

Say we have a whole range of app images in these repos but lets say we have 'appa' built and common to all registries and we want clusters to pull from their own regional locations.

Currently this might need an image path like.

europe-docker.pkg.dev/projectname/europe-registry-name/appa:tag
us-docker.pkg.dev/projectname/us-registry-name/appa:tag
europe-west2.pkg.dev/projectname/london-registry-name/appa:tag

This means that development teams need to alter their deployments for regional clusters to have specific image sources.

The proposal would be that an arbitrary strings could be established for replacement, a source and a destination so that we could contrive a configuration as follows using a standard image name.

Assuming our source for string replacement is then 'logical-registry' we could state the image to be...

logical-registry/appa:tag

and this would be the same for all regions but we would specify targets in regional clusters matching the paths for their most adjacent registries i.e.

europe-docker.pkg.dev/projectname/europe-registry-name - for European
us-docker.pkg.dev/projectname/us-registry-name - for US
europe-west2.pkg.dev/projectname/london-registry-name - for London

Then at mutate time these strings could be substituted prior to performing the next part of the mutation to determine the image sha.

It's feasible to use another webhook to do this, however, I am not sure at this stage that naming the webhooks alphabetically so that they operate mutation in a specific order is something that can be relied on in the longer term. I know it's practice, however, I can't see anything official that it's a standard that will be maintained.

If we did take that approach though looking at off the shelf solutions doesn't quite tick the mark, for example OPA Gatekeeper (and hence Anthos Policy Controller) only supports replacement of the domain part or the WHOLE path part including the image name - this cannot be coerced by putting a path into the domain part because of regex checks in the mutation code.

It would seem to be a request that isn't 100% aligned with digester, however, the question regarding the long term viability of applying mutations in a set order is disuading me from writing a new webhook to do this before checking here as this is a trivial change for digester and I would be happy to submit an MR including tests and documentation if it was of interest.

I chose to focus upon simple string replacement because my use case may not be quite what everybody else has and this could cater for other use cases for other users.

halvards commented 2 weeks ago

Thanks @iamasmith for submitting the idea and for the detailed explanation!

Wouldn't this be better handled on the client side, before sending the manifests to the cluster? E.g., using a Kustomize image transformer or a kpt setter. There may be other values that also need to change with different regions, e.g., endpoints for external dependencies such as caches, databases, other upstream services, etc. Orchestrating all changes client-side would likely lead to less confusion.

Regarding multiple webhooks and invocation order, k8s-digester sets reinvocationPolicy: IfNeeded and failurePolicy: Ignore in the MutatingWebhookConfiguration, so it should add the image digest regardless of invocation order. Worst case scenario it'll perform two registry lookups, where the first may fail if the placeholder image name doesn't exist.

iamasmith commented 2 weeks ago

@halvards thanks for the fast response, I need to dig in further on this as the request came proxy via a head of department for this ask.

They had been in conversation with a software architect who did not want to have to change image or anything else for the group of services that they govern and wanted to establish a working pattern to avoid changes like this and your response was pretty much verbatim my initial response. It's not necessarily right and pushes complexity elsewhere - often increasing it and we have scemarios where we have had to use mesh technologies to remap services so they are named identically between environments but actually go to different data sources, be they regional or development services rather than factoring in parameters to the deployment to cover these regional differences off.

It feels like they are perceiving the extra pipeline complexity is something that they are trying to avoid and possibly because in context there are over a thousand different services in our stack that are being rapidly moved to the cloud environment - the software changes are not difficult but the teams have relied on on-premises services teams to make those deployments similar.

It may be the wrong answer to cater for this in the way I mentioned vs handle it at source in kustomize or other tooling but it may also be something that as a team we are asked to provide.

wrt the failurePolicy: Ignore, we have this set to fail to avoid this https://github.com/google/k8s-digester/issues/25 which we enountered so reinvocation would require us to change that but as the estate grows rapidly then this also means that we'll have to add specific monitoring checks to look for this kind of runaway behaviour - we just find it safer to fail the deploy completely if the webhook response ever fails.

halvards commented 2 weeks ago

Let me know when you have more context. If you have questions outside of the specifics of k8s-digester that are easier to discuss via email, my work email address is visible on my profile page.