[Question] Will Kpt v1 come with a set of built-in functions

[yhrn]

Considering some existing core functionality, like setters, will be moved to a function along other transforms that will probably be considered core functionality in the future I'm wondering if Kpt will be delivered with these functions build in? There's a couple of reasons for this question:

1. Versioning - having to version the core functions separately from Kpt itself seems cumbersome.
2. Docker dependency - it would be nice to be able to use the core functionality of Kpt without requiring docker to be installed.
3. Performance - not having to pull images and launch containers could significantly improve performance, especially in a "clean/fresh" CI environment.

[wstrange]

Regarding the docker dependency: It really makes it more painful to run kpt safely inside a pod. For example - in a tekton pipeline. Today you need to map the docker socket - which a lot of environments will not allow.

Built in functions, and/or starlark like would avoid this.

[frankfarzan]

Having all functions be containerized is done for a number of reasons. Since functions are containerized, they can encapsulate different toolchains, languages, and runtimes. For example, the function container image can encapsulate:

• A binary built using kpt’s official Go or Typescript SDK
• Wrap an existing KRM tool such as kubeval
• Invoke a bash script performing low-level operations
• The interpreter for “executable configuration” such as Starlark or Rego

In fact, the basic set of functions that would be eligible if we had a "built-in" set are currently written in different languages and toolchains.

This model will sound familiar: functions are the client-side analog to Kubernetes controllers. Just as Kubernetes system orchestrates server-side containers, kpt CLI orchestrates client-side containers operating on configuration. One of the lessons we learned from k8s api machinary was having a built-in mechanism ("core" resources such Pod, Deployment) as a separate construct from extension mechanisms (CRD) will lead to unnecessary complexity.

By using containers, kpt can also provide some very useful guarantees: Functions can be hermetic. For correctness, security and speed, it’s desirable to be able to run functions hermetically without any privileges; preventing out-of-band access to the host filesystem and networking. This is critical when working with untrusted third-party packages and functions at scale.

I think @yhrn brings some good points about downsides of using containers, and I want to explore mitigations for them:

1. Docker dependency - it would be nice to be able to use the core functionality of Kpt without requiring docker to be installed.

Agree that any additional system requirement adds friction. Are you primarily thinking about local dev environment or CI/CD systems? In those environments, is docker not needed by other tooling?

2. Performance - not having to pull images and launch containers could significantly improve performance, especially in a "clean/fresh" CI environment.

I want to understand this better. Are you running kpt on a CI/CD system where there is no caching between invocations? I would like to explore adding caching to mitigate this. On our part, we can make the 1st party container images smaller if need be (e.g. using distroless builds).

3. Versioning - having to version the core functions separately from Kpt itself seems cumbersome.

This is actually a feature, not a bug IMO. There will be an ever growing number of functions which are independently released and versioned from kpt core. All 1st party functions will follow strict semver scheme and we guarantee that we don't break backwards compatibility without the user taking the explicit step of uprevving the functions used in their configuration to a new major version. Once kpt CLI 1.0 is released, a package you write will continue to work indefinitely with new releases of kpt CLI.

@wstrange Also brought up what we call "executable configurations" pattern such as Starlark or Rego. There's still a container involved there interpreting the configuration: starlark and enforce-gatekeeper functions respectively:

https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/master/functions/go

But the end-users doesn't need to develop custom images, just author configuration. Starlark specially is useful for small tasks where a) there are no existing functions available b) the task is not complex enough to warrant developing a function in a general purpose programming language.

[wstrange]

I thought starlark was builtin to kpt itself and did not require docker to be running?

I totally get (and love) the "UNIX" like pipeline concept. Docker is just such a pain the rear to run in Kubernetes itself. It's nice to be able to "virtualize" your CI/CD in K8S - turtles all the way down. We use Kaniko in Tekton to avoid DIND or mounting the docker socket.

Perhaps this is not kpt's issue to solve - but its a pain point that I am sure will come up.

[frankfarzan]

I thought starlark was builtin to kpt itself and did not require docker to be running?

In kpt 1.0, it's becoming just another function similar to other "runtime functions" such as Rego (gatekeeper) and CEL in the future.

I totally get (and love) the "UNIX" like pipeline concept. Docker is just such a pain the rear to run in Kubernetes itself. It's nice to be able to "virtualize" your CI/CD in K8S - turtles all the way down. We use Kaniko in Tekton to avoid DIND or mounting the docker socket.

Perhaps this is not kpt's issue to solve - but its a pain point that I am sure will come up.

FYI, There's a kpt fn export command that shows how to run kpt on various CI/CD systems:

https://googlecontainertools.github.io/kpt/guides/consumer/function/export/

For Tekton, it mounts the docker socket which you don't seem a fan of. What problems does it cause in your experience? Believer it or not, we don't have a lot of users running Tekton. Majority of the use case is running kpt on {local, Cloud Build, GitHub actions}.

[wstrange]

Mounting the docker socket in a K8S cluster is risky, and lots of security focused organizations won't allow it due the possibilty of container escape, etc. I suppose if you are running a private cluster, and you trust your developers its not a huge deal (trying to be pragmatic here..). This is really the entire motivation for Kaniko (docker builds without the docker daemon).

Things like Google Cloud Build provide isolation beyond the docker daemon - so less of an issue in those environments.

I just think a Kubernetes native tool ought to run well... in Kubernetes ;-)

[frankfarzan]

Yes, we should have a good story for running functions on k8s. Full stop. However, not containerizing the functions is not the solution and it would like throwing out the baby with the bath water.

[yhrn]

@frankfarzan

1. Docker dependency - it would be nice to be able to use the core functionality of Kpt without requiring docker to be installed.

Agree that any additional system requirement adds friction. Are you primarily thinking about local dev environment or CI/CD systems? In those environments, is docker not needed by other tooling?

I was also mainly thinking about running in k8s. On use case I was considering was an operator using setters to hydrate manifests based on CR fields values (using something like kubebuilder declarative pattern

2. Performance - not having to pull images and launch containers could significantly improve performance, especially in a "clean/fresh" CI environment.

I want to understand this better. Are you running kpt on a CI/CD system where there is no caching between invocations? I would like to explore adding caching to mitigate this. On our part, we can make the 1st party container images smaller if need be (e.g. using distroless builds).

Our home rolled CI system does cache images but it tries to be smart about it and will only cache images referenced as "action containers" in its domain model, i.e. it doesn't understand Kpt functions and will not keep them in the cache after a successful build. This is obviously something I could try to get somebody to look at but it's not something I work with directly. Smaller images would be a good thing.

3. Versioning - having to version the core functions separately from Kpt itself seems cumbersome.

This is actually a feature, not a bug IMO. There will be an ever growing number of functions which are independently released and versioned from kpt core. All 1st party functions will follow strict semver scheme and we guarantee that we don't break backwards compatibility without the user taking the explicit step of uprevving the functions used in their configuration to a new major version. Once kpt CLI 1.0 is released, a package you write will continue to work indefinitely with new releases of kpt CLI.

First I want to clarify that I'm not suggesting that any functions should only be built-in. My thought was that basically the Go functions in kpt-functions-catalog could be built into the Kpt binary but would also be available as images for anybody wanting more control over versions.

My concerns are:

1. In the run locally/manually use case users will have many more versions to keep track of and decide when to update. Semver will not stop regressions from occurring and I know that teams using Kpt or Kustomize are already struggling a bit with making sure that everybody is using the same (or at least new enough) versions, e.g. when running various helper scripts. If there ends being 10x the number of versions to keep track of for what is perceived as core functionality I think that will be seen as a very negative thing.
2. When used in automation/CI we want to stay up-to-date for all the normal reasons like receiving security patches, etc. We have some automation around keeping CLIs like Kpt up-to-date but keeping function image versions up-to-date adds another layer. I'm aware of the fact that we'll be running more functions than the "core" ones but I'm guessing that for most cases it will either be the "core" ones or 1p functions that are more natural that we'll keep up-to-date anyway.

[frankfarzan]

@frankfarzan

1. Docker dependency - it would be nice to be able to use the core functionality of Kpt without requiring docker to be installed.

Agree that any additional system requirement adds friction. Are you primarily thinking about local dev environment or CI/CD systems? In those environments, is docker not needed by other tooling?

I was also mainly thinking about running in k8s. On use case I was considering was an operator using setters to hydrate manifests based on CR fields values (using something like kubebuilder declarative pattern

1. Performance - not having to pull images and launch containers could significantly improve performance, especially in a "clean/fresh" CI environment.

I want to understand this better. Are you running kpt on a CI/CD system where there is no caching between invocations? I would like to explore adding caching to mitigate this. On our part, we can make the 1st party container images smaller if need be (e.g. using distroless builds).

Our home rolled CI system does cache images but it tries to be smart about it and will only cache images referenced as "action containers" in its domain model, i.e. it doesn't understand Kpt functions and will not keep them in the cache after a successful build. This is obviously something I could try to get somebody to look at but it's not something I work with directly. Smaller images would be a good thing.

1. Versioning - having to version the core functions separately from Kpt itself seems cumbersome.

This is actually a feature, not a bug IMO. There will be an ever growing number of functions which are independently released and versioned from kpt core. All 1st party functions will follow strict semver scheme and we guarantee that we don't break backwards compatibility without the user taking the explicit step of uprevving the functions used in their configuration to a new major version. Once kpt CLI 1.0 is released, a package you write will continue to work indefinitely with new releases of kpt CLI.

First I want to clarify that I'm not suggesting that any functions should only be built-in. My thought was that basically the Go functions in kpt-functions-catalog could be built into the Kpt binary but would also be available as images for anybody wanting more control over versions.

My concerns are:

1. In the run locally/manually use case users will have many more versions to keep track of and decide when to update. Semver will not stop regressions from occurring and I know that teams using Kpt or Kustomize are already struggling a bit with making sure that everybody is using the same (or at least new enough) versions, e.g. when running various helper scripts. If there ends being 10x the number of versions to keep track of for what is perceived as core functionality I think that will be seen as a very negative thing.
2. When used in automation/CI we want to stay up-to-date for all the normal reasons like receiving security patches, etc. We have some automation around keeping CLIs like Kpt up-to-date but keeping function image versions up-to-date adds another layer. I'm aware of the fact that we'll be running more functions than the "core" ones but I'm guessing that for most cases it will either be the "core" ones or 1p functions that are more natural that we'll keep up-to-date anyway.

But what does it mean for a "core" function not to be explicitly versioned? It could mean:

a) Do not provide backwards compatibility guarantees (no semver) b) Implicitly provide semver major version 1 indefinitely.

I think you're talking about b) since it's unacceptable to have to change all existing kpt package when updating the kpt CLI. Our current plan is not radically different:

c) Explicitly provide semver but make it ergonomic.

In option c), we tag images in 3 different ways:

1. vX.Y.Z
2. vX.Y
3. vX

If you specify just the major number (vX) in the configs, then kpt will automatically update the minor and patch numbers (See #1798). Similarly, if you specify major/minor (vX.Y), then you automatically get patches and security updates.

So if you're interested in option b), then always provide v1. You don't need to think about versions after that (Until you actually want to use a backwards-incompatible new hotness). Does that address the above concern?

@mengqiy FYI.

The other thing is that we don't have a concept of "a core function written in Go" today. Common 1st party functions are developed in different toolchains (e.g. kubeval which we collaborated with you is a TS wrapper that invokes the kubeval binary.).

[yhrn]

But what does it mean for a "core" function not to be explicitly versioned?

I'm not sure that I have said that a "core" function shouldn't be explicitly versioned. What I'm trying to suggest is that version X.Y.Z of kpt could include version A.B.C of one "core" function and D.E.F of another. From a semver perspective I guess that this would require Kpt to bump the major version whenever a core function does so or include both major versions of the function from then on. But I also wonder how big of a problem that would be, i.e. how common new major versions will be. On the other hand I feel that I'm missing a bit of context here on how functions will be used in Kpt v1 so my thinking might not make much sense.

As for option c) in the context of any automation this doesn't help much if you also want fully deterministic, reproducible builds.

The other thing is that we don't have a concept of "a core function written in Go" today. Common 1st party functions are developed in different toolchains (e.g. kubeval which we collaborated with you is a TS wrapper that invokes the kubeval binary.).

Yes, I realize this but looking at the Go functions my feeling is that this might be decent set of "core" functionality for manipulating KRM (potentially excluding enforce-gatekeeper). Things like kubeval feels a bit less "core" to me.

I realize that from an architectural perspective it's much cleaner to not do any of what I'm suggesting, but I guess what I'm worried about is that this will be a nice modular, pluggable architecture that is also somewhat impractical to use. I truly do believe that if a team is considering what new tool to introduce and it turns out that one of the tools require you to either manage 5-10 separate component versions or use "mutable tags" such as v1 to get functionality, that is generally expected to be part of the tool, it will be seen as a negative.

I also don't want to drag this discussion out indefinitely. I think I have explained my concerns, and especially around versioning, you can just see it as concerns, not something we see as a show-stopper.

[frankfarzan]

I think there are orthogonal concerns here:

1. Executing function containers on k8s (e.g. Using Tekton) https://github.com/GoogleContainerTools/kpt/issues/2158
2. Versioning: I opened a separate issue #1905 for this, and provided a strawman of what I think you're looking for. This is orthogonal to whether a function is containerized or not. Even if execute Go binaries, they would still have versioning and the same concern.
3. Performance concerns around executing container images such as how images are cached on CI/CD systems. https://github.com/GoogleContainerTools/kpt/issues/2304

[frankfarzan]

Each of the the three areas discussed have tracking issues now. Closing this one.