Open mengqiy opened 2 years ago
cc: @aodinokov
I think this is a great list. I would suggest that if we put a GRPC/REST interface in front of function execution, a lot of these can look very similar to each other, and we can also amortize the launch time over multiple invocations.
The most commonly used catalog functions could just be built-in, as in kustomize. #1905
Remote functions: The approach is interesting, but I don't foresee offering managed endpoints. Users would need a function platform of some kind to run them.
And, yes, we do need alternatives to docker. Current performance is unusable. #2469
For completeness, we also used to support export to orchestrators: #226
I would suggest that if we put a GRPC/REST interface in front of function execution, a lot of these can look very similar to each other, and we can also amortize the launch time over multiple invocations.
Do you mean the 4th option ? or something else ?
And, yes, we do need alternatives to docker. Current performance is unusable.
Users who don't want to use docker for (performance or licensing or other reasons), exec
or starlark
runtime will help them only for their custom functions. But if they need to use common functions such as set-namespace
etc from the catalog, then either we will have to provide binaries for catalog functions or provide built-in in kpt. So these three (exec
, runtime
and built-in
) will be required to eliminate the docker dependency completely. Distribution/packaging of binaries or starlark (reusable) and ensuring security is something users will have to solve themselves.
@karlkfi also had some thoughts / concerns
To summarize my concerns...
Executing functions in containers creates significant limitations:
docker run
, with all its overheadkpt fn render
would be even slowerAll of the above make it really hard to use kpt fn render
as part of a server-side product.
Possible Solutions:
All of the above options allow running kpt fn render
in a container, without requiring a VM or a VM scheduler.
I've been playing with alternative execution engines in kpt & kustomize. A sneak-peek is here: https://github.com/justinsb/kpt/commits/runtime_engines I've been operating on the assumption that there is no one-size-fits-all solution, in that different OSes and environments will require different approaches. But I think we can eliminate a lot of the overhead, and there's no need for Docker. Running containers-in-containers is a critical use-case and one of my goals is to understand which of the options work (well) inside a container.
I like @karlkfi 's suggestion of supporting k8s pods as a target. Supporting remote execution via REST/gRPC also makes sense IMO, I have a (more complicated) exploration where we do that even if we launch a container locally and it works well, it does let us amortize the launch cost, though that cost is pretty minimal with direct linux-namespace sandboxing.
For smaller functions, I think the slowest step is now loading the OpenAPI which we do to reattach comments - optimizing OpenAPI would be a big win across the board.
Pluggable engines makes sense to me. These options don't need to be mutually exclusive. I'm looking forward to see whether we can make Starlark simple enough. The typescript sdk isn't bad, and maybe useful in a UI. I also think we should have built-in functions. And "docker" shouldn't be a requirement.
I like the approach of unpacking the containers and basically just exec'ing them in chroots. If cgroups work, that's also fine. The whole pipeline would run in a container. The model is one of shared responsibility, just like running any containers in Kubernetes. Doesn't seem inherently worse than helm charts. At some point I imagine verifying signatures. And perhaps other metadata, along the lines of grafeas.io
When executing functions as part of a service, we'll need to decide what identity to run them with. Launching them on demand makes it easier to use the launcher's identity.
Couple of comments/suggestions:
A CUJ we want to enable: I should be able to fetch any off-the-shelf public package and be able to render it with zero set-up. As such, I think executing binaries is worst option. You lose portability and need to effectively re-implement mechanisms provided by containers (e.g. image registry and caching, limiting host network or filesystem access, etc).
While the options for provders aren't mutually exclusive, the options for what we invest in for our standard package catalog is more constrained. We can't reasonably port every function to every function provider. If only some of the providers work both server-side and client-side we might want to focus our efforts on those.
Ref #2302
Replace the kpt fn pipeline with a Kubernetes pipeline operator, using a CRD as interface. This would be similar to Argo Workflows, except more specific in purpose.
@karlkfi We have a use case in Crossplane (https://github.com/crossplane/crossplane/issues/2524) in which we'd like a controller to use KRM functions, and were thinking along the same lines as this. ArgoCD lays out a few of the approaches they've explored to execute container pipelines at https://argoproj.github.io/argo-workflows/workflow-executors/. In my mind something like the emissary
approach is the most appealing - that is, injecting a 'wrapper' entrypoint into each of a pod's containers to ensure they:
One snag is that emissary
requires workflows to explicitly specify the command and args for each containerized process in the workflow so that emissary
knows what it needs to wrap - presumably the controller could figure this out itself by introspecting the OCI image metadata but that would involve the controller pulling it down (separately from the Kubelet doing so). In the context of KRM functions this could possibly be alleviated by extending the function spec as it exists today to mandate/recommend a predictable entry point (/bin/krm-fn
or something).
We really like the idea of a pipeline of containers, since it seems like that has the lowest barrier to entry for our users. We'd like to also support web hooks - probably KRM functions as web hooks - but we're conscious that there's potentially a higher barrier to entry on that approach as the issue here mentions.
I would like to use
kpt fn run
inside CI/CD pipelines that run in a cluster... https://github.com/GoogleContainerTools/kpt/issues/2158
So as far as I can tell, a pipeline based on Kaniko is incompatible with kpt fn render
at the moment, correct? I'd really like to get resource mutation off my local machine.
For what it's worth, I've had luck with running buildkitd as a deployment in kubernetes, along with a docker-in-docker deployment/service. If I set DOCKER_HOST
in my build stage environments, to point to the docker service, things like kpt work. The result is that I can run docker buildx build, bake etc
and it works just like I was running it on my laptop in terms of processes expecting docker availability.
Example here: https://github.com/bzub/kpt-pkgs/blob/37fd0dc305b1463124e19065e353f21a6e501fae/Dockerfile#L28
Update: I've since switched to using exec
instead of image
for functions in CI/CD: https://github.com/bzub/kpt-pkgs/pull/3/files
We're also exploring WASM: #3299.
What about developing a Kptfile frontend for buildkit? buildkitd
is easily deployed via dockerd, container, Kubernetes deployment/statefulset, etc. Especially with the buildx
tool. The way KRM functions and hydration via kpt work reminds me a lot of a Dockerfile with multiple build stages working on input/output files in series. Also a kpt function runtime option that is trivial to run without root privileges would be huge for me at least, which buildkit seems to deliver.
@droot is this a P0?
There are at least the following runtimes that we can consider to support in kpt:
The following table compares the pros and cons of these runtimes.
Feedbacks and thoughts are welcome!