search

tensorchord / envd

🏕️ Reproducible development environment
https://envd.tensorchord.ai/
Apache License 2.0
2.03k stars 159 forks source link

feasibility-research(doc): Remote build with buildkit or kaniko #510

Closed gaocegege closed 2 years ago

gaocegege commented 2 years ago

Description

We should investigate how to build envd images remotely with buildkit or kaniko. And we should document it.

CI/CD systems to support:

Message from the maintainers:

Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.

gaocegege commented 2 years ago

/assign

gaocegege commented 2 years ago

It is hard to support envd with kaniko since it does not support dockerfile/1.2.0 or 1.3.0 now. RUN --mount=cache is hard to run.

https://github.com/GoogleContainerTools/kaniko/issues/1712

gaocegege commented 2 years ago

Currently, we support context in #512 . In the next PR we should support kube-pod builder in the context.

gaocegege commented 2 years ago

We need to decide if we should use kubernetes client-go or controller-runtime client to create the buildkitd pod on Kubernetes.

I think the client in controller-runtime wraps client-go a lot and it should be easier to use.

WDYT @hezhizhen @knight42

knight42 commented 2 years ago

I think the client in controller-runtime wraps client-go a lot and it should be easier to use.

@gaocegege Yeah the client provided by controller-runtime is a general one and is more flexible(if we need to access some custom CRs, we only need to register theire scheme).

The only limitation came to my mind is that if we need some functionalities more than objects' CRUD, such as executing a command in container, streaming container logs, port-forwarding etc, then we could only use client-go.

gaocegege commented 2 years ago

Thanks for the explanation! forget that the controller-runtime client does not support sub-resources. I think we should use client-go directly.

arcosx commented 2 years ago

Hi, I've been working on something similar recently.I also think it is possible to use client-go directly. As I see https://github.com/docker/buildx/blob/57156ee95cccc172e3757520283b04d09f1646e0/driver/kubernetes/driver.go#L192 and https://github.com/vmware-tanzu/buildkit-cli-for-kubectl/blob/3cdb0965332177c4f254ea81b96c9ce7b8c46d3d/pkg/driver/kubernetes/driver.go#L229 have related work in them that is similar to your needs. They both do it by passing in the client-go and reading the local kubeconfig.It actually create a tcp channel between the local and Pod.

gaocegege commented 2 years ago

Yep. Currently, we support communicating with the buildkit on the Kubernetes with the help of context:

envd context create --name test --builder-name <pod-name> --use --builder kube-pod

Next step is to support envd up with the help of client-go.

aseaday commented 2 years ago

It this issued related to https://github.com/tensorchord/envd/issues/556 I want to solve that issue but there seems to be a lack of specific context.