Open Scalahansolo opened 5 months ago
Agreed! WDYT @jedevc @sipsma @shykes?
No objections, I think this could definitely be an improvement for running dagger functions in k8s/etc :tada: Previously, in the standalone SDK model, I don't think this was as important, since you could just use a docker runtime for your SDK language, but with the module model, we only need a single CLI entrypoint, and the engine provisions the language component.
I'd suggest we put this in registry.dagger.io/cli
to contrast with registry.dagger.io/engine
- that was it's clear which one is which, and users don't end up with the wrong one!
Ideally, this can be a very lightweight image: it just needs the dagger CLI, but may also need to include git
/etc and any other referenced binaries.
The steps to do this would be:
cli
and engine
images. I think in the future they will be more tightly coupled: one will be "CLI only" and the other will be "CLI + buildkit runtime". This is a topic for production readiness.@dlorenc suggested that Wolfi might be a good candidate for this, thoughts?
@shykes could you elaborate a bit on point 2? I think the engine image probably won't have the CLI embedded, it'll just have the engine as we build it today - I don't necessarily see that changing dramatically, but I guess this also touches on #5583 (like everything else I guess lol).
For 3, for the CLI, I actually think we might be able to do something like distroless? I think we probably just need a little collection of static binaries. For the engine, I would like to unify the GPU/standard builds to use a common base - ideally, I'd also like to avoid ubuntu, due to size constraints. If we could get alpine/wolfi to install all the right CUDA stuff (agh, it would be nice to have some tests for this in CI) - then yup, we should go all in on that :tada:
Happy to work with you on the cuda libraries! We're getting those packaged now. We can setup something distroless with CUDA for you!
@jedevc can you share an example build with CUDA that we can use for testing with wolfi?
Nice, thanks both! :tada:
I don't think we have a guide, but I can point you to a couple places (sorry, just throwing something together around kubecon):
Container
needs to be chained using ExperimentalWithAllGPUs
- see here._EXPERIMENTAL_DAGGER_RUNNER_HOST
.I don't want to derail y'alls thought process on the split between the engine and CLI images, but the CLI image is definitely pretty pressing for the team over at Motion, especially since the Official Docs for getting the Dagger CLI running on a gitlab runner is failing for me when I try to use it on a self hosted Gitlab runner.
@Scalahansolo happy to follow-up with the gitlab issue on discord if you want! I'd guess though that maybe .local/bin
isn't in the PATH
- could you share the output of echo $PATH
before you run the dagger --help
step?
(this is potentially an issue in our guide as well, cc @vikram-dagger)
(this is potentially an issue in our guide as well, cc @vikram-dagger)
The change to use .local/bin
was made in https://github.com/dagger/dagger/pull/6262/commits/ab0a8019dc29958fa4063c264edeba1621ad7019, so it would be good to have @gerhard 's thoughts on this and also how to change it in the install doc
For what it's worth, I would love if this custom CLI image had git
in it. Thought process here is that due to https://github.com/dagger/dagger/issues/6113 we will need to be able clone our dagger repo into our CI runners in order to call the functions until that issue is closed out.
For what it's worth, I would love if this custom CLI image had
git
in it. Thought process here is that due to #6113 we will need to be able clone our dagger repo into our CI runners in order to call the functions until that issue is closed out.
Great idea. We can manage the contents of the CLI container image... With a Dagger pipeline of course :)
I created a Wolfi image with dagger, git and nvidia libraries. This was built with apko, I realise the final version will need to built inside dagger, but thought you might like something to play with and check github workflows etc:
$ docker run -it amouat/dagger-cuda
b3b8c3e3308b:/# dagger version
dagger v0.10.3 (registry.dagger.io/engine) linux/arm64
b3b8c3e3308b:/# curl https://google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
b3b8c3e3308b:/# git version
git version 2.44.0
b3b8c3e3308b:/# nvidia-smi
<error message as I don't have a GPU and kernel modules>
There's a version without cuda at amouat/dagger
.
The cuda version was built from the following apko:
contents:
keyring:
- https://packages.wolfi.dev/os/wolfi-signing.rsa.pub
- https://packages.cgr.dev/extras/chainguard-extras.rsa.pub
repositories:
- https://packages.wolfi.dev/os
- https://packages.cgr.dev/extras
packages:
- ca-certificates-bundle
- wolfi-base
- git
- curl
- dagger
- nvidia-driver
- nvidia-tools
entrypoint:
command: /bin/sh -l
The shell can be removed by deleting wolfi-base.
I'm not sure of the GPU use case -- it might be that we need to add more libraries/binaries to the image.
I have recently built my own dagger-cli
base image using apko
. Once there is an official image, I will definitely switch over. This image is designed for use within CI platforms like GitLab, https://github.com/purpleclay/dagger-cli
I really like where this discussion is going. Big supporter of having the CLI baked into the Engine image. There should be a single canonical artefact for all of Dagger. cc @shykes
Agreed. From a recent [discord discussion]():
EXPERIMENTAL_DAGGER_RUNNER_HOST
; buildkit.toml
and its contents; /var/run/buildkit.sock
. Generally anything with the name "buildkit" in it, should be made private and exposed via the CLI.
- The engine OCI image should include the CLI. It should be the canonical way to distribute the CLI.
I'd like to highlight one potential "downside" about this aspect which is the resulting size of the underlying OCI image. The last (v0.11.4) image of the engine is around 450mb and even though it's not a huge size, it adds up for each CI pipeline that needs to download it vs the few MBs that a CLI only image could result in.
What are you trying to do?
The dagger docs for gitlab / github / etc have each job installing curl followed by installing the dagger CLI to use for a given CI run. It would be much nicer DX, and probably faster to just have an official docker image that can be used that has the CLI already installed?
Why is this important to you?
No response
How are you currently working around this?
No response