[feature request] use kaniko within a custom container

[samuela]

For my use case, I need to be able to run something like docker build or kaniko from within a container. I don't care if kaniko runs as root from within the container, so long as I don't have to run the container in --privileged mode. Ultimately I'd like to be able to build untrusted Dockerfiles within a Kubernetes cluster.

From the README, it sounds like this is not advisable: "kaniko is meant to be run as an image, gcr.io/kaniko-project/executor. We do not recommend running the kaniko executor binary in another image, as it might not work."

What would it take to enable this use case?

[morsik]

Kaniko is just Go application, so it should work in another system, but it modifies system it works on and snapshots it. Maybe some paths are hardcoded and this could make problem.

I'm not dev of this project, but what's your use case? If you use CI you can just run kaniko image in next stage and use build artifacts to retrieive stuff from previous stage. That's how I do it in GitLab CI.

[priyawadhwa]

Hey @samuela, is there any reason you can't use the kaniko image directly? It sounds like it would be a good fit for your use case, as kaniko is meant to build images in a Kubernetes cluster without the --privileged flag.

[samuela]

I need to pull in some sources in a custom way, and then mess around a bit before passing things off to the build system. Along the way I also need to snag the logs of the container build step in real-time. That would all be pretty straightforward if I could just call kaniko as a CLI like docker from within my own container, but I'm not sure that it would be possible with the current recommended kaniko usage model.

[priyawadhwa]

You could try injecting the kaniko executor binary in your own container and running it when you need to build the image, but no promises it will work. Let us know how it goes :)

[ffahri]

@samuela hey did you try that if so did you encounter any problem with custom container kaniko ?

[samuela]

@ffahri No, I've switched back to docker in a VM for the time being. I may take another look at kaniko in the future but not being able to use kaniko as a simple executable (or at least support for that use case) was a dealbreaker for us.

[miguelitoq76]

@priyawadhwa Hello, we are currently using only the kaniko executor binary in a custom image based on Ubuntu 16.04 (without docker,gvisor), add the beginning we had some issues. but now we are able to build applications (e.g. a LAMP-stack, a tomcat-server) in a gitlab-ci pipeline.

We found some workarrounds -based on some posts here-

a) the dockerfile, has to start with a FROM scratch FROM -your base image-

Without the FROM scratch we are facing some interferences, between the buildsystem and the build itself

b) the config files should be located on /kaniko/.docker/ this path. Other path e.g /root/* is not protected.

If you have more questions, please ask... This is my first post here....

Now a question/request to make this more handy... Is it possible to run the FROM scratch by default, before our Dockerfile would be processed? Our do you know any other solution for this?

Thanks a Lot! This is a great tool!

[priyawadhwa]

Hey @miguelitoq76, super cool that you were able to get kaniko to work in an ubuntu container 😎!

Unfortunately, since we build kaniko with the intention that it will only run in our official image, we can't implement FROM scratch by default. I'm actually not sure why adding FROM scratch is fixing the problem, so I'm not sure if there's a better way to do it. Please let us know if you learn anything more or have any more questions :)

[miguelitoq76]

@priyawadhwa to have a clean image without interference with the starting image, we need a Deleting filesystem before the build starts... Without this we had some problems to create some symlinks into a SSL folder.

With the FROM scratch and the executor --cleanup we get a clean filesystem, and we can run a imagebuild without problems..

other workarround executor --no_push --cleanup --dockerfile=${echo "FROM scratch"}

So far, it would be nice to have a switch to delete the filesysteme before the build starts... The executable and the configurations are in the kaniko folder...

if the build starts with this sequence, we got all symlinks and files: (INFO[0002] Deleting filesystem... !!!) INFO[0000] No base image, nothing to extract
INFO[0000] Taking snapshot of full filesystem...
INFO[0002] Deleting filesystem...

INFO[0002] Deleting filesystem... solves some issues... Why?

[cvgw]

Supporting kaniko in custom containers is out of scope for the timing being, but we can leave this issue open for future discussion

[jaideepr97]

Hey everybody,

I have a similar use case to what has been described here by other folks, where I need to inject a tar ball into the kaniko container before I can let the build begin, but as of now I don't have a way to manually trigger a kaniko build at a time when I am ready to (after the container is already running )

super cool that @miguelitoq76 got it running in an ubuntu container, but I was wondering if there was any way I could use the kaniko:debug image in order to somehow access the busybox shell before the actual build is started if someone has tried something along similar lines or has any other ideas it would be super appreciated!

[sumanthkumarc]

+1 for simple kaniko binary instead of using entire image. binary is definitely lot more convinient and easy to work with .

[karolmie1]

Kaniko was created to run without docker daemon, yet it requires one to run the image. It's a bit self defeating. In my case it makes running whole build logic as simple gradle invocation impossible on cloud.

[morsik]

@karolmie1 then don't use Docker Daemon ;) You can always use podman to run Kaniko. Documentation clearly says, you need to run it as container.

Also, I don't think this is the case for this issue anyway - it tells about different problem.

And, since the method how Kaniko is working (taking snapshot of whole filesystem) - it's quite difficult to run it outside of container currently.

[trallnag]

With the following Dockerfile I managed to build a kaniko-alpine image from within a kaniko image:

ARG ARTIFACTORY_REGISTRY_DOCKER

FROM $ARTIFACTORY_REGISTRY_DOCKER/gcr.io/kaniko-project/executor:v1.6.0-debug AS kaniko

FROM scratch
FROM alpine

ENV PACKAGES \
    git \
    bash \
    coreutils

RUN apk --update-cache add $PACKAGES

COPY --from=kaniko /kaniko/.config /kaniko/.config
COPY --from=kaniko /kaniko/.docker /kaniko/.docker
COPY --from=kaniko /kaniko/executor /kaniko/executor
COPY --from=kaniko /kaniko/ssl /kaniko/ssl
COPY --from=kaniko /kaniko/warmer /kaniko/warmer

ENV HOME /root
ENV USER /root
ENV PATH $PATH:/kaniko
ENV SSL_CERT_DIR /kaniko/ssl/certs
ENV DOCKER_CONFIG /kaniko/.docker/

References:

• https://github.com/Pernod-Ricard/kaniko-alpine/blob/main/Dockerfile