Run kaniko as non-root user within the container

[dlorenc]

Right now kaniko requires root for a few reasons:

• To unpack the base image into / with proper permissions
• To execute RUN commands that require root

We should be able to solve the first case using an init container. This container will copy the executor binary into a shared volume.

Then the "build" container image is actually the base image in the FROM line, and the entrypoint gets set to the binary we just copied into the shared volume.

This also takes advantage of layer caching by letting the underlying runtime unpack the base layers.

We can't really fix the second part until something like user namespaces are supported in kubernetes, but we can allow builds to run with only the permissions they need until then.

[cyphar]

Using the tools we've been developing as part of the rootless-containers project (https://github.com/rootless-containers) it is possible to do both of those things completely unprivileged (though it does require ptrace to do it convincingly). You can even do it inside a container, by nesting the rootless container inside the outer one (though at the moment there are some runc bugs that we are working on resolving).

[dlorenc]

Yeah, the ptrace trick is something we've looked at with fakeroot-ng (although ptrace is blocked by the default docker capability set). Do you have links to the runc bugs you're working through?

[dlorenc]

@cyphar do you have any links on running runc inside a container without something like the "--privileged" flag?

[AkihiroSuda]

@dlorenc

There are two options:

1. Modify container runtime implementations to support RawAccess-ing /proc

Docker-side PR: https://github.com/moby/moby/pull/36644 Kubernetes-side proposal: https://github.com/kubernetes/community/pull/1934 Jess's blog: https://blog.jessfraz.com/post/building-container-images-securely-on-kubernetes/

2. Modify kernel to allow mounting unprivileged and non-fully visible procfs

https://patchwork.kernel.org/patch/10322481/

[cyphar]

Yeah, the ptrace trick is something we've looked at with fakeroot-ng (although ptrace is blocked by the default docker capability set).

I would have to double-check this, but when you set up a rootless container you have the full capability set (though we drop capabilities just as in the privileged case, so you'd need to add CAP_SYS_PTRACE to the capability set). So it is possible that the seccomp profile will not blovk ptrace once inside the rootless container (I will have to test this first though, since I'm not sure which *_capable variant is being used by seccomp in that context.

do you have any links on running runc inside a container without something like the "--privileged" flag?

At the moment the main issue us the /proc mounting problem that @AkihiroSuda just mentioned. Aside from that issue, you should be able to just run a rootless container in a Docker container (with the note that you need to whitelist unshare(CLONE_NEWUSER) in the seccomp profile). Another fix for the /proc mounting issue is to mount an empty procfs from an empty pid namespace inside the Docker container, which will fix the EPERM you currently get.

[r2d4]

Why do you need a nested runtime to begin with?

Runtime configuration security can always be configured in the "parent" build container, seccomp profiles applied, etc. You would do this in Kubernetes by setting pod security policies (of course not all the features are fully implemented today).

Kaniko is meant to be ran from inside a container only and never as a standalone "runtime". It almost seems like the wrong separation of concerns to have set of default security applied during the "parent" container, and then have the nested container apply something else (albeit only more strict).

I can't imagine nested containers would be easier to manage with Kubernetes non-container security policies like RBAC, since you would really like to scope whoever is running the build to the nested non-root container policies, which is the minimal scope.

[cyphar]

The nested containers suggestion is so that the build doesn't require root (which it currently does). Docker (or cri-o or Kubernetes -- though we are working on those slowly but surely) doesn't support rootless containers.

[r2d4]

Ah I guess I meant once docker or cri-o supports rootless containers, then we won't need the nested container? The only purpose of the nested container is that the parent runtime doesnt support rootless while the nested runtime does.

[cyphar]

Sure, though with Docker and cri-o you would still be in a bit of grey area (in most cases I would assume the kubelet would be running as root even if we had rootless support in the entire stack -- though the CloudFoundry folks have shown that people are actually okay with rootless in a lot of circumstances).

Really it depends how much you want to push the rootless thing. Personally (for the stuff I'm working on) I think that if you have any code-path that requires uid=0 then it is not an acceptable solution, so I wouldn't classify having cri-o (as root) spawning "rootless" containers as being acceptable. But of course different people have different levels of fanaticism. :smile_cat:

(Note that Docker will likely never support rootless directly, purely due to the amount of work required -- though I'd be happy to be proven wrong. containerd has some patches for it but it requires running in a specific environment unlike umoci and similar.)

[ianmiell]

This is a very disappointing thread to read. I was given to understand from the publicity that kaniko did not require any privileges:

https://cloudplatform.googleblog.com/2018/04/introducing-kaniko-Build-container-images-in-Kubernetes-and-Google-Container-Builder-even-without-root-access.html

this is a bit of a holy grail for our org, since the build environment is very locked-down.

[cyphar]

@ianmiell You can do unprivileged builds with umoci. If you prefer Dockerfile-style builds there's orca-build (I am currently working on combining orca-build into the umoci project so it's more obvious). There's also a lot of rootless containers work being done in https://github.com/rootless-containers.

Several organisations already use umoci because it provides completely unprivileged builds (you can pair with rootless runc to also run containers with it). See https://rootlesscontaine.rs/ for some more detail of what is going on there.

[ianmiell]

@cyphar thanks, but I was hoping for something more ready-packaged. I've a house full of yaks already :)

[dlorenc]

I agree we need to clarify the documentation and messaging a bit more here.

Kaniko was designed to run in a container in any kubernetes cluster, today. umoci and a few other tools are really cool too, kubernetes and docker just don't support the necessary settings to let them run without the --privileged flag.

@ianmiell, can you describe exactly how your build environment is locked down today?

This issue is about letting kaniko run without root at all, provided your build doesn't require root to run. User namespaces will be required to let builds that require root run without root on the host, but that's not possible today in kubernetes without the --privileged flag or something similar.

[ianmiell]

If anyone's interested, I've got a reproducible build of a poc using the tools described by @cyphar above (and with his help) here:

https://github.com/ianmiell/shutit-orca-build

on Centos. Does require a little sysctl work to enable (in the script).

[cyphar]

Regarding yak-shaving, yeah we're working on it. @AkihiroSuda has runrootless which wraps a lot of the stuff we have been working on -- but at the moment there are quite a few other problems that need to be solved before we can even start polishing it (unprivileged network bridges for instance, as well as a lot of other ancillary kernel work such as bind-mount mappings). And the current ptrace method that PRoot uses will no longer be necessary once @tych0 gets full-on seccomp syscall emulation merged into Linux (making emulation much faster, and easier to do (in theory)). A lot of this stuff is still changing.

[norpol]

Just stumbled upon Kaniko once more and asking myself if with the Docker gVisor support, is it now possible to build images without root or docker-in-docker within a Docker or Kubernetes host?

[dlorenc]

Yup, if you have gvisor configured you should be good to go: https://github.com/GoogleContainerTools/kaniko/blob/master/README.md#running-kaniko-in-gvisor

[ghost]

For anyone searching for a solution on how to run kaniko on a podsecuritypolicy-secured k8s cluster, I found this article useful (best security you can get with kaniko on k8s).

[yehiyam]

@kcatro could you share a link to the article you mentioned?

[ghost]

Oh boi, failed on ctrl+v -.-

Here's the article: https://kurtmadel.com/posts/native-kubernetes-continuous-delivery/building-container-images-with-kubernetes/. I had to ditch the seccomp annotations though for server v1.11.8. When present, they just do not allow kaniko pod to start.

[shakenfr]

This is a very disappointing thread to read. I was given to understand from the publicity that kaniko did not require any privileges:

100% agreed with this remark !!

[cmdjulian]

Any news on the issue? Would be really cool to run kaniko as non-root user inside the container.

[srfrnk]

I'm trying to run Kaniko within a Kubernetes context in a secure fashion. All guides recommend using container securityContext as such:

securityContext:
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  runAsNonRoot: true
  capabilities:
    drop:
      - ALL

However setting that makes Kaniko fail. Any thoughts on how I can use Kaniko witout breaking the security of my system?

[06kellyjac]

runAsNonRoot: true requires a non 0 UID but kaniko requires a 0 UID so as-is no. I do want to test some stuff though.

[srfrnk]

Thanks @06kellyjac - I've started looking into that myself... forked the repo and created a branch with a test case... you can see it here Maybe I could be of help...

[cmdjulian]

Following minimal permissions are working for me @srfrnk, all other caps are not needed, and as stated beforehand, currently kaniko has to run as root.

securityContext:
  capabilities:
    drop: [ALL]
    add: [CHOWN, FOWNER, SETUID, SETGID, DAC_OVERRIDE]
  privileged: false
  allowPrivilegeEscalation: false

[srfrnk]

Thanks for the workaround @cmdjulian. I'm checking that however I'm not sure these settings are secure enough. Running the proposed through snyk yields:

{
      "severity": "medium",
      "description": "",
      "resolve": "Set `securityContext.runAsNonRoot` to `true`",
      "id": "SNYK-CC-K8S-10",
      "impact": "Container could be running with full administrative privileges",
      "msg": "input.spec.template.spec.containers[kaniko-runner].securityContext.runAsNonRoot",
      "subType": "Deployment",
      "issue": "Container is running without root user control",
      "publicId": "SNYK-CC-K8S-10",
      "title": "Container is running without root user control",
      "references": [
        "CIS Docker Benchmark 1.2.0 - 5.5 Ensure sensitive host system directories are not mounted on containers",
        "https://kubernetes.io/docs/concepts/policy/pod-security-policy/#users-and-groups",
        "https://kubernetes.io/blog/2016/08/security-best-practices-kubernetes-deployment/"
      ],
      "isIgnored": false,
      "iacDescription": {
        "issue": "Container is running without root user control",
        "impact": "Container could be running with full administrative privileges",
        "resolve": "Set `securityContext.runAsNonRoot` to `true`"
      },
      "lineNumber": 18,
      "documentation": "https://snyk.io/security-rules/SNYK-CC-K8S-10",
      "isGeneratedByCustomRule": false,
      "path": [
        "[DocId: 0]",
        "input",
        "spec",
        "template",
        "spec",
        "containers[kaniko-runner]",
        "securityContext",
        "runAsNonRoot"
      ]
},
{
      "severity": "low",
      "description": "",
      "resolve": "Set `securityContext.readOnlyRootFilesystem` to `true`",
      "id": "SNYK-CC-K8S-8",
      "impact": "Compromised process could abuse writable root filesystem to elevate privileges",
      "msg": "input.spec.template.spec.containers[kaniko-runner].securityContext.readOnlyRootFilesystem",
      "subType": "Deployment",
      "issue": "`readOnlyRootFilesystem` attribute is not set to `true`",
      "publicId": "SNYK-CC-K8S-8",
      "title": "Container is running with writable root filesystem",
      "references": [
        "CIS Docker Benchmark 1.2.0 - Ensure that the container's root filesystem is mounted as read only",
        "https://kubernetes.io/docs/concepts/policy/pod-security-policy/#volumes-and-file-systems",
        "https://kubernetes.io/blog/2016/08/security-best-practices-kubernetes-deployment/"
      ],
      "isIgnored": false,
      "iacDescription": {
        "issue": "`readOnlyRootFilesystem` attribute is not set to `true`",
        "impact": "Compromised process could abuse writable root filesystem to elevate privileges",
        "resolve": "Set `securityContext.readOnlyRootFilesystem` to `true`"
      },
      "lineNumber": 20,
      "documentation": "https://snyk.io/security-rules/SNYK-CC-K8S-8",
      "isGeneratedByCustomRule": false,
      "path": [
        "[DocId: 0]",
        "input",
        "spec",
        "template",
        "spec",
        "containers[kaniko-runner]",
        "securityContext",
        "readOnlyRootFilesystem"
      ]
    }

Maybe just trying to add readOnlyRootFilesystem: true would suffice for now?

[06kellyjac]

You can try that but depending on how kaniko builds the container and outputs results it probably needs some writable mount, so you might need to mount an emptydir and set the working directory or change the build location via flags.

Also it wouldnt fix the medium runAsNonRoot

[cmdjulian]

@srfrnk readOnlyRootFilesystem is not very relevant for kaniko as kaniko is not a long running service. This only gets relevant if an attacker infects your container and persists some data in the containers root fs so that on each restart the malicious files are still present. As kaniko terminates after a very short amount of time, this is not a real risk in my opinion. If somebody has a different opinion let me know.
Also, you have to mount the /kaniko dir as rw. The problem here is, the kaniko executable lives in the /kaniko dir, by mounting an emptyDir at /kaniko the executable gets overridden and kaniko can't start anymore.
As previously mentioned, kaniko has to run as root because it is using chroot to build the image, I don't see a way how you could change that.
In my opinion kaniko is still way more secure then docker. Kaniko doesn't have to run privileged. I you really want to get rid of the root user you could also have a look at img, but it seems not maintained anymore.
In contrast to kaniko there is no root user running it, but you have to add seccomp and app armor = unconfined. In the end its a trade of. You could have a look over here, if you're interested in a detailed comparison from DockerCon2019 between the different approaches.

[srfrnk]

Thanks or the explanation @cmdjulian. I managed to work around the /kaniko folder issue by changing the writable part to something like /kaniko/data and mounting that from an emptyDir volume. The only problem was that you have to use the / (root) for the fs and you can't at the same time mount that as it contains the rest of the files which have to come from the image. Besides because you have to be root the read only filesystem will not really be the worst part anyway. I understand why kaniko would require running as root and then running with a r/w filesystem becomes not so much of an issue anymore with the reasons you gave also being part of that... I agree with your points about kaniko being the most secure option. Of all existing solutions I still think kaniko is the best for my use case. I'm thinking the main risk is that the resulting pod would still be a weak point in security as it will allow attackers access to the network and I will try to add security around that instead.

[cmdjulian]

Can you show me which exact configuration made the readOnlyRootFs work? If I understood you correctly you made that work.
At our place we also put some NetworkPolicies in place to prevent fetching data from unauthorized sources. I actually feel very safe with the other restrictions regarding the caps and the short lifespan of the kaniko jobs

[srfrnk]

See this commit: https://github.com/srfrnk/kaniko/commit/fff4d70a729b03a40cc2271534dd0a629e343a49 Seems like all the issues with write permissions are resolved when I run that... However it fails with:

Dec 16, 2021 @ 10:10:44.972 | INFO[0001] cmd: /bin/sh
Dec 16, 2021 @ 10:10:44.972 | INFO[0001] args: [-c apk add --update curl wget bash]
Dec 16, 2021 @ 10:10:44.972 | INFO[0001] Running: [/bin/sh -c apk add --update curl wget bash]
Dec 16, 2021 @ 10:10:44.973 | error building image: error building stage: failed to execute command: starting command: fork/exec /bin/sh: no such file or directory

/bin is now mounted into an emptyDir - which means it should have been populated with the filesystem from the base image (in this case alpine...) right?

[akash2237778]

See this commit: srfrnk@fff4d70 Seems like all the issues with write permissions are resolved when I run that... However it fails with:

Dec 16, 2021 @ 10:10:44.972 | �[36mINFO�[0m[0001] cmd: /bin/sh
Dec 16, 2021 @ 10:10:44.972 | �[36mINFO�[0m[0001] args: [-c apk add --update curl wget bash]
Dec 16, 2021 @ 10:10:44.972 | �[36mINFO�[0m[0001] Running: [/bin/sh -c apk add --update curl wget bash]
Dec 16, 2021 @ 10:10:44.973 | error building image: error building stage: failed to execute command: starting command: fork/exec /bin/sh: no such file or directory

/bin is now mounted into an emptyDir - which means it should have been populated with the filesystem from the base image (in this case alpine...) right?

Hey @srfrnk, are you able to run kaniko as a NonRootUser? If so, can you please provide the workaround?

[aaron-prindle]

We can't really fix the second part until something like user namespaces are supported in kubernetes, but we can allow builds to run with only the permissions they need until then.

It seems that user namespaces we're added in Kubernetes 1.25 in [alpha] state: https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/

Might be worth investigating how they can be used with kaniko as mentioned in the thread