search

acorn-io / runtime

A simple application deployment framework built on Kubernetes
https://docs.acorn.io/
Apache License 2.0
1.13k stars 100 forks source link

Run buildkitd pod in rootless mode #599

Open tekumara opened 2 years ago

tekumara commented 2 years ago

My corporate kubes cluster prevents pods running with securityContext privileged=true. So unfortunately acorn's buildkitd pod fails to start.

Also unfortunately I'm not a cluster admin, so can't easily change our cluster's settings. I first created a loft vcluster, to have my own kube control plane, and then installed acorn.

Are there good reasons for running buildkitd as a privileged pod? Or would it be possible to run buildkitd in rootless mode?

ibuildthecloud commented 2 years ago

Buildkit rootless seems to need unconfined seccomp and apparmor which very well might be reject too. For rootless setups probably the best approach would be to support another builder like kaniko that runs rootless better.

ibuildthecloud commented 2 years ago

Another thought is to support client side building. In your environment do you have docker on your laptop/desktop?

tekumara commented 2 years ago

Buildkit rootless seems to work in our cluster. I have docker locally on my mac m1 (arm64). Although I do like building in the cluster, because the cluster is amd64 and has a faster network to our package repositories.

ibuildthecloud commented 2 years ago

It's logical for us to switch to rootless. If your cluster is going to reject privileged, if it rejects rootless your no worse off than before.

cjellick commented 2 years ago

@ibuildthecloud - i wonder if @iwilltry42 should tackle this while he's refactoring buildkit to be exposed as a service LB. Tentatively putting it in v0.4 and assigning to him under that premise