Can't do rootless workspaces with `coder` and `envbuilder` in k8s cluster

[shiipou]

I want to create a rootless workspace with coder and envbuilder in my homelab kubernetes cluster, but I have this error where I can't figure how to pass it.

error: compile devcontainer.json: create features directory: mkdir /.envbuilder/features: permission denied

It seems it doesn't find the .devcontainer folder where the devcontainer.json file is.

It work fine if I didn't set the user (So it will be root)

Full logs

```bash envbuilder - Build development environments from repositories in a container #1: 📦 Cloning https://github.com/shiipou/simplon-2024-brief-06-TooManyChoco to /workspace... #1: Enumerating objects: 27, done. #1: Counting objects: 3% (1/27) #1: Counting objects: 7% (2/27) #1: Counting objects: 11% (3/27) #1: Counting objects: 14% (4/27) #1: Counting objects: 18% (5/27) #1: Counting objects: 22% (6/27) #1: Counting objects: 25% (7/27) #1: Counting objects: 29% (8/27) #1: Counting objects: 33% (9/27) #1: Counting objects: 37% (10/27) #1: Counting objects: 40% (11/27) #1: Counting objects: 44% (12/27) #1: Counting objects: 48% (13/27) #1: Counting objects: 51% (14/27) #1: Counting objects: 55% (15/27) #1: Counting objects: 59% (16/27) #1: Counting objects: 62% (17/27) #1: Counting objects: 66% (18/27) #1: Counting objects: 70% (19/27) #1: Counting objects: 74% (20/27) #1: Counting objects: 77% (21/27) #1: Counting objects: 81% (22/27) #1: Counting objects: 85% (23/27) #1: Counting objects: 88% (24/27) #1: Counting objects: 92% (25/27) #1: Counting objects: 96% (26/27) #1: Counting objects: 100% (27/27) #1: Counting objects: 100% (27/27), done. #1: Compressing objects: 5% (1/19) #1: Compressing objects: 10% (2/19) #1: Compressing objects: 15% (3/19) #1: Compressing objects: 21% (4/19) #1: Compressing objects: 26% (5/19) #1: Compressing objects: 31% (6/19) #1: Compressing objects: 36% (7/19) #1: Compressing objects: 42% (8/19) #1: Compressing objects: 47% (9/19) #1: Compressing objects: 52% (10/19) #1: Compressing objects: 57% (11/19) #1: Compressing objects: 63% (12/19) #1: Compressing objects: 68% (13/19) #1: Compressing objects: 73% (14/19) #1: Compressing objects: 78% (15/19) #1: Compressing objects: 84% (16/19) #1: Compressing objects: 89% (17/19) #1: Compressing objects: 94% (18/19) #1: Compressing objects: 100% (19/19) #1: Total 27 (delta 8), reused 16 (delta 4), pack-reused 0 #1: 📦 Cloned repository! [459.376322ms] error: compile devcontainer.json: create features directory: mkdir /.envbuilder/features: permission denied envbuilder - Build development environments from repositories in a container #1: 📦 Cloning https://github.com/shiipou/simplon-2024-brief-06-TooManyChoco to /workspace... #1: 📦 The repository already exists! [104.122µs] error: compile devcontainer.json: create features directory: mkdir /.envbuilder/features: permission denied envbuilder - Build development environments from repositories in a container ```

Here the rootless template I wrote :

Details

```terraform terraform { required_providers { coder = { source = "coder/coder" } kubernetes = { source = "hashicorp/kubernetes" } } } provider "coder" { } variable "use_kubeconfig" { type = bool description = <<-EOF Use host kubeconfig? (true/false) Set this to false if the Coder host is itself running as a Pod on the same Kubernetes cluster as you are deploying workspaces to. Set this to true if the Coder host is running outside the Kubernetes cluster for workspaces. A valid "~/.kube/config" must be present on the Coder host. EOF default = false } variable "namespace" { type = string description = "The Kubernetes namespace to create workspaces in (must exist prior to creating workspaces). If the Coder host is itself running as a Pod on the same Kubernetes cluster as you are deploying workspaces to, set this to the same namespace." } data "coder_parameter" "repo" { name = "repo" display_name = "Repository (auto)" order = 1 description = "Select a repository to automatically clone and start working with a devcontainer." mutable = true option { name = "vercel/next.js" description = "The React Framework" value = "https://github.com/vercel/next.js" } option { name = "Dev-Container's Images" description = "To use it, choose the children directory with `devcontainer_dir` to `src/java/.devcontainer` or to the image you want." value = "https://github.com/devcontainers/images/" } option { name = "Custom" icon = "/emojis/1f5c3.png" description = "Specify a custom repo URL below" value = "custom" } } data "coder_parameter" "custom_repo_url" { name = "custom_repo" display_name = "Repository URL (custom)" order = 2 default = "" description = "Optionally enter a custom repository URL, see [awesome-devcontainers](https://github.com/manekinekko/awesome-devcontainers)." mutable = true } data "coder_parameter" "devcontainer_dir" { name = "devcontainer_dir" display_name = "Dev Container directory" order = 3 default = ".devcontainer" description = "Optionnally enter a custom path for the .devcontainer directory. Useful if the template is in a subfolder of the repository (e.g: https://github.com/devcontainers/images/ has devcontainer_dir to `src/java/.devcontainer`)" mutable = true } data "coder_parameter" "cpu" { name = "cpu" display_name = "CPU" order = 4 description = "The number of CPU cores" default = "2" icon = "/icon/memory.svg" mutable = true option { name = "2 Cores" value = "2" } option { name = "4 Cores" value = "4" } option { name = "6 Cores" value = "6" } option { name = "8 Cores" value = "8" } } data "coder_parameter" "memory" { name = "memory" display_name = "Memory" order = 5 description = "The amount of memory in GB" default = "2" icon = "/icon/memory.svg" mutable = true option { name = "2 GB" value = "2" } option { name = "4 GB" value = "4" } option { name = "6 GB" value = "6" } option { name = "8 GB" value = "8" } } data "coder_parameter" "home_disk_size" { name = "home_disk_size" display_name = "Home disk size" order = 6 description = "The size of the home disk in GB" default = "10" type = "number" icon = "/emojis/1f4be.png" mutable = false validation { min = 1 max = 99999 } } provider "kubernetes" { # Authenticate via ~/.kube/config or a Coder-specific ServiceAccount, depending on admin preferences config_path = var.use_kubeconfig == true ? "~/.kube/config" : null } data "coder_provisioner" "me" {} data "coder_workspace" "me" {} resource "coder_agent" "main" { os = "linux" arch = data.coder_provisioner.me.arch startup_script = <<-EOT set -e # install and start code-server curl -fsSL https://code-server.dev/install.sh | sh -s -- --method=standalone --prefix=/tmp/code-server --version 4.11.0 /tmp/code-server/bin/code-server --auth none --port 13337 >/tmp/code-server.log 2>&1 & EOT dir = "/workspace" # The following metadata blocks are optional. They are used to display # information about your workspace in the dashboard. You can remove them # if you don't want to display any information. # For basic resources, you can use the `coder stat` command. # If you need more control, you can write your own script. metadata { display_name = "CPU Usage" key = "0_cpu_usage" script = "coder stat cpu" interval = 10 timeout = 1 } metadata { display_name = "RAM Usage" key = "1_ram_usage" script = "coder stat mem" interval = 10 timeout = 1 } metadata { display_name = "Home Disk" key = "3_home_disk" script = "coder stat disk --path $${HOME}" interval = 60 timeout = 1 } metadata { display_name = "CPU Usage (Host)" key = "4_cpu_usage_host" script = "coder stat cpu --host" interval = 10 timeout = 1 } metadata { display_name = "Memory Usage (Host)" key = "5_mem_usage_host" script = "coder stat mem --host" interval = 10 timeout = 1 } metadata { display_name = "Load Average (Host)" key = "6_load_host" # get load avg scaled by number of cores script = <

[dal13002]

@shiipou I see this issue is marked "completed". Wondering if there was a fix implemented for this / we can expect to run as nonroot in newer versions of envbuilder?

[shiipou]

@shiipou I see this issue is marked "completed". Wondering if there was a fix implemented for this / we can expect to run as nonroot in newer versions of envbuilder?

Oups I think I missclicked. That not supposed to be closed.

[johnstcn]

Envbuilder currently uses kaniko to build workspace images, and kaniko requires root privileges inside the container. You may be able to remap the root user to a non-root user on the host, but building rootless inside the container won't be possible in the near future.

[shiipou]

Envbuilder currently uses kaniko to build workspace images, and kaniko requires root privileges inside the container. You may be able to remap the root user to a non-root user on the host, but building rootless inside the container won't be possible in the near future. @johnstcn

Ok, thanks for your answer. That work well in root for now.