[BUG] k3d v5.5.1 still not works with Podman rootless

tbshrst commented 1 year ago

Still experiencing the same issues with rootless podman as described in Issue 1052. According to one answer the problem was fixed in k3d v5.4.2.

What did you do

How was the cluster created?

export DOCKER_SOCK=/var/run/docker.sock
export DOCKER_HOST=unix:///var/run/docker.sock
k3d cluster create test

What did you expect to happen

The cluster should be created.

Screenshots or terminal output

INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-test'                   
INFO[0000] Created image volume k3d-test-images         
INFO[0000] Starting new tools node...                   
ERRO[0000] Failed to run tools container for cluster 'test' 
INFO[0001] Creating node 'k3d-test-server-0'            
INFO[0001] Creating LoadBalancer 'k3d-test-serverlb'    
INFO[0001] Using the k3d-tools node to gather environment information 
INFO[0001] Starting new tools node...                   
ERRO[0001] Failed to run tools container for cluster 'test' 
ERRO[0001] failed to gather environment information used for cluster creation: failed to run k3d-tools node for cluster 'test': failed to create node 'k3d-test-tools': runtime failed to create node 'k3d-test-tools': failed to create container for node 'k3d-test-tools': docker failed to create container 'k3d-test-tools': Error response from daemon: make cli opts(): making volume mountpoint for volume /var/run/docker.sock: mkdir /var/run/docker.sock: permission denied 
ERRO[0001] Failed to create cluster >>> Rolling Back    
INFO[0001] Deleting cluster 'test'                      
INFO[0002] Deleting cluster network 'k3d-test'          
INFO[0002] Deleting 1 attached volumes...               
FATA[0002] Cluster creation FAILED, all changes have been rolled back!

Which OS & Architecture

k3d runtime-info
arch: amd64
cgroupdriver: systemd
cgroupversion: "2"
endpoint: /var/run/docker.sock
filesystem: extfs
infoname: hostname
name: docker
os: fedora
ostype: linux
version: 4.5.0

Which version of `k3d`

k3d version
k3d version v5.5.1
k3s version v1.26.4-k3s1 (default)

Which version of docker podman

podman version
Client:       Podman Engine
Version:      4.5.0
API Version:  4.5.0
Go Version:   go1.19.7
OS/Arch:      linux/amd64

Gianluca755 commented 1 year ago

Stupid question, did you use the official instructions? https://k3d.io/v5.5.2/usage/advanced/podman/?h=podman#using-rootless-podman

iwilltry42 commented 1 year ago

Hey @tbshrst ! The error there is making volume mountpoint for volume /var/run/docker.sock: mkdir /var/run/docker.sock: permission denied, which means that basically docker run -v /var/run/docker.sock:/var/run/docker.sock doesn't work which is required to start the k3d-tools container to setup the environment for k3d. That's nothing that k3d can do itself if it doesn't have permissions.

tcassaert commented 1 year ago

I don't think you can say that rootless Podman works, if you still rely on a docker run -v /var/run/docker.sock:/var/run/docker.sock, so to me this does look like something k3d should solve.

Can't this docker run... be replaced by a podman run... which mounts the podman socket?

EDIT I just saw that OP didn't really follow the guide for rootless podman. I did follow the steps in the docs, but I see the same result.

$ systemctl --user enable --now podman.socket
$ systemctl --user status podman.socket                                                                                     1 ↵
● podman.socket - Podman API Socket
     Loaded: loaded (/usr/lib/systemd/user/podman.socket; enabled; preset: enabled)
     Active: active (listening) since Mon 2023-09-25 08:49:22 CEST; 2 days ago
   Triggers: ● podman.service
       Docs: man:podman-system-service(1)
     Listen: /run/user/1000/podman/podman.sock (Stream)
     CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/podman.socket

Sep 25 08:49:22 workbox systemd[1360]: Listening on Podman API Socket.

$ XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR:-/run/user/$(id -u)}
$ export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock
$ echo $DOCKER_HOST
unix:///run/user/1000/podman/podman.sock
$ k3d cluster create test
WARN[0001] Failed to get random free port: failed to create tcp listener: listen tcp 192.168.130.12:0: bind: cannot assign requested address
WARN[0001] Falling back to internal port 6443 (may be blocked though)...
INFO[0001] Prep: Network
INFO[0001] Created network 'k3d-test'
INFO[0001] Created image volume k3d-test-images
INFO[0001] Starting new tools node...
INFO[0002] Creating node 'k3d-test-server-0'
ERRO[0002] Failed to run tools container for cluster 'test'
INFO[0002] Creating LoadBalancer 'k3d-test-serverlb'
INFO[0003] Using the k3d-tools node to gather environment information
INFO[0003] Starting new tools node...
ERRO[0004] Failed to run tools container for cluster 'test'
ERRO[0004] failed to gather environment information used for cluster creation: failed to run k3d-tools node for cluster 'test': failed to create node 'k3d-test-tools': runtime failed to create node 'k3d-test-tools': failed to create container for node 'k3d-test-tools': docker failed to create container 'k3d-test-tools': Error response from daemon: make cli opts(): making volume mountpoint for volume /var/run/docker.sock: mkdir /var/run/docker.sock: permission denied
ERRO[0004] Failed to create cluster >>> Rolling Back
INFO[0004] Deleting cluster 'test'
INFO[0004] Deleting cluster network 'k3d-test'
INFO[0004] Deleting 1 attached volumes...
FATA[0004] Cluster creation FAILED, all changes have been rolled back!

tcassaert commented 1 year ago

When doing a sudo ln -s $XDG_RUNTIME_DIR/podman/podman.sock /var/run/docker.sock, it gets a step further. I guess this step should be added to the docs for rootless Podman?

But then it errors on

ERRO[0005] failed to gather environment information used for cluster creation: error starting existing tools node k3d-test-tools: docker failed to start container for node 'k3d-test-tools': Error response from daemon: crun: write to `/proc/self/oom_score_adj`: Permission denied: OCI permission denied

Which I tracked down to a problem with crun, that can be solved with https://github.com/containers/podman/issues/19930#issuecomment-1715401284.

tcassaert commented 11 months ago

I gave it another shot today and got it to work. The most important part that I was missing and that seems to be missing in the docs is that we need to inform the Kubelet to run in userspace:

---
apiVersion: k3d.io/v1alpha5
kind: Simple
options:
  k3s:
    extraArgs:
      - arg: "--kubelet-arg=feature-gates=KubeletInUserNamespace=true"
        nodeFilters:
          - server:*
          - agent:*

k3d-io / k3d