Running with rootless podman seems not to respect property Delegate=yes

plluksie commented 2 years ago

What happened: I'm not able to create cluster.

What you expected to happen: I'm able to create cluster.

How to reproduce it (as minimally and precisely as possible):

❯ sudo cat /etc/systemd/system/user@.service.d/delegate.conf
[Service]
Delegate=yes

❯ KIND_EXPERIMENTAL_PROVIDER=podman kind -v 5 create cluster
using podman due to KIND_EXPERIMENTAL_PROVIDER
enabling experimental podman provider
ERROR: failed to create cluster: running kind with rootless provider requires setting systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/
Stack Trace: 
sigs.k8s.io/kind/pkg/errors.New
    sigs.k8s.io/kind/pkg/errors/errors.go:28
sigs.k8s.io/kind/pkg/cluster/internal/create.validateProvider
    sigs.k8s.io/kind/pkg/cluster/internal/create/create.go:253
sigs.k8s.io/kind/pkg/cluster/internal/create.Cluster
    sigs.k8s.io/kind/pkg/cluster/internal/create/create.go:70
sigs.k8s.io/kind/pkg/cluster.(*Provider).Create
    sigs.k8s.io/kind/pkg/cluster/provider.go:182
sigs.k8s.io/kind/pkg/cmd/kind/create/cluster.runE
    sigs.k8s.io/kind/pkg/cmd/kind/create/cluster/createcluster.go:80
sigs.k8s.io/kind/pkg/cmd/kind/create/cluster.NewCommand.func1
    sigs.k8s.io/kind/pkg/cmd/kind/create/cluster/createcluster.go:55
github.com/spf13/cobra.(*Command).execute
    github.com/spf13/cobra@v1.4.0/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
    github.com/spf13/cobra@v1.4.0/command.go:974
github.com/spf13/cobra.(*Command).Execute
    github.com/spf13/cobra@v1.4.0/command.go:902
sigs.k8s.io/kind/cmd/kind/app.Run
    sigs.k8s.io/kind/cmd/kind/app/main.go:53
sigs.k8s.io/kind/cmd/kind/app.Main
    sigs.k8s.io/kind/cmd/kind/app/main.go:35
main.main
    sigs.k8s.io/kind/main.go:25
runtime.main
    runtime/proc.go:250
runtime.goexit
    runtime/asm_amd64.s:1571

Following @cwrau recommendation expressed in https://github.com/kubernetes-sigs/kind/issues/2872#issuecomment-1210772774:

❯ KIND_EXPERIMENTAL_PROVIDER=podman systemd-run --user --property=Delegate=yes kind -v 5 create cluster
Running as unit: run-r28fa7c60a5144fdea8b349e1033c4370.service

❯ systemctl --user | grep run-r28fa7c60a5144fdea8b349e1033c4370.service
  ● run-r28fa7c60a5144fdea8b349e1033c4370.service     loaded failed     failed    /home/lsiemiradzki/.local/bin/kind -v 5 create cluster                            

❯ journalctl -u run-r28fa7c60a5144fdea8b349e1033c4370.service
-- Logs begin at Tue 2022-05-24 15:06:54 CEST, end at Thu 2022-09-08 10:03:16 CEST. --
-- No entries --

Anything else we need to know?: It was working fine 1 month ago. In the meantime there were some standard package upgrades and security updates. I cannot precisely state what exactly has changed. It is Ubuntu: ❯ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.5 LTS Release: 20.04 Codename: focal

Environment:

kind version: (use kind version): kind v0.14.0 go1.18.2 linux/amd64
Kubernetes version: (use kubectl version): Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.13", GitCommit:"a43c0904d0de10f92aa3956c74489c45e6453d6e", GitTreeState:"clean", BuildDate:"2022-08-18T02:28:16Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

Docker version: (use docker info): I don't have docker, I'm using podman: ❯ podman info

host:
arch: amd64
buildahVersion: 1.27.0-dev
cgroupControllers:
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon_100:2.1.2~0_amd64
path: /usr/libexec/podman/conmon
version: 'conmon version 2.1.2, commit: '
cpuUtilization:
idlePercent: 94.79
systemPercent: 1.57
userPercent: 3.64
cpus: 8
distribution:
codename: focal
distribution: ubuntu
version: "20.04"
eventLogger: journald
hostname: pollrnd1007
idMappings:
gidmap:
- container_id: 0
  host_id: 1000
  size: 1
- container_id: 1
  host_id: 100000
  size: 65536
uidmap:
- container_id: 0
  host_id: 1000
  size: 1
- container_id: 1
  host_id: 100000
  size: 65536
kernel: 5.14.0-1051-oem
linkmode: dynamic
logDriver: journald
memFree: 5156880384
memTotal: 16142012416
networkBackend: cni
ociRuntime:
name: crun
package: crun_100:1.2-2_amd64
path: /usr/bin/crun
version: |-
  crun version UNKNOWN
  commit: ea1fe3938eefa14eb707f1d22adff4db670645d6
  spec: 1.0.0
  +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
remoteSocket:
path: /run/user/1000/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns_100:1.1.8-4_amd64
version: |-
  slirp4netns version 1.1.8
  commit: unknown
  libslirp: 4.3.1-git
  SLIRP_CONFIG_VERSION_MAX: 3
  libseccomp: 2.4.3
swapFree: 24574423040
swapTotal: 24574423040
uptime: 0h 27m 25.00s
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
search:
- docker.io
- quay.io
store:
configFile: /home/lsiemiradzki/.config/containers/storage.conf
containerStore:
number: 5
paused: 0
running: 1
stopped: 4
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/lsiemiradzki/.local/share/containers/storage
graphRootAllocated: 475998666752
graphRootUsed: 73949675520
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "true"
Supports d_type: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 61
runRoot: /run/user/1000/containers
volumePath: /home/lsiemiradzki/.local/share/containers/storage/volumes
version:
APIVersion: 4.2.0-rc2
Built: 1659567994
BuiltTime: Thu Aug  4 01:06:34 2022
GitCommit: ""
GoVersion: devel go1.20-f28fa952b5 Wed Aug 3 19:52:27 2022 +0000
Os: linux
OsArch: linux/amd64
Version: 4.2.0-rc2

OS (e.g. from /etc/os-release): Ubuntu 20.04.5 LTS

plluksie commented 2 years ago

Seems to be an issue with podman. It works with

❯ podman -v 
podman version 3.4.2

I'll check with podman project.

deftdawg commented 1 year ago

@plluksie I'm having the same issue with podman 4.3.1, did you get it resolved without downgrading or is there an issue in podman for it?

cloudguruab commented 1 year ago

@deftdawg you ever figure it out? I'm running podman 4.3.1 with the same issue

BenTheElder commented 1 year ago

Perhaps the systemd-run note https://kind.sigs.k8s.io/docs/user/rootless/#creating-a-kind-cluster-with-rootless-podman ?

deftdawg commented 1 year ago

@deftdawg you ever figure it out? I'm running podman 4.3.1 with the same issue

No, gave up and switched to running docker-rootless

cloudguruab commented 1 year ago

Ah, thanks for the update @BenTheElder & @deftdawg

sword-jin commented 8 months ago

I encounter the same problem:

❯ sudo cat /etc/systemd/system/user@.service.d/delegate.conf
[Service]
Delegate=yes

> podman version

Client:       Podman Engine
Version:      5.0.0-dev
API Version:  5.0.0-dev
Go Version:   go1.21.1
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

❯ KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster --name test --config kind-config.yaml
using podman due to KIND_EXPERIMENTAL_PROVIDER
enabling experimental podman provider
ERROR: failed to create cluster: running kind with rootless provider requires setting systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/

Overmorrow1501 commented 8 months ago

I encounter the same problem:

❯ sudo cat /etc/systemd/system/user@.service.d/delegate.conf
[Service]
Delegate=yes

> podman version

Client:       Podman Engine
Version:      5.0.0-dev
API Version:  5.0.0-dev
Go Version:   go1.21.1
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

❯ KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster --name test --config kind-config.yaml
using podman due to KIND_EXPERIMENTAL_PROVIDER
enabling experimental podman provider
ERROR: failed to create cluster: running kind with rootless provider requires setting systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/

I got the same problem

sword-jin commented 8 months ago

I encounter the same problem:

❯ sudo cat /etc/systemd/system/user@.service.d/delegate.conf
[Service]
Delegate=yes

> podman version

Client:       Podman Engine
Version:      5.0.0-dev
API Version:  5.0.0-dev
Go Version:   go1.21.1
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

❯ KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster --name test --config kind-config.yaml
using podman due to KIND_EXPERIMENTAL_PROVIDER
enabling experimental podman provider
ERROR: failed to create cluster: running kind with rootless provider requires setting systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/

I got the same problem

update your crun version

jocutajar commented 5 months ago

The unit file seems to be ignored on my systemd as well (debian bookworm).

This -p "Delegate=yes" seems to have done it:

systemd-run --scope --user -p "Delegate=yes" kind create cluster

Found the trick in regard to LXC.

podman version 4.3.1
crun version 1.8.1
systemd 252 (252.22-1~deb12u1)
kind version 0.22.0

jsmrcka commented 4 months ago

The unit file seems to be ignored on my systemd as well (debian bookworm).

This -p "Delegate=yes" seems to have done it:
systemd-run --scope --user -p "Delegate=yes" kind create cluster
Found the trick in regard to LXC.
* podman version 4.3.1

* crun version 1.8.1

* systemd 252 (252.22-1~deb12u1)

* kind version 0.22.0

That did the trick for me too, thanks.

podman 4.9.3
crun 1.14.4
systemd 255 (255.4-1-manjaro)
kind 0.22.0

domeales-paloit commented 2 months ago

The unit file seems to be ignored on my systemd as well (debian bookworm).

This -p "Delegate=yes" seems to have done it:
systemd-run --scope --user -p "Delegate=yes" kind create cluster
Found the trick in regard to LXC.

podman version 4.3.1

crun version 1.8.1

systemd 252 (252.22-1~deb12u1)

kind version 0.22.0

Worked for me too, cheers!

podman 5.1.1
crun 1.15
systemd 253 (253.5-1ubuntu6.1)
kind 0.23.0
Ubuntu 23.10

BenTheElder commented 2 months ago

mind adding an update to https://kind.sigs.k8s.io/docs/user/rootless/#creating-a-kind-cluster-with-rootless-podman ?

jstaf commented 2 months ago

None of these suggestions seem to work on OpenSUSE Tumbleweed. To reproduce:

Install Tumbleweed
Install either nerdctl or podman in rootless mode
Follow all the rootless setup steps here: https://kind.sigs.k8s.io/docs/user/rootless/
kind create cluster with either podman or nerdctl fails with:

ERROR: failed to create cluster: running kind with rootless provider requires setting systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/

khanhtc1202 commented 3 weeks ago

@BenTheElder I submitted a patch for the docs :pray:

kubernetes-sigs / kind

Running with rootless podman seems not to respect property Delegate=yes #2916