Failed to start ContainerManager open /proc/sys/kernel/panic: permission denied

[mar1ged]

Version:

k3s version v1.0.1 (e94a3c60)

Describe the bug

Failed to start ContainerManager open /proc/sys/kernel/panic: permission denied

To Reproduce

install k3s using the get shell script (without any special parameters) run kubectl get nodes

Expected behavior

I would like to see at least the current host

Actual behavior

Instead The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port? gets displayed

Additional context

I checked the log by running journalctl -u k3s. There I can see this at the end:

Dez 21 22:36:49 h12345678.stratoserver.net k3s[2328]: F1221 22:36:49.814798    2328 kubelet.go:1380] Failed to start ContainerManager open /proc/sys/kernel/panic: permission denied

As you can see I run this on a Strato VPS which uses Virtuozzo. In former times Strato didn't support docker on their virtualization platform but starting from November 2019 they do: https://docs.virtuozzo.com/virtuozzo_7_users_guide/advanced-tasks/setting-up-docker-in-containers.html I checked the server, it is able to run docker (in this case 18.09.7) without issues. The server uses Ubuntu 18.04.3.

This is the output of k3s check-config:

 sha256sum: good
- links: good

System:
- /sbin iptables v1.6.1: older than v1.8
- swap: disabled
- routes: ok

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

modprobe: module configs not found in modules.dep
error: cannot find kernel config 
  try running this script again, specifying the kernel config:
  set CONFIG=/path/to/kernel/.config or add argument /path/to/kernel/.config

[DavidZisky]

A possible solution is to run:

sudo apt-get install linux-image-$(uname -r)

but please also provide output of:

uname -r and ls /boot/config*

[mar1ged]

uname -r returns 4.15.0, when I install for that version I get an overall download of 26GB ;-)

/boot has no config file, ls -al /boot returns this:

drwxr-xr-x  2 root root 4096 Sep 27 20:24 .
drwxr-xr-x 23 root root 4096 Dez 21 22:11 ..

[DavidZisky]

oh boy, it looks like it's nothing k3s can fix, simply because it's due to the way how Strato works. I signed up for their VPS to quickly check but it's been an hour already and it's still not provisioned. Why would you use such a service? Can't you just use normal VPS provider? :p

[mar1ged]

I have others like Hetzner, DO, scaleway, OVH vh and vultr but I get 4 vCores , 100GB SSD and 8GB RAM for 5€, that's simply a bargain ;-) On Strato I run the stuff I always need and I have Plesk + all stuff I need for my email handling. My goal was to add some containers, but wanted to do it like on all the other providers.

[DavidZisky]

Well, ok. You just need to keep k3s somewhere else then. I'm not entirely sure how Virtuozzo works but it's not "real" VM somehow. It restricts a lot of stuff, and k3s simply can't access kernel modules which it needs. When I finally got my access I wasn't even able to see exact kernel version - virtuozzo abstracts it. That's why when trying to install kernel it tries to install all possible versions of it (therefore 26GB)

[mar1ged]

OK, I expected that this might be somehow troublesome and hoped that because Docker works now similar technologies might work too. What I don't understand: why should I install a kernel ? There is already one available, why is this one not enough ?

[DavidZisky]

Docker requires less privileges/kernel modules than K8S/K3S.

why should I install a kernel?

I recommended it because this issue:

modprobe: module configs not found in modules.dep

often happens where there is a mismatch between installed kernel and installed headers. Then simply installing a kernel again usually fixes it. But I installed different version on that Virtuozzo VPS and I am not even able to boot it up.

[mar1ged]

Yes, I see. I ran the same statement on a regular Ubuntu. It complains about the same missing module but it finds a config file and dumps a lot of stuff.

I tried to follow advice on superuser to find more about kernel config. But there is no /lib/modules/4.15.0/build/ and thus no .config. I even checked the directory where build should symlink too, but there is no /usr/src/linux-headers-5.3.0-24-generic. To be precise there is nothing below usr/src at all.

[mar1ged]

One more note: I played some more with the VPS and installed wireguard. Wireguard somehow detected a version: Building for 4.15.0 and 4.15.0-72-generic.

[jewelt]

Could you solve your problem with Strato and Kubernetes in the meantime? We have exactly the same problem. Do you know anything new?

[mar1ged]

No, I think that we can be thankful that docker works with Strato. I'm still searching for a priceworthy solution and I think that I will head for Scaleway Kapsule

[kennedyalexander]

Thanks for doing the investigation. This helped me. Only solution... New hosting company... Appreciate the work you have done.

[vioan]

is what I did, as well, switching from Strato to Hetzner and I am a lot happier.

[stale[bot]]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[unixfox]

Still happen.

[brandond]

This is not something that can be fixed in K3s; it appears to be an issue with the VPS provider's restricted environment.