Is it safe to upgrade to 1.11 if I want to use podman 3.3.0+

[leeaash]

As unifi just released 1.11.0 official firmware.

I just want to confirm that if it is safe to upgrade to 1.11 if I still want to use updated podman or I have to stay at 1.10?

[jonwilliams84]

No. I found out the hard way at 6am to my boy telling me he had no internet.

On inspection, the pihole container would not start, or in fact any containers:

WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: open /sys/fs/cgroup/libpod_parent/conmon/cpuset.cpus: open /sys/fs/cgroup/libpod_parent/conmon/cpuset.cpus.effective: no such file or directory
Error: OCI runtime error: unable to start container "65ee4bb475fb5c0313b7c0b5b80bbe8c1055c59f5a9c9980bae932515fff8aec": container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented

I have had to drop back to the stock podman.

[leeaash]

Sad news ...

So we would say goodbye to future updated firmware?

[jonwilliams84]

I wouldn't have thought so, I expect it just needs an update to the install scripts for podman.

I run the custom kernel and this was updated about 15 minutes after I logged an issue with 1.11.0 this morning. So for me, bar podman version everything is back to normal on my network.

To be honest, I probably wont update podman again unless I absolutely need to; I have realised I don't really need it. I only run frr and pihole additional to the unifi-os pod.

[justfly1111]

@jonwilliams84 you got custom kernel to run on 1.11? i tried getting custom kernel going on the beta firmware and get a kernel mismatch

[jonwilliams84]

Did you update the udm-kernel-tools to the latest version?

On Fri, 24 Dec 2021, 03:33 Justfly1111, @.***> wrote:

@jonwilliams84 https://github.com/jonwilliams84 you got custom kernel to run on 1.11? i tried getting custom kernel going on the beta firmware and get a kernel mismatch

— Reply to this email directly, view it on GitHub https://github.com/boostchicken/udm-utilities/issues/300#issuecomment-1000632862, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF2YHJLU5WOOE7Y5B7VVELDUSPS2FANCNFSM5KR7CZKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[ArmandH]

Just to confirm before i upgrade, i am not using a special podman just the standard with Adguardhome will that still work when upgrading to 1.11?

[justfly1111]

jon i had a beta version of 1.11.23 i didnt do the official upgrade to the full 1.11 cause i didnt see the release as of time of post buti got edge3 running perfectly now still no updated podman tho too afraid after what happened the last time to attemt to upgrade it again lmao

[chriscpritchard]

jon i had a beta version of 1.11.23 i didnt do the official upgrade to the full 1.11 cause i didnt see the release as of time of post buti got edge3 running perfectly now still no updated podman tho too afraid after what happened the last time to attemt to upgrade it again lmao

Is that on a UDM or UDM-P?

[mikesrus]

jon i had a beta version of 1.11.23 i didnt do the official upgrade to the full 1.11 cause i didnt see the release as of time of post buti got edge3 running perfectly now still no updated podman tho too afraid after what happened the last time to attemt to upgrade it again lmao

how to back to stock podman? i have same issue

# unifi-os shell
WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: open /sys/fs/cgroup/libpod_parent/conmon/cpuset.cpus: open /sys/fs/cgroup/libpod_parent/conmon/cpuset.cpus.effective: no such file or directory 
Error: OCI runtime error: panic: expected "name=systemd" path to be unified path "/sys/fs/cgroup/devices/libpod_parent/libpod-16b9aff8b48bbf9c9bef8b5427ed8231f191921331b722793359a5d38300dd72", got "/sys/fs/cgroup/systemd/libpod_parent/libpod-16b9aff8b48bbf9c9bef8b5427ed8231f191921331b722793359a5d38300dd72"

[jonwilliams84]

jon i had a beta version of 1.11.23 i didnt do the official upgrade to the full 1.11 cause i didnt see the release as of time of post buti got edge3 running perfectly now still no updated podman tho too afraid after what happened the last time to attemt to upgrade it again lmao

how to back to stock podman? i have same issue

# unifi-os shell
WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: open /sys/fs/cgroup/libpod_parent/conmon/cpuset.cpus: open /sys/fs/cgroup/libpod_parent/conmon/cpuset.cpus.effective: no such file or directory 
Error: OCI runtime error: panic: expected "name=systemd" path to be unified path "/sys/fs/cgroup/devices/libpod_parent/libpod-16b9aff8b48bbf9c9bef8b5427ed8231f191921331b722793359a5d38300dd72", got "/sys/fs/cgroup/systemd/libpod_parent/libpod-16b9aff8b48bbf9c9bef8b5427ed8231f191921331b722793359a5d38300dd72"

Remove the podman update script from the onboot directory and reboot.

[boostchicken]

So I am working on a theory which is kinda wild. BUt if you ran podman inside of a podman container........

[dvcrn]

Out of curiosity - what podman features are you using that needs 3.3.0?

I upgraded to 1.11 and podman 3 was no longer working, but all my basic containers like adguard and what not work perfectly on the default podman, so I just left it at that

[dantrainor]

So I am working on a theory which is kinda wild. BUt if you ran podman inside of a podman container........

Just to follow up, I did give this a shot and got pretty far, but it only moved the cgroups problem to a different place:

DEBU[0000] ExitCode msg: "time=\"2022-05-11t18:07:14z\" level=error msg=\"runc create failed: unable to start container process: error during container init: error setting cgroup config for prochooks process: bpf_prog_query(bpf_cgroup_device) failed: function not implemented\": oci runtime error" Error: OCI runtime error: time="2022-05-11T18:07:14Z" level=error msg="runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented"

While I was doing this I was reminded that I even went so far as to use fabianishere/udm-kernel to build a new kernel with BPF but that didn't work for, reasons. I don't remember the reasons so I'll revisit this. It might have been related to not having an arm64 around to compile on, and cross-build wasn't working for me.

I think you're right in that UBNT needs to get down with cgroupsv2 for all of the things, and we can put this behind us.

Just for fun I also tried with crun-1.4.5-linux-arm64 and this time it failed different:

DEBU[0000] ExitCode msg: "error opening file/proc/self/setgroups: no such file or directory: oci runtime attempted to invoke a command that was not found" Error: error opening file/proc/self/setgroups: No such file or directory: OCI runtime attempted to invoke a command that was not found

I think its related to the container not having CAP_SETUID because the parent container (unifi-os) not having it. The unifi-os container doesn't have it probably for the same reasons cgroups[v1|v2] isn't available to us.

In the interim, I made a systemd container which also contained an sshd, and from this container I treated it as a service container. From there, they were managed as an Ansible hots using a different SSH host port per container depending on the container. Not ideal, zero orchestration, it still hurts, but it almost does this properly.

p.s.: also ironic that the podman 1.6.1 release notes mentioned "Fixed a bug where rootless Podman on systems using CGroups V2 would not function with the cgroupfs CGroups manager"... bad timing? hah.

[dantrainor]

Got bored, did a thing:

# podman exec -it unifi-os podman run --rm hello-world
ERRO[0000] Failed to built-in GetDriver graph btrfs /var/lib/containers/storage 

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (arm64v8)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

# podman exec -it unifi-os podman version
ERRO[0000] Failed to built-in GetDriver graph btrfs /var/lib/containers/storage 
Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.16.15
Git Commit:   f6526ada1025c2e3f88745ba83b8b461ca659933-dirty
Built:        Mon Mar 21 07:59:31 2022
OS/Arch:      linux/arm64

I used fabianishere/udm-kernel-tools to enable a few features I needed:

root@debian:~/udm-kernel# cat .github/config/config.local.udm 
CONFIG_FUSE_FS=y
CONFIG_TEST_BPF=y
CONFIG_BPF=y
CONFIG_BPFILTER=y
CONFIG_BPF_SYSCALL=y
CONFIG_CGROUPS=y
CONFIG_CGROUP_BPF=y

Not sure where to go with this because in order to do this, a whole slew of prerequisites need to be satisfied to do what once was a very simple task with some helper scripts.

I'll continue to beat this up and see how well it ends up working out. I'll probably make a PR for some of this. If you want any more info/help, please reach out.

[gatesry]

@dantrainor - Thanks for the write-up. This is exciting to see! Any chance you could share the kernel image and/or preliminary steps to accomplish this?