Closed gtrabanco closed 2 years ago
Sorry.
Should I close it?
They did some crazy stuff on the latest firmware update. I am working to get around it. I have tried all different versions and mixtures of conmon, runc, crun, and podman and they all end up causing a fail somehow. I'll let you know if I figure out a fix.
It's curious why the systemd stuff is added in there as there is no systemd anywhere,, but on the device some systemd stuff is showing up. I can only imagine its stuff leaking in from the UDMP-SE which has systemd
I have a fix for this coming soon
update on the fix, i gotta couple different configs working and learned about a new config file i have to write. more testing to figure out a valid configuration. The main issue is they removed a syscall it looks like.
I just installed the latest podman script, and 3.3.0 seems to be working fine on the latest UDMP software.
However, every time I run the podman
command, I get the following warning:
WARN[0000] Failed to decode the keys ["storage.options.ostree_repo"] from "/etc/containers/storage.conf".
Is anyone else seeing this behavior?
I just installed the latest podman script, and 3.3.0 seems to be working fine on the latest UDMP software.
However, every time I run the
podman
command, I get the following warning:WARN[0000] Failed to decode the keys ["storage.options.ostree_repo"] from "/etc/containers/storage.conf".
Is anyone else seeing this behavior?
I did. I comment out ostree_repo = “”
in /etc/containers/storage.conf, then the warning gone. But I don’t know the comment out if have any other impact.
Got it. Does that file persist across reboots & upgrades? Or do I need to comment out via a on_boot.d script?
Got it. Does that file persist across reboots & upgrades? Or do I need to comment out via a on_boot.d script?
You can add a line sed -i 's/ostree_repo = ""/#ostree_repo = ""/' /etc/containers/storage.conf
in 01-podman-update.sh
I added that line to the script. Is everyone comfortable wit hthe podman upgrade working now? If so I will close the issue
Sorry but same issue when executing unifi-os shell
All the steps and output:
# cd /mnt/data/
# curl -O https://raw.githubusercontent.com/boostchicken/udm-utilities/master/podman-update/01-podman-update.sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1701 100 1701 0 0 6300 0 --:--:-- --:--:-- --:--:-- 6300
# ls
01-podman-update.sh crash-data lost+found ssh udapi-config ulcmd
adguard dpi-tracer on_boot.d system udm-le unifi-os
backupftp log podman traces ui-db wireguard
# chmod u+x 01-podman-update.sh
# ./01-podman-update.sh
# unifi-os shell
WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: open /sys/fs/cgroup/libpod_parent/conmon/cpuset.cpus: open /sys/fs/cgroup/libpod_parent/conmon/cpuset.cpus.effective: no such file or directory
Error: OCI runtime error: panic: expected "name=systemd" path to be unified path "/sys/fs/cgroup/devices/libpod_parent/libpod-3b8e789d9346cb38ed1b8ddf2c003c4f9eda19524acc8896093eec4561ed4359", got "/sys/fs/cgroup/systemd/libpod_parent/libpod-3b8e789d9346cb38ed1b8ddf2c003c4f9eda19524acc8896093eec4561ed4359"
goroutine 1 [running]:
github.com/opencontainers/runc/libcontainer.getUnifiedPath(0x400029ae10, 0x55869b5400, 0x0)
github.com/opencontainers/runc/libcontainer/factory_linux.go:59 +0x2cc
github.com/opencontainers/runc/libcontainer.cgroupfs2.func1(0x4000231730, 0x400029ae10, 0x4a, 0x7ffe24eedf)
github.com/opencontainers/runc/libcontainer/factory_linux.go:111 +0x30
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).Load(0x400023a090, 0x7ffe24eedf, 0x40, 0x0, 0x0, 0x1, 0x8)
github.com/opencontainers/runc/libcontainer/factory_linux.go:321 +0x14c
main.getContainer(0x400025a160, 0x55869c5e78, 0x8, 0x1, 0x40002233b8)
github.com/opencontainers/runc/utils_linux.go:89 +0x9c
main.execProcess(0x400025a160, 0x0, 0x0, 0x0)
github.com/opencontainers/runc/exec.go:114 +0x30
main.glob..func5(0x400025a160, 0x5586de93c0, 0x40002234f8)
github.com/opencontainers/runc/exec.go:104 +0x6c
github.com/urfave/cli.HandleAction(0x5586ab6d60, 0x5586b63bc8, 0x400025a160, 0x400025a160, 0x0)
github.com/urfave/cli@v1.22.1/app.go:523 +0x124
github.com/urfave/cli.Command.Run(0x55869c1f3e, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x55869d9b5a, 0x28, 0x0, ...)
github.com/urfave/cli@v1.22.1/command.go:174 +0x408
github.com/urfave/cli.(*App).Run(0x4000252000, 0x400013c000, 0xe, 0xe, 0x0, 0x0)
github.com/urfave/cli@v1.22.1/app.go:276 +0x588
main.main()
github.com/opencontainers/runc/main.go:163 +0xa7
The latest version has resolved the ostree_repo
message for me. No issues.
im having this issue where is the fix?
Have the same issue on 1.11.0-16 Running on the old 1.61 podman works fine but is not really wanted. Updating to new 3.3.0 podman causes a lot of issues.
Okay, just upgrade to 1.11.0-16 - and since I am new to the UDMP universe (coming from a USG4P), I didn't see this thread until after updating 😉
Anyway, podman 1.6.1 is still on the system - called /usr/bin/podman.old - but running that doesn't work either:
# /usr/bin/podman.old exec -ti unifi-os bash
Error: panic: expected "name=systemd" path to be unified path "/sys/fs/cgroup/devices/libpod_parent/libpod-6bfb827ebe2ecd17495201196e816f935cf4b906670b9f242c71228fb33890a3", got "/sys/fs/cgroup/systemd/libpod_parent/libpod-6bfb827ebe2ecd17495201196e816f935cf4b906670b9f242c71228fb33890a3"
goroutine 1 [running]:
github.com/opencontainers/runc/libcontainer.getUnifiedPath(0x40002c6b10, 0x555b362400, 0x0)
github.com/opencontainers/runc/libcontainer/factory_linux.go:59 +0x2cc
github.com/opencontainers/runc/libcontainer.cgroupfs2.func1(0x4000255810, 0x40002c6b10, 0x4a, 0x7fcd8f8ed4)
github.com/opencontainers/runc/libcontainer/factory_linux.go:111 +0x30
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).Load(0x400025e090, 0x7fcd8f8ed4, 0x40, 0x0, 0x0, 0x1, 0x8)
github.com/opencontainers/runc/libcontainer/factory_linux.go:321 +0x14c
main.getContainer(0x400022c160, 0x555b372e78, 0x8, 0x1, 0x40002433f4)
github.com/opencontainers/runc/utils_linux.go:89 +0x9c
main.execProcess(0x400022c160, 0x0, 0x0, 0x0)
github.com/opencontainers/runc/exec.go:114 +0x30
main.glob..func5(0x400022c160, 0x555b7963c0, 0x40002434f8)
github.com/opencontainers/runc/exec.go:104 +0x6c
github.com/urfave/cli.HandleAction(0x555b463d60, 0x555b510bc8, 0x400022c160, 0x400022c160, 0x0)
github.com/urfave/cli@v1.22.1/app.go:523 +0x124
github.com/urfave/cli.Command.Run(0x555b36ef3e, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x555b386b5a, 0x28, 0x0, ...)
github.com/urfave/cli@v1.22.1/command.go:174 +0x408
github.com/urfave/cli.(*App).Run(0x400027a000, 0x40000cc000, 0x5a, 0x60, 0x0, 0x0)
github.com/urfave/cli@v1.22.1/app.go:276 +0x588
main.main()
github.com/opencontainers/runc/main.go:163 +0xa78: OCI runtime error
# /usr/bin/podman.old version
Version: 1.6.1
RemoteAPI Version: 1
Go Version: go1.12.10
OS/Arch: linux/arm64
Running podman 3.4.0 looks similar:
# ./podman-v3.4.0 exec -ti unifi-os bash
WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: open /sys/fs/cgroup/libpod_parent/conmon/cpuset.cpus: open /sys/fs/cgroup/libpod_parent/conmon/cpuset.cpus.effective: no such file or directory
Error: OCI runtime error: panic: expected "name=systemd" path to be unified path "/sys/fs/cgroup/devices/libpod_parent/libpod-6bfb827ebe2ecd17495201196e816f935cf4b906670b9f242c71228fb33890a3", got "/sys/fs/cgroup/systemd/libpod_parent/libpod-6bfb827ebe2ecd17495201196e816f935cf4b906670b9f242c71228fb33890a3"
goroutine 1 [running]:
github.com/opencontainers/runc/libcontainer.getUnifiedPath(0x400019acf0, 0x556223b400, 0x0)
github.com/opencontainers/runc/libcontainer/factory_linux.go:59 +0x2cc
github.com/opencontainers/runc/libcontainer.cgroupfs2.func1(0x40001c1f80, 0x400019acf0, 0x4a, 0x7fd17ccedf)
github.com/opencontainers/runc/libcontainer/factory_linux.go:111 +0x30
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).Load(0x4000194090, 0x7fd17ccedf, 0x40, 0x0, 0x0, 0x1, 0x8)
github.com/opencontainers/runc/libcontainer/factory_linux.go:321 +0x14c
main.getContainer(0x400012c6e0, 0x556224be78, 0x8, 0x1, 0x40001b339c)
github.com/opencontainers/runc/utils_linux.go:89 +0x9c
main.execProcess(0x400012c6e0, 0x0, 0x0, 0x0)
github.com/opencontainers/runc/exec.go:114 +0x30
main.glob..func5(0x400012c6e0, 0x556266f3c0, 0x40001b34f8)
github.com/opencontainers/runc/exec.go:104 +0x6c
github.com/urfave/cli.HandleAction(0x556233cd60, 0x55623e9bc8, 0x400012c6e0, 0x400012c6e0, 0x0)
github.com/urfave/cli@v1.22.1/app.go:523 +0x124
github.com/urfave/cli.Command.Run(0x5562247f3e, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x556225fb5a, 0x28, 0x0, ...)
github.com/urfave/cli@v1.22.1/command.go:174 +0x408
github.com/urfave/cli.(*App).Run(0x400014a700, 0x40000200e0, 0xe, 0xe, 0x0, 0x0)
github.com/urfave/cli@v1.22.1/app.go:276 +0x588
main.main()
github.com/opencontainers/runc/main.go:163 +0xa78
Am I right that this is not a podman problem, but rather a runc problem?
Okay, just upgrade to 1.11.0-16 - and since I am new to the UDMP universe (coming from a USG4P), I didn't see this thread until after updating 😉
Anyway, podman 1.6.1 is still on the system - called /usr/bin/podman.old - but running that doesn't work either:
# /usr/bin/podman.old exec -ti unifi-os bash Error: panic: expected "name=systemd" path to be unified path "/sys/fs/cgroup/devices/libpod_parent/libpod-6bfb827ebe2ecd17495201196e816f935cf4b906670b9f242c71228fb33890a3", got "/sys/fs/cgroup/systemd/libpod_parent/libpod-6bfb827ebe2ecd17495201196e816f935cf4b906670b9f242c71228fb33890a3" goroutine 1 [running]: github.com/opencontainers/runc/libcontainer.getUnifiedPath(0x40002c6b10, 0x555b362400, 0x0) github.com/opencontainers/runc/libcontainer/factory_linux.go:59 +0x2cc github.com/opencontainers/runc/libcontainer.cgroupfs2.func1(0x4000255810, 0x40002c6b10, 0x4a, 0x7fcd8f8ed4) github.com/opencontainers/runc/libcontainer/factory_linux.go:111 +0x30 github.com/opencontainers/runc/libcontainer.(*LinuxFactory).Load(0x400025e090, 0x7fcd8f8ed4, 0x40, 0x0, 0x0, 0x1, 0x8) github.com/opencontainers/runc/libcontainer/factory_linux.go:321 +0x14c main.getContainer(0x400022c160, 0x555b372e78, 0x8, 0x1, 0x40002433f4) github.com/opencontainers/runc/utils_linux.go:89 +0x9c main.execProcess(0x400022c160, 0x0, 0x0, 0x0) github.com/opencontainers/runc/exec.go:114 +0x30 main.glob..func5(0x400022c160, 0x555b7963c0, 0x40002434f8) github.com/opencontainers/runc/exec.go:104 +0x6c github.com/urfave/cli.HandleAction(0x555b463d60, 0x555b510bc8, 0x400022c160, 0x400022c160, 0x0) github.com/urfave/cli@v1.22.1/app.go:523 +0x124 github.com/urfave/cli.Command.Run(0x555b36ef3e, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x555b386b5a, 0x28, 0x0, ...) github.com/urfave/cli@v1.22.1/command.go:174 +0x408 github.com/urfave/cli.(*App).Run(0x400027a000, 0x40000cc000, 0x5a, 0x60, 0x0, 0x0) github.com/urfave/cli@v1.22.1/app.go:276 +0x588 main.main() github.com/opencontainers/runc/main.go:163 +0xa78: OCI runtime error # /usr/bin/podman.old version Version: 1.6.1 RemoteAPI Version: 1 Go Version: go1.12.10 OS/Arch: linux/arm64
Running podman 3.4.0 looks similar:
# ./podman-v3.4.0 exec -ti unifi-os bash WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: open /sys/fs/cgroup/libpod_parent/conmon/cpuset.cpus: open /sys/fs/cgroup/libpod_parent/conmon/cpuset.cpus.effective: no such file or directory Error: OCI runtime error: panic: expected "name=systemd" path to be unified path "/sys/fs/cgroup/devices/libpod_parent/libpod-6bfb827ebe2ecd17495201196e816f935cf4b906670b9f242c71228fb33890a3", got "/sys/fs/cgroup/systemd/libpod_parent/libpod-6bfb827ebe2ecd17495201196e816f935cf4b906670b9f242c71228fb33890a3" goroutine 1 [running]: github.com/opencontainers/runc/libcontainer.getUnifiedPath(0x400019acf0, 0x556223b400, 0x0) github.com/opencontainers/runc/libcontainer/factory_linux.go:59 +0x2cc github.com/opencontainers/runc/libcontainer.cgroupfs2.func1(0x40001c1f80, 0x400019acf0, 0x4a, 0x7fd17ccedf) github.com/opencontainers/runc/libcontainer/factory_linux.go:111 +0x30 github.com/opencontainers/runc/libcontainer.(*LinuxFactory).Load(0x4000194090, 0x7fd17ccedf, 0x40, 0x0, 0x0, 0x1, 0x8) github.com/opencontainers/runc/libcontainer/factory_linux.go:321 +0x14c main.getContainer(0x400012c6e0, 0x556224be78, 0x8, 0x1, 0x40001b339c) github.com/opencontainers/runc/utils_linux.go:89 +0x9c main.execProcess(0x400012c6e0, 0x0, 0x0, 0x0) github.com/opencontainers/runc/exec.go:114 +0x30 main.glob..func5(0x400012c6e0, 0x556266f3c0, 0x40001b34f8) github.com/opencontainers/runc/exec.go:104 +0x6c github.com/urfave/cli.HandleAction(0x556233cd60, 0x55623e9bc8, 0x400012c6e0, 0x400012c6e0, 0x0) github.com/urfave/cli@v1.22.1/app.go:523 +0x124 github.com/urfave/cli.Command.Run(0x5562247f3e, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x556225fb5a, 0x28, 0x0, ...) github.com/urfave/cli@v1.22.1/command.go:174 +0x408 github.com/urfave/cli.(*App).Run(0x400014a700, 0x40000200e0, 0xe, 0xe, 0x0, 0x0) github.com/urfave/cli@v1.22.1/app.go:276 +0x588 main.main() github.com/opencontainers/runc/main.go:163 +0xa78
Am I right that this is not a podman problem, but rather a runc problem?
It looks like runc problem.
Is it safe to upgrade to 1.10.4 which was released couple days ago?
I have had no issues with it
Sent from my iPhone
On Oct 18, 2021, at 18:53, leeaash @.***> wrote:
Is it safe to upgrade to 1.10.4 which was released couple days ago?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.
Updated podman to 3.3.0 with the latest script and same issues as everyone else - can't get into unifi-os shell and none of the containers start. Couldn't find a workaround yet besides leaving out the podman update and sticking with 1.6.1
Network controller still doesn't start for for me, but at least everything is online
Network controller issue was related to system running out of space and had nothing to do with podman 3.3.0 upgrade so scrap that
I still don't know how to fix this. Running on 1.11.0-19 at this point.
# unifi-os shell
WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: open /sys/fs/cgroup/libpod_parent/conmon/cpuset.cpus: open /sys/fs/cgroup/libpod_parent/conmon/cpuset.cpus.effective: no such file or directory
Error: OCI runtime error: panic: expected "name=systemd" path to be unified path "/sys/fs/cgroup/devices/libpod_parent/libpod-6f265cdb51b96adbdfd545e397f5efdc2bf6dd720458e6e276857a38e311880b", got "/sys/fs/cgroup/systemd/libpod_parent/libpod-6f265cdb51b96adbdfd545e397f5efdc2bf6dd720458e6e276857a38e311880b"
goroutine 1 [running]:
github.com/opencontainers/runc/libcontainer.getUnifiedPath(0x4000254c00, 0x555f7cb400, 0x0)
github.com/opencontainers/runc/libcontainer/factory_linux.go:59 +0x2cc
github.com/opencontainers/runc/libcontainer.cgroupfs2.func1(0x40001d5730, 0x4000254c00, 0x4a, 0x7ff1b67edf)
github.com/opencontainers/runc/libcontainer/factory_linux.go:111 +0x30
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).Load(0x40001de090, 0x7ff1b67edf, 0x40, 0x0, 0x0, 0x1, 0x8)
github.com/opencontainers/runc/libcontainer/factory_linux.go:321 +0x14c
main.getContainer(0x40001ae160, 0x555f7dbe78, 0x8, 0x1, 0x40001c5321)
github.com/opencontainers/runc/utils_linux.go:89 +0x9c
main.execProcess(0x40001ae160, 0x0, 0x0, 0x0)
github.com/opencontainers/runc/exec.go:114 +0x30
main.glob..func5(0x40001ae160, 0x555fbff3c0, 0x40001c54f8)
github.com/opencontainers/runc/exec.go:104 +0x6c
github.com/urfave/cli.HandleAction(0x555f8ccd60, 0x555f979bc8, 0x40001ae160, 0x40001ae160, 0x0)
github.com/urfave/cli@v1.22.1/app.go:523 +0x124
github.com/urfave/cli.Command.Run(0x555f7d7f3e, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x555f7efb5a, 0x28, 0x0, ...)
github.com/urfave/cli@v1.22.1/command.go:174 +0x408
github.com/urfave/cli.(*App).Run(0x40001fa000, 0x40000c2000, 0xe, 0xe, 0x0, 0x0)
github.com/urfave/cli@v1.22.1/app.go:276 +0x588
main.main()
github.com/opencontainers/runc/main.go:163 +0xa78
#
Hi @boostchicken, version 3.40 is also not uploaded to "https://github.com/boostchicken/udm-utilities/tree/master/podman-update/bin". 3.3.0 is the latest download available. Also the shell script is pointing to 3.3.0. Does version 3.4.0 have changes regarding to systemd?
This is now fixed
@boostchicken I'm sorry to report that it isn't, in fact, fixed. I just did a clean install on 1.11.0-21, and I'm still getting the exact same crash, and no access to unifi-os shell
.
Did you mean to push changes to the repo? It was last updated 4 days ago.
issue still present on 1.11.0-21 & 1.11.0-19
I'd recommend leaving Podman on the old version for the time being unless you really need one of the newer versions features
Do you know how to safely roll back podman? Attempting to use podman.old results in a very similar error message.
In the first message there is a workaround to use old podman. Basically avoid the update by disabling the update script and reboot.
Do you know how to safely roll back podman? Attempting to use podman.old results in a very similar error message.
Yes, like gtrabanco said, just don't execute the podman update script and reboot. The UDM always goes back to it's initial state, then executes your boot scripts one by one
Describe the bug I could not get into
unifi-os shell
. So, as you can see in Workaround part I had to disable the startup update script forpodman
,runc
&conmon
because it fails when updating is done.I had the issue described in official forum (log is also in the bottom of this issue) by error thinking was an update problem to beta 1.11.0.14.
The problems are:
unifi-os
container.unifi-os
is not done.To Reproduce Steps to reproduce the behavior:
unifi-os
Only when you have script 01-podman-update.sh.Expected behavior Update podman, runc & conmon and start all containers without problems.
Screenshots If applicable, add screenshots to help explain your problem.
UDM Information
Log report
I tried to execute the script manually to see if there is any error in the process but no error was given.
Workaround
To solve the issue I had to disable
01-podman-update.sh
on startup and reboot.NOTE: Rename of
/usr/bin/podman.old
,/usr/bin/runc.old
and/usr/libexec/podman/conmon.old
to its normal values did not resolve the issue.