search

unifi-utilities / unifios-utilities

A collection of enhancements for UnifiOS based devices
GNU General Public License v3.0
3.9k stars 419 forks source link

Podman failing to start on UDM SE v3.0.10 #428

Open mayankst opened 1 year ago

mayankst commented 1 year ago

Describe the bug Podan is failing to start on UDM SE v3.0.10 with following error

Oct 18 12:28:33 UDM-SE podman[1465457]: Error: OCI runtime error: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented

To Reproduce Steps to reproduce the behavior: Start the pi-hole container as service.

ExecStart=/usr/bin/podman run --cidfile=%t/%n.ctr-id --cgroups=no-conmon --pull=always --rm --sdnotify=conmon --replace -d --network dns --name pihole -e "TZ=America/Los Angeles" -v "/mnt/data/etc-pihole/:/etc/pihole/" -v "/mnt/data/pihole/etc-dnsmasq.d/:/etc/dnsmasq.d/" --dns=127.0.0.1 --dns=1.1.1.1 --dns=8.8.8.8 --hostname pi.hole -e VIRTUAL_HOST="pi.hole" -e PROXY_LOCATION="pi.hole" -e ServerIP="192.168.4.3" -e IPv6="False" pihole/pihole:latest

UDM Information

chrisdooks commented 1 year ago

+1 I think it's because of the upgrade of Debian, it requires Podman v4 to work with the new kernel.

mayankst commented 1 year ago

@chrisdooks Do you know how to build new podman version for unifi? I tried following the instructions in the readme but doesn't seem to work. @boostchicken Can you please help?

If you can provide the instructions to build, then I can build and open a PR.

chrisdooks commented 1 year ago

I'm afraid not, it's above my level of knowledge.

mayankst commented 1 year ago

@boostchicken can you please help. I tried building podman on wsl following your instructions but I am getting compile errors due to missing libseccomp. I tried installing arm64 version of that lib but wasn't successful as dependencies won't install. Main base dependency is libc6:arm64, when I try to install that, apt wants to remove amd64 version of that. So, I am not sure what to do next. Can you please help as pi-hole is completely down due to this incompatibility.

mayankst commented 1 year ago

So, I am able to compile the podman but container still won't start. Keep getting Exit status 1.

sanderdewit commented 1 year ago

I'm interested in this as well. @mayankst can you share the compiling instructions?

T3LC0 commented 1 year ago

@mayankst how did you solve the multiarch problem with getting that libc6:arm64 package installed? Trying to compile this myself since support had me upgrade to EA to fix a problem that turned out to not be fixable and I want to run home bridge on my UDM SE with the EA software.

mayankst commented 1 year ago

@T3LC0 Yea, I got it to compile but unfortunately looks like Unifi have borked the any ability to run containers in v3 of the OS. There are some major dependencies missing from the OS which are causing runc to not work. At this point I have given up (spent over a month on this researching and debugging) and moved on to alternative solution.

dquach93 commented 1 year ago

@mayankst whats the alternative solution you moved to?

mayankst commented 1 year ago

@mayankst whats the alternative solution you moved to?

Running Ad Guard home on home assistant. Could have done pi-hole on rPi as well, but I already have the home assistant so it was easier.

dquach93 commented 1 year ago

@mayankst whats the alternative solution you moved to?

Running Ad Guard home on home assistant. Could have done pi-hole on rPi as well, but I already have the home assistant so it was easier.

Ah i see. Im using this for wpa_supplicant. Didnt see this issue till after updating to 3.0.12 today...

emilvissing commented 1 year ago

@T3LC0 Yea, I got it to compile but unfortunately looks like Unifi have borked the any ability to run containers in v3 of the OS. There are some major dependencies missing from the OS which are causing runc to not work. At this point I have given up (spent over a month on this researching and debugging) and moved on to alternative solution.

did you try setting up podman on 3.x using the latest podman/runc/conmon that @boostchicken pushed a couple of days ago?

TotalGriffLock commented 1 year ago

@T3LC0 Yea, I got it to compile but unfortunately looks like Unifi have borked the any ability to run containers in v3 of the OS. There are some major dependencies missing from the OS which are causing runc to not work. At this point I have given up (spent over a month on this researching and debugging) and moved on to alternative solution.

did you try setting up podman on 3.x using the latest podman/runc/conmon that @boostchicken pushed a couple of days ago?

Can confirm same behaviour here on a UDM-SE on v3.0.13 and that's with @boostchicken latest published podman. It would seem that the BPF_CGROUP_DEVICE call has been removed from the new kernel on v3 of the OS.

kennylerma commented 1 year ago

Just noting I have the same issue :( I grabbed the latest podman for UDM Pro SE and still the same start error.

boostchicken commented 1 year ago

https://github.com/fabianishere/udm-kernel-tools

You need that for the UDM-SE

simkin commented 1 year ago

https://github.com/fabianishere/udm-kernel-tools

You need that for the UDM-SE

Kindly elborate as I have some doubts this will work: https://github.com/fabianishere/udm-kernel-tools/issues/82 If it does work, which custom kernel must be loaded ? Edge?

mazzy89 commented 1 year ago

@boostchicken the project is not yet ready for UDM Pro SE.

mazzy89 commented 1 year ago

Lookign at the kernel file what it has been stripped out by the stock kernel is the CONFIG_BPF_SYSCALL kernel parameter. it must be set to y because Podman same as other containers runtime use nowadays ebpf to hooks programs into the kernel. The only way here is to have a custom kernel into our UDM SE and to achieve this, we need to bring the support for UDM SE into here https://github.com/fabianishere/udm-kernel-tools and have installed into our UDM SE the edge version. I will try to check what it is needed to be done there.

andrewmiskell commented 1 year ago

@mayankst whats the alternative solution you moved to?

Running Ad Guard home on home assistant. Could have done pi-hole on rPi as well, but I already have the home assistant so it was easier.

Ah i see. Im using this for wpa_supplicant. Didnt see this issue till after updating to 3.0.12 today...

On 2.x firmwares, you can just install the Debian package for wpa_supplicant instead of running it in a container so you can break the reliance on podman for that. (Assuming you're using it like I am with AT&T Fiber).

Haven't tried it on 3.x yet (I'm using a UDM and just got the 2.x firmware yesterday).

SirUli commented 1 year ago

Apparently rootless containers fail. e.g. the https://hub.docker.com/r/tusc/chrony-udm still works for me.

foxcris commented 1 year ago

I am on UDM SE v3.0.13. There you can install podman directly via apt from the debian repository. Have you tried to use that package? Installation runs without any errors but i haven't tried to run a container yet. Here the output of podman info:

host:
  arch: arm64
  buildahVersion: 1.19.6
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 4
  distribution:
    distribution: debian
    version: "11"
  eventLogger: journald
  hostname: UDM-SE
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.19.152-ui-alpine
  linkmode: dynamic
  memFree: 161476608
  memTotal: 4137881600
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 7509372928
  swapTotal: 7516188672
  uptime: 191h 36m 17.01s (Approximately 7.96 days)
registries: {}
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphStatus: {}
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.15.15
  OsArch: linux/arm64
  Version: 3.0.1
mazzy89 commented 1 year ago

@foxcris it won't work. This is not a problem of podman. The problem is in the OS kernel which come with the newer firmware version.

foxcris commented 1 year ago

@mazzy89 You are right. Just tried the hello world container and i am getting:

podman run hello-world
Resolved "hello-world" as an alias (/etc/containers/registries.conf.d/shortnames.conf)
Trying to pull docker.io/library/hello-world:latest...
Getting image source signatures
Copying blob 7050e35b49f5 done
Copying config 46331d942d done
Writing manifest to image destination
Storing signatures
Error: open /proc/self/uid_map: no such file or directory
bennettp123 commented 1 year ago

@foxcris I think the issue is probably crun—it requires the kernel to be compiles with CONFIG_USER_NS, but the UDM kernels aren't.

What happens if you switch to the older runc ociRuntime instead?

apt-get remove crun
apt-get install runc
jimmy-ungerman commented 1 year ago

What happens if you switch to the older runc ociRuntime instead?

Just tried this on my UDM SE v3.0.13 and still the same issue.

foxcris commented 1 year ago

@bennettp123 Same error when i use runc instaed of crun.

foxcris commented 1 year ago

Isn't it an option to run a standard debian linux kernel?

frasderp commented 1 year ago

@foxcris any progress on this?

foxcris commented 1 year ago

No. I didn't had time to try to use a standard Debian kernel. Don't want to brick my udm se. Till now I can install podman but it's not working due to the missing kernel features.

cftechwiz commented 1 year ago

FWIW - I did actually brick my UDM on one occasion when I accidentally executed an rm -rf at the wrong subfolder and removed a good chunk of the OS. Turns out there is a neat trick to bring it back to stock from an embedded image.
The instructions I got from Support were as follows:

  1. Remove the unit from your network and disconnect the cables from the unit.
  2. Press down the reset button for 40+ seconds without power and cables.
  3. Release the reset button and power the unit.
  4. Again press the reset button for 15+ seconds.

I did this and it worked like a charm.

cftechwiz commented 1 year ago

Hope someone can devise a solution, at present, I am seriously re-considering a downgrade back to 2.5.11. REALLY need to be able to run FRR for BGP. Is anyone aware of another way? Hate to think it, but it really feels like this was done with intentionality.

andrewmiskell commented 1 year ago

Hope someone can devise a solution, at present, I am seriously re-considering a downgrade back to 2.5.11. REALLY need to be able to run FRR for BGP. Is anyone aware of another way? Hate to think it, but it really feels like this was done with intentionality.

Does it have to be a container? FRR provides packages for all recent versions of Debian, which is the core OS using in 2.x and 3.x.

IIRC, 3.x's core is Debian Bullseye. You should be able to just install the arm64 package and configure it normally.

jimmy-ungerman commented 1 year ago

I'm fairly new to my UDM-SE and this project, but do we know if packages we install on our own are reloaded on reboot or maintain through upgrades?

andrewmiskell commented 1 year ago

I'm fairly new to my UDM-SE and this project, but do we know if packages we install on our own are reloaded on reboot or maintain through upgrades?

I know they persistent through reboots, as it does on mine. Can't speak to a firmware upgrade yet.

WisdomWolf commented 1 year ago

The packages definitely do not persist through firmware upgrades. I had installed fish on the previous firmware (and stupidly had set it as my default shell) so after the upgrade I lost SSH access because it was pointing at an invalid shell. Unfortunately since there is only one user I had to factory reset in order to restore access. I only figured out the issue by having the logs sent to a remote syslog server so I could read them.

mazzy89 commented 1 year ago

It's quite hilarious how the lack of a container runtime in a linux box is send to panic a lot of ubiquiti users. The industry has run without containers runtime for ages. You can do everything that you can do with podman also without podman. I run bgp having bird binary on the machine. You can persist packages as well. It might be not handy as running a container runtime. Containers is well know thay have made deployment and lifecycle of the app easier but still you can do eveyrhing even without podman.

wjbridge commented 1 year ago

I was trying to get ADgaurdhome running without podman but I cannot attach it to a network interface (i.e. 192.168.1.0) on port 53. Port 53 was taken by another program. Does anybody have any guidelines on how to do this without podman?

foxcris commented 1 year ago

You should bind the already installed dns resolver (probably dnsmasq) to local host and configure adguardhome to bind to your specific local network network ips.

An other option is to bind dnsmasq to a different port.

19.01.2023 01:23:18 wjbridge @.***>:

I was trying to get ADgaurdhome running without podman but I cannot attach it to a network interface (i.e. 192.168.1.0) on port 53. Port 53 was taken by another program. Does anybody have any guidelines on how to do this without podman?

— Reply to this email directly, view it on GitHub[https://github.com/unifi-utilities/unifios-utilities/issues/428#issuecomment-1396275895], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AACYGU5FQXGAWF3QGV72SA3WTCCOXANCNFSM6AAAAAARINPA2Q]. You are receiving this because you were mentioned.[Verfolgungsbild][https://github.com/notifications/beacon/AACYGU7Y4K656B57WVMVUR3WTCCOXA5CNFSM6AAAAAARINPA2SWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSTHF5LO.gif]

cftechwiz commented 1 year ago

It's quite hilarious how the lack of a container runtime in a linux box is send to panic a lot of ubiquiti users. The industry has run without containers runtime for ages. You can do everything that you can do with podman also without podman. I run bgp having bird binary on the machine. You can persist packages as well. It might be not handy as running a container runtime. Containers is well know thay have made deployment and lifecycle of the app easier but still you can do eveyrhing even without podman.

Just an easy way to leverage MacVLAN, and eliminate port conflicts, maintain isolation.

wjbridge commented 1 year ago

You should bind the already installed dns resolver (probably dnsmasq) to local host and configure adguardhome to bind to your specific local network network ips. An other option is to bind dnsmasq to a different port. 19.01.2023 01:23:18 wjbridge @.***>: … I was trying to get ADgaurdhome running without podman but I cannot attach it to a network interface (i.e. 192.168.1.0) on port 53. Port 53 was taken by another program. Does anybody have any guidelines on how to do this without podman? — Reply to this email directly, view it on GitHub[#428 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AACYGU5FQXGAWF3QGV72SA3WTCCOXANCNFSM6AAAAAARINPA2Q]. You are receiving this because you were mentioned.[Verfolgungsbild][https://github.com/notifications/beacon/AACYGU7Y4K656B57WVMVUR3WTCCOXA5CNFSM6AAAAAARINPA2SWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSTHF5LO.gif]

@foxcris Do you know how to do this? I don't really know how to change dnsmasq to a different port within Unifi or change it to local host only for all networks defined. Any help for a noob would be appreciated.

boostchicken commented 1 year ago

It's quite hilarious how the lack of a container runtime in a linux box is send to panic a lot of ubiquiti users. The industry has run without containers runtime for ages. You can do everything that you can do with podman also without podman. I run bgp having bird binary on the machine. You can persist packages as well. It might be not handy as running a container runtime. Containers is well know thay have made deployment and lifecycle of the app easier but still you can do eveyrhing even without podman.

Let us not judge our container loving users! Alas, it does enable people to do things that escape their current knowledge / expertise.

I am sure everyone would be grateful if you wrote steps on how to achieve your setup without containers and contributed them to the repo @mazzy89

boostchicken commented 1 year ago

You should bind the already installed dns resolver (probably dnsmasq) to local host and configure adguardhome to bind to your specific local network network ips. An other option is to bind dnsmasq to a different port. 19.01.2023 01:23:18 wjbridge @.***>: … I was trying to get ADgaurdhome running without podman but I cannot attach it to a network interface (i.e. 192.168.1.0) on port 53. Port 53 was taken by another program. Does anybody have any guidelines on how to do this without podman? — Reply to this email directly, view it on GitHub[#428 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AACYGU5FQXGAWF3QGV72SA3WTCCOXANCNFSM6AAAAAARINPA2Q]. You are receiving this because you were mentioned.[Verfolgungsbild][https://github.com/notifications/beacon/AACYGU7Y4K656B57WVMVUR3WTCCOXA5CNFSM6AAAAAARINPA2SWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSTHF5LO.gif]

@foxcris Do you know how to do this? I don't really know how to change dnsmasq to a different port within Unifi or change it to local host only for all networks defined. Any help for a noob would be appreciated.

If you don't already know how to do this I suggest sticking with containers and letting the CNI handle it for you. You need to create a new netns and have your programs run in it. The CNI plugins just automate this for you.

mazzy89 commented 1 year ago

I do love containers too and I'm a big supporter of kubernetes and anything related to containers technologies as you can see from my GH. But containers are a way among many ones to get to the final result. Let's not sediment us to just use container in a box where officially they are not allowed anymore from the vendor. Adaption is the way.

The way how I achieve the same thing is via saltstack. I'm not sure if users here they do want to learn saltstack and manage their router with that. If there is the interest I will definitely share it but I doubt it.

boostchicken commented 1 year ago

I do love containers too and I'm a big supporter of kubernetes and anything related to containers technologies as you can see from my GH. But containers are a way among many ones to get to the final result. Let's not sediment us to just use container in a box where officially they are not allowed anymore from the vendor. Adaption is the way.

The way how I achieve the same thing is via saltstack. I'm not sure if users here they do want to learn saltstack and manage their router with that. If there is the interest I will definitely share it but I doubt it.

Nowhere did it say containers are not allowed. There just isn't support for it out of the box anymore. I would argue containers are better because it keeps all customization and stuff isolated, leading to less issues.

boostchicken commented 1 year ago

The way how I achieve the same thing is via saltstack. I'm not sure if users here they do want to learn saltstack and manage their router with that. If there is the interest I will definitely share it but I doubt it.

I didn't think anyone would have interest in this repo, much less it growing to become the bible on UniFi OS Console customization. Put it out there! If it works well you may kill the need for containers, the community will decide!

mazzy89 commented 1 year ago

Ok @boostchicken i will definitely.

aescolastico commented 1 year ago

is there an alternate way to run homebridge on udmse outside of podman?

aescolastico commented 1 year ago

this still seems above my head, but it seems from the convo, that it might be a while before the podman/udm kernel gets updated. so for now ill just install homebridge directly on the udm via their debian 11 package https://github.com/homebridge/homebridge/wiki/Install-Homebridge-on-Debian-or-Ubuntu-Linux

aescolastico commented 1 year ago

it just occurred to me that homebridge probably cant be discovered by homekit if its advertising on the same ip as your networks default gateway. which is the case when you install it on the udmse.

aescolastico commented 1 year ago

is it possible to add a virtual interface to the debian install on the udmse? the purpose would be to assign an ip to homebridge that isnt the gateway address. perhaps that could be used as an alternative to container networking.