search

ES-Nix / podman-rootless

Example of using nix + flakes to have podman rootless working
MIT License
14 stars 0 forks source link

The CNI plugins, cni and cni-plugins, and the podman rootless with sudo #11

Open PedroRegisPOAR opened 2 years ago

PedroRegisPOAR commented 2 years ago

Abstract

I think the problem is related to this:

The default list is:

cni_plugin_dirs = [
"/usr/local/libexec/cni",
"/usr/libexec/cni",
"/usr/local/lib/cni",
"/usr/lib/cni",
"/opt/cni/bin",
]

From: https://github.com/containers/common/blob/main/docs/containers.conf.5.md#network-table

TODO: does it solve the problem? 

rootless_storage_path = "$HOME/.local/share/containers/storage"

https://github.com/containers/podman/blob/c26af00c4bf5aec458868b5afd44e7a88ddcf46d/vendor/github.com/containers/storage/storage.conf#L24

cni_plugin_dirs = [
  "/usr/local/libexec/cni",
  "/usr/libexec/cni",
  "/usr/local/lib/cni",
  "/usr/lib/cni",
  "/opt/cni/bin",
  "/nix/store/some thing",
]

But for conmon it includes /run/current-system/sw/bin/conmon:

conmon_path=[
    "/usr/libexec/podman/conmon",
    "/usr/local/libexec/podman/conmon",
    "/usr/local/lib/podman/conmon",
    "/usr/bin/conmon",
    "/usr/sbin/conmon",
    "/usr/local/bin/conmon",
    "/usr/local/sbin/conmon",
    "/run/current-system/sw/bin/conmon",
]

Plausible solution not using symbolic links, as the documentation says, configure the cni_plugin_dirs. Where find a config file example? The FORMAT section does not give an example. Maybe search in github :bulb:

Maybe related: https://github.com/containers/podman/issues/11358#issuecomment-908414055

Maybe it explains why the network named podman is not created by default: cni and How To Install Podman on Debian 10/9, TODO: test it.

Details

TODO: add VM commads here

echo 'Start uidmap instalation!' \
&& sudo apt-get update \
&& sudo apt-get install -y uidmap \
&& echo 'End uidmap instalation!' \
&& echo 'Start a instalation with nix!' \
&& nix \
    profile \
    install \
    github:ES-Nix/podman-rootless/from-nixpkgs \
    nixpkgs#cni \
    nixpkgs#cni-plugins \
&& echo 'Start bypass sudo podman stuff...' \
&& sudo \
    --preserve-env \
    su \
    -c \
      "echo $USER ALL=\(ALL\) NOPASSWD:SETENV: $(readlink $(which podman)) >> /etc/sudoers" \
&& sudo \
    sed \
      -i \
      's@Defaults\ssecure_path=\"@&'"$HOME"'\/.nix-profile\/bin:@' \
      /etc/sudoers \
&& echo 'End bypass sudo podman stuff...' \
&& nix store gc \
&& sudo -k -n podman network create podman \
&& sudo -k -n podman pull busybox \
&& sudo reboot
sudo -k -n podman run -it --rm busybox echo 'Ok!'
sudo ln -fsv $(which firewall) /usr/lib/cni/firewall
sudo ln -fsv $(which bridge) /usr/lib/cni/bridge
sudo ln -fsv $(which portmap) /usr/lib/cni/portmap
sudo ln -fsv $(which tuning) /usr/lib/cni/tuning
sudo ln -fsv $(which host-local) /usr/lib/cni/host-local

Now it must work:

sudo -k -n podman run -it --rm busybox echo 'Ok!'
sudo mkdir -p /usr/lib/cni \
&& sudo ln -fsv $(which bandwidth) /usr/lib/cni/bandwidth \
&& sudo ln -fsv $(which bridge) /usr/lib/cni/bridge \
&& sudo ln -fsv $(which dhcp) /usr/lib/cni/dhcp \
&& sudo ln -fsv $(which firewall) /usr/lib/cni/firewall \
&& sudo ln -fsv $(which host-device) /usr/lib/cni/host-device \
&& sudo ln -fsv $(which host-local) /usr/lib/cni/host-local \
&& sudo ln -fsv $(which ipvlan) /usr/lib/cni/ipvlan \
&& sudo ln -fsv $(which loopback) /usr/lib/cni/loopback \
&& sudo ln -fsv $(which macvlan) /usr/lib/cni/macvlan \
&& sudo ln -fsv $(which portmap) /usr/lib/cni/portmap \
&& sudo ln -fsv $(which ptp) /usr/lib/cni/ptp \
&& sudo ln -fsv $(which sbr) /usr/lib/cni/sbr \
&& sudo ln -fsv $(which static) /usr/lib/cni/static \
&& sudo ln -fsv $(which tuning) /usr/lib/cni/tuning \
&& sudo ln -fsv $(which vlan) /usr/lib/cni/vlan \
&& sudo ln -fsv $(which vrf) /usr/lib/cni/vrf \
&& sudo ln -fsv $(which crio) /usr/lib/crio

Per discussion, it sounds like we're going to swap it back to a Requires. I remember it being swapped originally so rootless Podman did not need to install root-only dependencies, but if that results in broken installations, it doesn't seem to be worth it. From: https://github.com/containers/podman/issues/3679#issuecomment-589177020

TODO: https://gitlab.com/steveeJ/infra/-/blob/72b24bc3fda768c0c34cc9606321ac4df691b66a/nix/home-manager/programs/podman.nix

PedroRegisPOAR commented 2 years ago 
echo 'Start kvm stuff...' \
&& getent group kvm || sudo groupadd kvm \
&& sudo usermod --append --groups kvm "$USER" \
&& echo 'End kvm stuff!' \
&& echo 'Start cgroup v2 instalation...' \
&& sudo mkdir -p /etc/systemd/system/user@.service.d \
&& sudo sh -c "echo '[Service]' >> /etc/systemd/system/user@.service.d/delegate.conf" \
&& sudo sh -c "echo 'Delegate=yes' >> /etc/systemd/system/user@.service.d/delegate.conf" \
&& sudo \
    sed \
    --in-place \
    's/^GRUB_CMDLINE_LINUX="/&cgroup_enable=memory swapaccount=1 systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all/' \
    /etc/default/grub \
&& sudo grub-mkconfig -o /boot/grub/grub.cfg \
&& echo 'End cgroup v2 instalation...' \
&& echo 'Start uidmap instalation!' \
&& sudo apt-get update \
&& sudo apt-get install -y uidmap \
&& echo 'End uidmap instalation!' \
&& echo 'Start a lot of instalation with nix!' \
&& nix \
    profile \
    install \
    github:ES-Nix/podman-rootless/from-nixpkgs \
    nixpkgs#cni \
    nixpkgs#cni-plugins \
    nixpkgs#kubernetes-helm \
    nixpkgs#minikube \
    nixpkgs#ripgrep \
&& echo 'Start bypass sudo podman stuff...' \
&& sudo \
    --preserve-env \
    su \
    -c \
      "echo $USER ALL=\(ALL\) NOPASSWD:SETENV: $(readlink $(which podman)) >> /etc/sudoers" \
&& sudo \
    sed \
      -i \
      's@Defaults\ssecure_path=\"@&'"$HOME"'\/.nix-profile\/bin:@' \
      /etc/sudoers \
&& echo 'End bypass sudo podman stuff...' \
&& sudo mkdir -p /usr/lib/cni \
&& sudo ln -fsv $(which firewall) /usr/lib/cni/firewall \
&& sudo ln -fsv $(which bridge) /usr/lib/cni/bridge \
&& sudo ln -fsv $(which portmap) /usr/lib/cni/portmap \
&& sudo ln -fsv $(which tuning) /usr/lib/cni/tuning \
&& sudo ln -fsv $(which host-local) /usr/lib/cni/host-local \
&& nix store gc \
&& sudo -k -n podman network create podman \
&& sudo reboot
minikube start --driver=podman
sudo podman exec -it minikube bash -c 'podman --version && which podman && docker --version'

image

minikube kubectl -- apply -f https://k8s.io/examples/application/shell-demo.yaml
minikube kubectl -- get pod shell-demo

minikube kubectl -- exec --stdin --tty shell-demo -- /bin/bash -c 'ls -al /'
minikube kubectl -- delete pod shell-demo