Kubernetes: dbus error sockopt_get_peersec: Invalid argument

Va-Pros commented 7 months ago

Good afternoon! \ Faced the same problem as #601 . Launching a container in docker with cgroup v2 and user namespace remapping works great locally. However, the following problem occurs in kubernetes:

...
2024-04-10 Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
2024-04-10   [1/30]: configuring certificate server instance
2024-04-10   [2/30]: stopping certificate server instance to update CS.cfg
2024-04-10   [3/30]: backing up CS.cfg
2024-04-10   [4/30]: Add ipa-pki-wait-running
2024-04-10   [5/30]: secure AJP connector
2024-04-10   [6/30]: reindex attributes
2024-04-10   [7/30]: exporting Dogtag certificate store pin
2024-04-10   [8/30]: disabling nonces
2024-04-10   [9/30]: set up CRL publishing
2024-04-10   [10/30]: enable PKIX certificate path discovery and validation
2024-04-10   [11/30]: authorizing RA to modify profiles
2024-04-10   [12/30]: authorizing RA to manage lightweight CAs
2024-04-10   [13/30]: Ensure lightweight CAs container exists
2024-04-10   [14/30]: Ensuring backward compatibility
2024-04-10   [15/30]: starting certificate server instance
2024-04-10   [16/30]: configure certmonger for renewals
2024-04-10   [error] CalledProcessError: CalledProcessError(Command ['/bin/systemctl', 'start', 'certmonger.service'] returned non-zero exit status 1: 'Job for certmonger.service failed because the control process exited with error code.\nSee "systemctl status certmonger.service" and "journalctl -xeu certmonger.service" for details.\n')
2024-04-10 FreeIPA server configuration failed.

Accordingly, I found in the logs that dbus-broker.service could not start:

Apr 10 11:51:04 freeipa-statefulset-0 systemd[1]: Started D-Bus System Message Bus.
Apr 10 11:51:04 freeipa-statefulset-0 systemd[1]: Reached target Minimal target for containerized FreeIPA server.
Apr 10 11:51:04 freeipa-statefulset-0 dbus-broker-launch[255]: ERROR sockopt_get_peersec @ ../src/util/sockopt.c +41: Invalid argument
Apr 10 11:51:04 freeipa-statefulset-0 dbus-broker-launch[255]:       broker_new @ ../src/broker/broker.c +105
Apr 10 11:51:04 freeipa-statefulset-0 dbus-broker-launch[255]:       run @ ../src/broker/main.c +261
Apr 10 11:51:04 freeipa-statefulset-0 dbus-broker-launch[255]:       main @ ../src/broker/main.c +295
Apr 10 11:51:04 freeipa-statefulset-0 dbus-broker-launch[241]: ERROR service_add @ ../src/launch/service.c +921: Transport endpoint is not connected
Apr 10 11:51:04 freeipa-statefulset-0 dbus-broker-launch[241]:       launcher_add_services @ ../src/launch/launcher.c +804
Apr 10 11:51:04 freeipa-statefulset-0 dbus-broker-launch[241]:       launcher_run @ ../src/launch/launcher.c +1409
Apr 10 11:51:04 freeipa-statefulset-0 dbus-broker-launch[241]:       run @ ../src/launch/main.c +152
Apr 10 11:51:04 freeipa-statefulset-0 dbus-broker-launch[241]:       main @ ../src/launch/main.c +178
Apr 10 11:51:04 freeipa-statefulset-0 dbus-broker-launch[241]: Exiting due to fatal error: -107
Apr 10 11:51:04 freeipa-statefulset-0 systemd[1]: dbus-broker.service: Main process exited, code=exited, status=1/FAILURE
Apr 10 11:51:04 freeipa-statefulset-0 systemd[1]: dbus-broker.service: Failed with result 'exit-code'.

Part of the yaml container:

          image: freeipa/freeipa-server:rocky-9
          name: freeipa
          args:
           - "ipa-server-install"
          volumeMounts:
          - name: freeipa-data
            mountPath: "/data"
          - name: freeipa-run
            mountPath: "/run"
          - name: freeipa-tmp
            mountPath: "/tmp"
          - name: freeipa-journal
            mountPath: "/var/log/journal"
          - name: freeipa-shm
            mountPath: "/dev/shm"
          - name: cgroups
            mountPath: "/sys/fs/cgroup"
            readOnly: false
      volumes:
      - name: freeipa-run
        emptyDir: {}
      - name: freeipa-tmp
        emptyDir: {}
      - name: freeipa-journal
        emptyDir: {}
      - name: freeipa-shm
        emptyDir:
          medium: Memory
          sizeLimit: 2Gi
      - name: cgroups
        hostPath:
          path: /sys/fs/cgroup
          type: ""

We use:

kubernetes v1.27.7
kernel-version 5.4.0-110-generic
container-runtime containerd://1.6.20-4
cgroup v2
selinux: disabled

Since there is no docker on the nodes, we cannot install user namespace remapping in the docker daemon. Do you have any ideas about launching dbus-broker.service in this environment?

Thanks!

abbra commented 7 months ago

Please see https://kubernetes.io/blog/2023/09/13/userns-alpha/ and references there in 'Node system requirements' section.

Va-Pros commented 7 months ago

Please see https://kubernetes.io/blog/2023/09/13/userns-alpha/ and references there in 'Node system requirements' section.

Thanks for the answer, apparently, user namespace remapping will not be able to be used in the current environment. But is user namespace remapping necessary to run dbus-broker.service?

adelton commented 7 months ago

Technically it might not be and we saw FreeIPA container test running on OpenShift without namespaces in the past. But running a fairly big stack as true root on the host might not be what you want in any serious production setup anyway. I assume that similar to the example in https://github.com/freeipa/freeipa-container/issues/601 you attempt to run it with privileged: true. And the disabled SELinux also is not something you'd want.

That error might be something small or silly, or it might mean that the dbus broker tries to connect somewhere on the host it has no business connecting. You might even be able to get it working by disabling seccomp but it is a setup you likely do not want to rely on.

Lately we've been testing (in GitHub Actions in this repo) with K3s with docker (via cri-dockerd) which allowed us to set { "userns-remap": "default" }, which incidently avoids the dbus issue. Having a separate node in your Kubernetes cluster where you'd use the docker runtime and then assigning the FreeIPA pod to that node might be your best bet at the moment.

Va-Pros commented 7 months ago

Technically it might not be and we saw FreeIPA container test running on OpenShift without namespaces in the past. But running a fairly big stack as true root on the host might not be what you want in any serious production setup anyway. I assume that similar to the example in #601 you attempt to run it with privileged: true. And the disabled SELinux also is not something you'd want.

That error might be something small or silly, or it might mean that the dbus broker tries to connect somewhere on the host it has no business connecting. You might even be able to get it working by disabling seccomp but it is a setup you likely do not want to rely on.

Lately we've been testing (in GitHub Actions in this repo) with K3s with docker (via cri-dockerd) which allowed us to set { "userns-remap": "default" }, which incidently avoids the dbus issue. Having a separate node in your Kubernetes cluster where you'd use the docker runtime and then assigning the FreeIPA pod to that node might be your best bet at the moment.

Thanks for your reply! I am not using privileged true, instead I am using cgroup mounting. But this is not enough, apparently, as you wrote, "userns-remap": "default" } is needed.

adelton commented 7 months ago

Out of curiosity, what is the the OS and version on that worker node?

Va-Pros commented 7 months ago

Out of curiosity, what is the the OS and version on that worker node?

AstraLinuxSE 8.3.0-6

freeipa / freeipa-container

Kubernetes: dbus error sockopt_get_peersec: Invalid argument #606