search

haproxytech / helm-charts

Helm chart for HAProxy Kubernetes Ingress Controller
Apache License 2.0
155 stars 121 forks source link

binding to privileged port fail #219

Closed owanio1992 closed 10 months ago

owanio1992 commented 10 months ago

hello I am using helm install 1.10.10 version ingress and change value 

controller:
  containerPort:
    http: 80
    https: 443
    stat: 1024
  unprivileged: true
  allowPrivilegeEscalation: true
  kind: DaemonSet
  daemonset:
    useHostNetwork: true

but will get error

s6-overlay-preinit: fatal: unable to chown /var/run/s6: Operation not permitted

and I add CHOWN to securityContext.capabilities.add can fix this issue hope can update this config to helm thank very much

dkorunic commented 10 months ago

@owanio1992 There is no reason to bind to privileged ports for container. Can you explain your use-case?

owanio1992 commented 10 months ago

my case is I deploy in on-premises k8s and I set DNS to node IP ingress use host network (sorry my bad, I miss this config in early post ) so I can receive traffic from cluster out side , don't need extra load balancer for this reason, I want bind privileged ports for container

dkorunic commented 10 months ago

Can you try with these values and report if you are still seeing any kind of error?

controller:
  containerPort:
    http: 80
    https: 443
    stat: 1024
  kind: DaemonSet
  daemonset:
    useHostNetwork: true
    useHostPort: true
    hostPorts:
      http: 80
      https: 443
      stat: 1024
owanio1992 commented 10 months ago

Can you try with these values and report if you are still seeing any kind of error?

controller:
  containerPort:
    http: 80
    https: 443
    stat: 1024
  kind: DaemonSet
  daemonset:
    useHostNetwork: true
    useHostPort: true
    hostPorts:
      http: 80
      https: 443
      stat: 1024

I tested this config still get same error 

s6-overlay-preinit: fatal: unable to chown /var/run/s6: Operation not permitted
dkorunic commented 10 months ago

OK, can you give us some more details about your K8s environment? I would need K8s version and if there are any special pod/container restrictions in place, as well as Chart version you are running. To explain, typically our IC container runs unprivileged as UID/GID 1000 and mentioned /var/run/s6 should be created on /run/s6 (since /var/run is a symlink to /run) which is a tmpfs which has r/w/x permissions for everyone. For this initialisation to fail is fairly unusual, since folder should be already under haproxy user and chown is esentially noop so permissions shouldn't be needed.

owanio1992 commented 10 months ago

my k8s version: v1.28.4 Chart version: 1.36.1

thank for check this issue

dkorunic commented 10 months ago

Has there been any special security restrictions/hardening in place? Any kind of custom/3rd party Pod Security Admission setup, anything in particular apart from regular K8s v1.28 installation?

owanio1992 commented 10 months ago

no, don't have any extra config just a regular K8s v1.28

dkorunic commented 10 months ago

I have thoroughly checked the Helm Chart as well as tried to reproduce on v1.29 and v1.28 K8s clusters without any success. Additional capabilities shouldn't be needed. The only thing that might be a reason for this is if UID/GID translations are failing on your host for some reason. Have you tried running with:

controller:
  unprivileged: false
owanio1992 commented 10 months ago

I have thoroughly checked the Helm Chart as well as tried to reproduce on v1.29 and v1.28 K8s clusters without any success. Additional capabilities shouldn't be needed. The only thing that might be a reason for this is if UID/GID translations are failing on your host for some reason. Have you tried running with:

controller:
  unprivileged: false

thank you for reply yes, with unprivileged: false without error I fond I can't reproduce in minikube I will going to check it's OS problem ? my OS is ubuntu 22.04

dkorunic commented 10 months ago

@owanio1992 I would kindly ask if you can double check audit logs from your host (/var/log/audit/audit.log or similar) to identify if seccomp has been causing issues when running unprivileged.

owanio1992 commented 10 months ago

I get this audit log thank for help me check issue

time->Thu Feb  1 02:34:20 2024
type=PROCTITLE msg=audit(1706754860.515:827): proctitle="/bin/s6-overlay-preinit"
type=PATH msg=audit(1706754860.515:827): item=0 name="/var/run/s6" inode=7 dev=00:98 mode=040755 ouid=0 ogid=1000 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1706754860.515:827): cwd="/"
type=SYSCALL msg=audit(1706754860.515:827): arch=c000003e syscall=92 success=no exit=-1 a0=401b16 a1=3e8 a2=3e8 a3=0 items=1 ppid=18563 pid=18579 auid=4294967295 uid=1000 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=4294967295 comm="s6-overlay-prei" exe="/bin/s6-overlay-preinit" subj=cri-containerd.apparmor.d key="chown_operation_not_permitted"
dkorunic commented 10 months ago

Dear @owanio1992, now it's up to you to either modify matching AppArmor profile and add capability chown or move from enforcing to complain mode. In any case this doesn't relate to Helm Chart itself and I'll close the issue.