search

aws / eks-pod-identity-agent

Apache License 2.0
61 stars 9 forks source link

[EKS] [BAD-DECISION]: EKS Pod Identity agent daemonset mapped to node-port 80 #10

Open ChrisMcKee opened 3 months ago

ChrisMcKee commented 3 months ago

Copied over from https://github.com/aws/containers-roadmap/issues/2356

Community Note

Tell us about your request What do you want us to build?

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem.

We tried to install the eks-pod-identity-agent addon so that we could set the auth config to allow both options. The addon installs as a daemonset with HostNetwork set to true, pod permissions to map to the node, and a default port set to 80. The instant that the service started to install, all of our Haproxy ingress pods were evicted so that the identity service could map to port 80.

I'd love to know the rationale that went into choosing to map the node-port to what is literally the main http port; and then not to document how to change it to avoid collisions. Through all the documentation that mentions it the only warning is here https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html#pod-id-considerations and it's a note rather than informative. The majority of links go straight to https://docs.aws.amazon.com/eks/latest/userguide/pod-id-agent-setup.html which doesnt mention it at all.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: eks-pod-identity-agent
  namespace: kube-system
spec:
  template:    
    spec:
      containers:
        - name: eks-pod-identity-agent
          image: >-
            602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/eks-pod-identity-agent:0.1.6
          command:
            - /go-runner
            - /eks-pod-identity-agent
            - server
          args:
            - '--port'
            - '80'
            - '--cluster-name'
            - prod-eks-cluster
            - '--probe-port'
            - '2703'
          ports:
            - name: proxy
              containerPort: 80
              protocol: TCP
            - name: probes-port
              containerPort: 2703
              protocol: TCP
          securityContext:
            capabilities:
              add:
                - CAP_NET_BIND_SERVICE
      hostNetwork: true

Are you currently working around this issue? How are you currently solving this problem?

Uninstalled the Addon

ChrisMcKee commented 3 months ago

image

tanvp112 commented 3 months ago

This design will also break Fargate only cluster: https://github.com/terraform-aws-modules/terraform-aws-eks/issues/2850#issuecomment-1860808423

manusucerquia commented 1 month ago

This also breaks when enabling the addon in a cluster running Project Contour's Envoy proxy (they use the same ports in hostNetwork). Has anyone found a work around this issue or is there any news?

suhaspotluri commented 1 month ago

Also breaking our Traefik daemonset installation as we install Traefik with hostPort 80 and 443

joncolby commented 2 weeks ago

does eks-pod-identity-agent really require listening on port 80 ? what is exactly this port used for? if another port were configured, will everything still work?

taraspos commented 2 weeks ago

does eks-pod-identity-agent really require listening on port 80 ? what is exactly this port used for? if another port were configured, will everything still work?

https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html#pod-id-considerations

The EKS Pod Identity Agent uses the hostNetwork of the node and it uses port 80 and port 2703 on a link-local address on the node. This address is 169.254.170.23 for IPv4 and [fd00:ec2::23] for IPv6 clusters.

If you disable IPv6 addresses, or otherwise prevent localhost IPv6 IP addresses, the agent can't start. To start the agent on nodes that can't use IPv6, follow the steps in Disable IPv6 in the EKS Pod Identity Agent to disable the IPv6 configuration.

That's the port that actually being hit when pods requesting credentials via GET http://169.254.170.23:80

jvdadda commented 4 days ago

does eks-pod-identity-agent really require listening on port 80 ? what is exactly this port used for? if another port were configured, will everything still work?

With current config, yes, they can change it to target a different port but it seems not important enough to work on it yet. 🙄