search

truecharts / public

Community Helm Chart Repository
https://truecharts.org
GNU Affero General Public License v3.0
1.16k stars 615 forks source link

[clustertool] RC40 - IPs defined in clusterenv.yaml can be stolen by traefik dashboard #29131

Closed Qwal-1 closed 2 weeks ago

Qwal-1 commented 2 weeks ago

Chart Name

N.A.

Operating System

other

Deployment Method

FluxCD

Chart Version

N.A.

Kubernetes Events

N.A.

Chart Logs

N.A.

Chart Configuration

N.A.

Describe the bug

RC40 starting with an empty folder and default generated configuration.

When traefik is being installed, the traefik-dashboard automatically gets assigned an IP in the metallb range, this seems to always be the lowest IP available. If this happens to be the IP that has been set for the kubernetes-dashboard in clusterenv.yaml, the loadbalancer for kubernetes-dashboard will then be forced to deploy on a different ip.

clusterenv.yaml ``` ## The Following are required by ClusterTool and CANNOT be removed # Ensure VIP is different from all master IPs VIP: 192.168.0.250 ## Defines the MasterNode IP # Please be aware, we only support x86_64 for the first node, ARM will NOT work by default MASTER1IP: 192.168.0.253 # Defines the gateway for all nodes GATEWAY: 192.168.0.1 # Defines the ip range metallb is allowed to use METALLB_RANGE: 192.168.0.240-192.168.0.249 # IP Traefik will use for externally exposing ingress-resources (NOT its dashboard) TRAEFIK_IP: 192.168.0.242 # IP Blocky DNS will be exposed on BLOCKY_IP: 192.168.0.243 # Sets the Kubernetes Dashboard IP. Has to be within METALLB_RANGE and not in use DASHBOARD_IP: 192.168.0.240 ## Configures Traefik, ClusterIssuer and Blocky for said domain, using these credentials # domain is the base/highest level of your domain, so without subdomain, example: truecharts.org, not somesite.truecharts.org # To disable, please see the comments in their respective helm-values.yaml files! # When disabled, please use placeholder values instead. DOMAIN_0: "example.com" DOMAIN_0_EMAIL: "email@example.com" DOMAIN_0_CLOUDFLARE_TOKEN: "3x4mpl3t0k3n" # Used to automatically generate a sshkey-pair for FluxCD # Has to start with ssh:// # Please make sure to NOT contain any special characters in the URL except "-" GITHUB_REPOSITORY: ssh://git@github.com/Qwal-1/talos-config2.git ## Uncomment and adapt to automatically setup your dockerhub credentials ## This should prevent much of the potential ratelimits # DOCKERHUB_USER: dockerhub-username # DOCKERHUB_PASSWORD: dockerhub-password ## DO NOT ALTER PODNET: 172.16.0.0/16 SVCNET: 172.17.0.0/16 ```
kubectl describe service/traefik -n traefik ``` $ kubectl describe service/traefik -n traefik Name: traefik Namespace: traefik Labels: app=traefik-29.4.0 app.kubernetes.io/instance=traefik app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=traefik app.kubernetes.io/version=3.2.0 helm-revision=2 helm.sh/chart=traefik-29.4.0 helm.toolkit.fluxcd.io/name=traefik helm.toolkit.fluxcd.io/namespace=traefik release=traefik service.name=main Annotations: meta.helm.sh/release-name: traefik meta.helm.sh/release-namespace: traefik metallb.universe.tf/allow-shared-ip: traefik metallb.universe.tf/ip-allocated-from-pool: main Selector: app.kubernetes.io/instance=traefik,app.kubernetes.io/name=traefik,pod.name=main Type: LoadBalancer IP Family Policy: SingleStack IP Families: IPv4 IP: 172.17.126.162 IPs: 172.17.126.162 LoadBalancer Ingress: 192.168.0.240 Port: main 9000/TCP TargetPort: 9000/TCP Endpoints: 172.16.0.110:9000,172.16.0.189:9000 Session Affinity: None External Traffic Policy: Cluster Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal IPAllocated 30m metallb-controller Assigned IP ["192.168.0.240"] Normal nodeAssigned 18m (x14 over 30m) metallb-speaker announcing from node "k8s-control-1" with protocol "layer2" Normal nodeAssigned 9m53s (x2 over 9m53s) metallb-speaker announcing from node "k8s-control-1" with protocol "layer2" ```
kubectl describe service/kubernetes-dashboard -n kubernetes-dashboard ``` $ kubectl describe service/kubernetes-dashboard -n kubernetes-dashboard Name: kubernetes-dashboard Namespace: kubernetes-dashboard Labels: app=kubernetes-dashboard-1.10.0 app.kubernetes.io/instance=kubernetes-dashboard app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=kubernetes-dashboard app.kubernetes.io/version=latest helm-revision=2 helm.sh/chart=kubernetes-dashboard-1.10.0 helm.toolkit.fluxcd.io/name=kubernetes-dashboard helm.toolkit.fluxcd.io/namespace=kubernetes-dashboard release=kubernetes-dashboard service.name=main Annotations: meta.helm.sh/release-name: kubernetes-dashboard meta.helm.sh/release-namespace: kubernetes-dashboard metallb.universe.tf/allow-shared-ip: kubernetes-dashboard metallb.universe.tf/ip-allocated-from-pool: main metallb.universe.tf/loadBalancerIPs: 192.168.0.241 Selector: app.kubernetes.io/instance=kubernetes-dashboard,app.kubernetes.io/name=kubernetes-dashboard,pod.name=main Type: LoadBalancer IP Family Policy: SingleStack IP Families: IPv4 IP: 172.17.29.24 IPs: 172.17.29.24 LoadBalancer Ingress: 192.168.0.241 Port: main 80/TCP TargetPort: 3000/TCP Endpoints: 172.16.0.214:3000 Session Affinity: None External Traffic Policy: Cluster Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning AllocationFailed 32m metallb-controller Failed to allocate IP for "kubernetes-dashboard/kubernetes-dashboard": can't change sharing key for "kubernetes-dashboard/kubernetes-dashboard", address also in use by traefik/traefik Normal IPAllocated 29m metallb-controller Assigned IP ["192.168.0.241"] Normal nodeAssigned 20m (x4 over 29m) metallb-speaker announcing from node "k8s-control-1" with protocol "layer2" Normal nodeAssigned 11m metallb-speaker announcing from node "k8s-control-1" with protocol "layer2" ```

To Reproduce

  1. start with empty folder
  2. clustertool init
  3. fill out clusterenv.yaml and make sure to assign the lowest still available IP in the metallb_range to DASHBOARD_IP
  4. clustertool init
  5. clustertool genconfig
  6. clustertool talos apply

Expected Behavior

The traefik dashboard should somehow be prevented from being assigned any IP which is set in the clusterenv.yaml or should also have to be manually defined.

Screenshots

N.A.

Additional Context

N.A.

I've read and agree with the following

mooneydude commented 2 weeks ago

Same here and bug has been occurring since before RC1. Only occurs with traefik and when gap in metallb range assignment

PrivatePuffin commented 2 weeks ago

Same here and bug has been occurring since before RC1. Only occurs with traefik and when gap in metallb range assignment

Thats complete bullshit. Because traefik didn't even exist on RC1.

PrivatePuffin commented 2 weeks ago

When traefik is being installed, the traefik-dashboard automatically gets assigned an IP in the metallb range

It does not though. We set autoAssign: false on metallb-config.

Which disables this behavior.

PrivatePuffin commented 2 weeks ago

None-the-less, I've added an extra precaution.

Just to be clear though: The traefik dashboard should somehow be prevented from being assigned any IP which is set in the clusterenv.yaml

Thats not what we do, we are not writhing helm-charts for clustertool, nor will they ever contain special magic-sauce for it. Thats not within our sphere of influence.

mooneydude commented 2 weeks ago

Same here and bug has been occurring since before RC1. Only occurs with traefik and when gap in metallb range assignment

Thats complete bullshit. Because traefik didn't even exist on RC1.

Correct. The bug did occur once traefik was installed after the initial bootstrap. I should've been more clear and only stated this in effort to help with back tracing.

PrivatePuffin commented 2 weeks ago

Correct. The bug did occur once traefik was installed after the initial bootstrap.

Okey, but your personal modifications aren't relevant to clustertool bug tracing tbh. It only made it harder to follow the issue, instead of making it easier.

The only thing we need for bug tracing is the required info.