alibaba / higress

🤖 AI Gateway | AI Native API Gateway
https://higress.io
Apache License 2.0
2.81k stars 462 forks source link

给gateway设置loadBalancerClass后报错may not change once set #1138

Open w7team opened 1 month ago

w7team commented 1 month ago

higress version: v1.4.1

在k3s上安了cilium后,需要给gateway设置loadBalancerClass,设置后报错,之前使用traefik设置都是正常的

{"error":"K8S error: Service \"higress-gateway\" is invalid: spec.loadBalancerClass: Invalid value: \"io.cilium\/node\": may not change once set"}

---
apiVersion: v1
kind: Service
metadata:
  name: higress-gateway
  namespace: higress-system
  uid: 04f1660f-4310-455b-855b-97af76d98604
  resourceVersion: "2284"
  creationTimestamp: "2024-07-19T09:41:15Z"
  labels:
    app: higress-gateway
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: higress-gateway
    app.kubernetes.io/version: 1.4.1
    helm.sh/chart: higress-core-1.4.1
    higress: higress-system-higress-gateway
  annotations:
    meta.helm.sh/release-name: higress
    meta.helm.sh/release-namespace: higress-system
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
spec:
  ports:
  - name: http2
    protocol: TCP
    port: 80
    targetPort: 80
    nodePort: 31341
  - name: https
    protocol: TCP
    port: 443
    targetPort: 443
    nodePort: 30233
  selector:
    app: higress-gateway
    higress: higress-system-higress-gateway
  clusterIP: 10.43.66.67
  clusterIPs:
  - 10.43.66.67
  type: LoadBalancer
  sessionAffinity: None
  externalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  allocateLoadBalancerNodePorts: true
  internalTrafficPolicy: Cluster
  loadBalancerClass: "io.cilium/node"
status:
  loadBalancer:
    ingress:
    - ip: 10.0.72.46
...
CH3CHO commented 1 month ago

这个应该是K8s的错吧,不允许修改loadBalancerClass。

w7team commented 1 month ago

之前使用traefik设置都是正常的,我用的k3s

CH3CHO commented 1 month ago

Invalid value: "io.cilium/node": may not change once set"

改之前和计划改到的 Service YAML 能贴一下吗?

w7team commented 1 month ago

image 我上面给贴上去了,只增加了这一行

w7team commented 1 month ago

我新建一个nginx,然后给nginx的service设置loadBalancerClass: "io.cilium/node"也没问题,感觉是higress的限制

CH3CHO commented 1 month ago

我新建一个nginx,然后给nginx的service设置loadBalancerClass: "io.cilium/node"也没问题,感觉是higress的限制

这个错就不是 Higress 报的。我怀疑会不会和 ipFamilyPolicy 或者 allocateLoadBalancerNodePorts 有关。你建的 nginx service 这些参数都和 Higress 的一样吗?

image

尤其是这里可以看到 LB 已经分配到 IP 了。

w7team commented 1 month ago

image 你看看,都一样。之前那个loadbalancerclass的pr是不是还没发到release

k3s默认是使用svlb(klipper-lb)转发的,如果安装了cilium,默认那个就不能用了,需要设置loadbalancerclass