search

kakao / network-node-manager

network-node-manager is a kubernetes controller that controls the network configuration of a node to resolve network issues of kubernetes.
Apache License 2.0
108 stars 20 forks source link

Dual-stack services fail #10

Closed bootc closed 3 years ago

bootc commented 3 years ago

If I create a dual-stack LoadBalancer service (say, using Nordix/assign-lb-ip), network-node-manager is confused by this and tries to add IPv4 iptables rules for the IPv6 external IP:

2021-03-18T23:27:07.341Z    INFO    controllers.Service.reconcile   create iptables rules   {"service": "bootc-system/ingress-nginx-controller", "externalIP": "192.0.2.52", "clusterIP": "10.43.150.86"}
2021-03-18T23:27:07.441Z    INFO    controllers.Service.reconcile   create iptables rules   {"service": "bootc-system/ingress-nginx-controller", "externalIP": "2001:db8::5", "clusterIP": "10.43.150.86"}
2021-03-18T23:27:07.445Z    ERROR   controllers.Service.reconcile   iptables v1.8.3 (legacy): host/network `2001:db8::5' not found
Try `iptables -h' or 'iptables --help' for more information.
    {"service": "bootc-system/ingress-nginx-controller", "error": "exit status 2"}
github.com/go-logr/zapr.(*zapLogger).Error
    /go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
github.com/kakao/network-node-manager/pkg/rules.CreateRulesExternalCluster
    /workspace/pkg/rules/rule_external_cluster.go:314
github.com/kakao/network-node-manager/controllers.(*ServiceReconciler).Reconcile
    /workspace/controllers/service_controller.go:208
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:256
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
    /go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
    /go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:88
2021-03-18T23:27:07.445Z    ERROR   controller-runtime.controller   Reconciler error    {"controller": "service", "request": "bootc-system/ingress-nginx-controller", "error": "exit status 2"}
github.com/go-logr/zapr.(*zapLogger).Error
    /go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:258
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
    /go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
    /go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:88

The service that triggers the above looks like this (redacted for clarity):

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx-controller
  namespace: bootc-system
spec:
  clusterIP: 10.43.150.86
  clusterIPs:
  - 10.43.150.86
  - 2001:db8:ff::edd5
  externalIPs:
  - 192.0.2.52
  - 2001:db8::5
  externalTrafficPolicy: Local
  ipFamilies:
  - IPv4
  - IPv6
  ipFamilyPolicy: RequireDualStack
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 192.0.2.52
    - ip: 2001:db8::5

It seems like the controller needs to look at the clusterIPs field rather than the singular clusterIP field and match the address family for the externalIP it is looking at.

ssup2 commented 3 years ago

@bootc Yes. Now network-node-manager doesn't support dual-stack in a service. I will work for this problem. Thanks :)

bootc commented 3 years ago

I actually have a work-in-progress branch for this, if you can hang on a few hours.

bootc commented 3 years ago

I have a Docker image with the above available at bootc/network-node-manager:issue-10.