search

haproxytech / kubernetes-ingress

HAProxy Kubernetes Ingress Controller
https://www.haproxy.com/documentation/kubernetes/
Apache License 2.0
719 stars 202 forks source link

The backend remains down after recreating the pod #259

Closed bsctl closed 3 years ago

bsctl commented 3 years ago

The backend remains down after recreating the pod

We have HAProxy Ingress Controller 1.4.7 deployed on Kubernetes 1.19 as daemonset and hostNetwork. It has been deployed with Helm Chart 1.6.1.

Observed Behaviour

We exposed a pod by the Ingress and everything is working as expected. When we delete the pod, it gets created again by its deployment controller. However, the HAProxy configuration is not updated correctly and the pod is no more reachable from the external (503 Service Unavailable).

Steps we followed

  1. Expose the pod with an Ingress like this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: console
  namespace: console-system
  annotations:
    ingress.kubernetes.io: ssl-redirect
    kubernetes.io/ingress.class: caas
spec:
  rules:
  - host: gui.lab
    http:
      paths:
      - backend:
          serviceName: console
          servicePort: console
        path: /
  - host: cli.lab
    http:
      paths:
      - backend:
          serviceName: console
          servicePort: sidecar
        path: /
  1. Check the HAProxy backend configuration 
...
backend console-system-console-console                                                                                                                  
  mode http                                                                                                                                             
  balance leastconn                                                                                                                                     
  option forwardfor                                                                                                                                     
backend console-system-console-console                                                                                                                  
  mode http                                                                                                                                             
  balance leastconn                                                                                                                                     
  option forwardfor                                                                                                                                     
  server SRV_1Cb4u 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_vOjoj 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_9LMsT 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_df3bo 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_L1zJ7 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_9mLKh 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_bCCdP 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_LEW8r 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_dSduU 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_C47vr 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_n7Ku4 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_Vg7Ux 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_07ztP 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_Xrp7U 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_AtCDj 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_doNdn 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_vkBX7 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_wLxNI 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_QsPyp 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_NL8y6 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_epI4D 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_atTka 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_20zQl 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_vcatO 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_nOpLu 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_JlGWI 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_kIeHW 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_Lw01P 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_GgHJu 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_ykVux 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_ifT2z 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_kwDK2 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_q5ve0 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_uuna0 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_zfpDK 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_SPOrh 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_Mpa2s 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_8uQhs 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_WOeUx 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_DFelg 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_hKw79 10.38.140.165:9000 check weight 128
  1. Check the exposed pod
NAME                      READY   STATUS    RESTARTS   AGE   IP              NODE               NOMINATED NODE   READINESS GATES
console-d5568bc7c-skl2w   2/2     Running   0          28m   10.38.140.165   infra01   <none>           <none>
  1. Check if pod is reachable from the Ingress Controller
curl -k https://gui.lab
<html><body><h1>It works!</h1>
</body></html>
  1. Delete the pod, it gets recreated with a different IP address
NAME                      READY   STATUS    RESTARTS   AGE   IP              NODE               NOMINATED NODE   READINESS GATES
console-d5568bc7c-85wkd   2/2     Running   0          20s   10.38.140.166   infra01   <none>           <none>
  1. Tail logs from the HAProxy Ingress Controller
2020/12/16 15:09:20 TRACE   controller.go:106 HAProxy config sync started
2020/12/16 15:09:20 TRACE   controller.go:217 HAProxy config sync terminated
2020/12/16 15:09:23 DEBUG   events.go:375 server 'console-system-console-sidecar/SRV_n7Ku4' changed status to ready
2020/12/16 15:09:23 DEBUG   events.go:375 server 'console-system-console-sidecar/SRV_hKw79' changed status to maint
2020/12/16 15:09:23 DEBUG   events.go:375 server 'console-system-console-sidecar/SRV_n7Ku4' changed status to ready
  1. Check again the HAProxy configuration:
  backend console-system-console-consolebackend console-system-console-consolev
  mode http                                                                                                                                             
  balance leastconn                                                                                                                                     
  option forwardfor                                                                                                                                     
  server SRV_1Cb4u 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_vOjoj 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_9LMsT 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_df3bo 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_L1zJ7 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_9mLKh 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_bCCdP 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_LEW8r 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_dSduU 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_C47vr 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_n7Ku4 127.0.0.1:9000 disabled check weight 128       <--- see here                                                                                      
  server SRV_Vg7Ux 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_07ztP 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_Xrp7U 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_AtCDj 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_doNdn 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_vkBX7 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_wLxNI 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_QsPyp 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_NL8y6 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_epI4D 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_atTka 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_20zQl 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_vcatO 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_nOpLu 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_JlGWI 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_kIeHW 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_Lw01P 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_GgHJu 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_ykVux 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_ifT2z 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_kwDK2 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_q5ve0 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_uuna0 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_zfpDK 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_SPOrh 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_Mpa2s 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_8uQhs 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_WOeUx 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_DFelg 127.0.0.1:9000 disabled check weight 128                                                                                             
  server SRV_hKw79 10.38.140.165:9000 check weight 128                 <--- see here                                                                                 
  server SRV_gEkdI 127.0.0.1:9000 disabled check weight 128
  1. Check if pod is reachable from the Ingress Controller
curl -k https://gui.lab
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
bsctl commented 3 years ago

This is the values.yaml we used to deploy HAProxy Ingress Controller

controller:
  affinity: {}
  config:
    forwarded-for: "true"
    load-balance: "leastconn"
    ssl-redirect: "true"
  containerPort:
    http: 80
    https: 443
    stat: 1024
  daemonset:
    hostPorts:
      http: 80
      https: 443
      stat: 1024
    useHostNetwork: true
    useHostPort: false
  defaultTLSSecret:
    enabled: true
    secret: ingress-certs
  dnsConfig: {}
  dnsPolicy: ClusterFirst
  extraArgs: []
  extraEnvs: []
  image:
    pullPolicy: IfNotPresent
    repository: haproxytech/kubernetes-ingress
    tag: '{{ .Chart.AppVersion }}'
  imageCredentials:
    password: null
    registry: null
    username: null
  ingressClass: caas
  initContainers: []
  kind: DaemonSet
  livenessProbe:
    failureThreshold: 3
    initialDelaySeconds: 0
    path: /healthz
    periodSeconds: 10
    port: 1042
    scheme: HTTP
    successThreshold: 1
    timeoutSeconds: 1
  logging:
    level: trace
    traffic:
      syslog-server:
        address: stdout
        format: raw
        facility: daemon
  name: controller
  nodeSelector:
    node-role.kubernetes.io/infra: ""
  podAnnotations: {}
  podLabels: {}
  priorityClassName: ""
  publishService:
    enabled: false
    pathOverride: ""
  readinessProbe:
    failureThreshold: 3
    initialDelaySeconds: 0
    path: /healthz
    periodSeconds: 10
    port: 1042
    scheme: HTTP
    successThreshold: 1
    timeoutSeconds: 1
  replicaCount: 2
  resources:
    requests:
      cpu: 100m
      memory: 64Mi
  service:
    annotations: {}
    enablePorts:
      http: true
      https: true
      stat: true
    enabled: false
    externalIPs: []
    healthCheckNodePort: 0
    labels: {}
    loadBalancerIP: ""
    loadBalancerSourceRanges: []
    nodePorts: {}
    ports:
      http: 80
      https: 443
      stat: 1024
    targetPorts:
      http: http
      https: https
      stat: stat
    tcpPorts: []
    type: NodePort
  strategy: {}
  tolerations: []
defaultBackend:
  affinity: {}
  containerPort: 8080
  extraEnvs: []
  image:
    pullPolicy: IfNotPresent
    repository: k8s.gcr.io/defaultbackend-amd64
    runAsUser: 65534
    tag: 1.5
  name: default-backend
  nodeSelector:
    node-role.kubernetes.io/infra: ""
  podAnnotations: {}
  podLabels: {}
  priorityClassName: ""
  replicaCount: 1
  resources:
    requests:
      cpu: 10m
      memory: 16Mi
  service:
    port: 8080
  serviceAccount:
    create: true
  tolerations: []
podSecurityPolicy:
  annotations: {}
  enabled: enable
rbac:
  create: true
serviceAccount:
  create: true
  name: kubernetes-ingress
Mo3m3n commented 3 years ago

Hi There was a number of adjustments to endpoints handling in latest version: 1.4.10 can give it a try and let us know if you still experience this issue ?

bsctl commented 3 years ago

And this is the stat page when we hit the issue:

image

bsctl commented 3 years ago

Hi There was a number of adjustments to endpoints handling in latest version: 1.4.10 can give it a try and let us know if you still experience this issue ?

@Mo3m3n sure

bsctl commented 3 years ago

@Mo3m3n with 1.4.10, logs are a bit more sparkling but we hit the same behaviour.

Is there any other check we can do in order to help the analysis? 

2020/12/16 16:40:20 TRACE   controller.go:106 HAProxy config sync started
2020/12/16 16:40:20 TRACE   controller.go:208 HAProxy config sync terminated
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_CGPSB' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_NERUd' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_Jcg2N' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_yhSsm' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_D3gX5' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_PrGRh' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_7Qtqy' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_wZk2A' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_XPDNg' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_aTt85' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_DYWUE' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_ZVlwi' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_u0XWL' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_vv5xG' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_TeaV9' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_5fyme' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_2PP5S' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_Q4l1d' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_gEj2E' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:233 server 'console-system-console-sidecar/SRV_qnbkD' changed status to ready
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_MplJ2' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_P52a3' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_hK0nZ' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_KfRtd' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_kOmVU' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_CaHAG' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:233 server 'console-system-console-sidecar/SRV_3E9LW' changed status to ready
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_y8ZU0' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_eXeeY' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_5PiSN' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_IOGtd' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_94IRB' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_2CfnQ' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_EWH1j' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_3KCZx' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_tfl9q' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_SWOQc' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_nHuCu' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_LR7Nk' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_halWl' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_YBgPt' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_6mOUz' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_YWuAn' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_1Rbn2' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_VyDUN' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_B9ssd' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_s3uWF' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_O6aE4' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_fdnA1' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_SWl8j' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_moJCX' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_wCdsI' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_cS7x5' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_uLWVx' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_iL6W6' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_SoNTm' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_nyfnD' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_wpAJz' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_ZUGuT' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_FRUJB' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_duRYu' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_41Ipo' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_4Byv1' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_tUK5V' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_w7TKM' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_pvV6c' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_RpLZt' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_PgpOY' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_gCllB' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_NIckV' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_Mhmp6' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_sKdQm' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_Qdiuw' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_vWZuR' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_tvme4' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_Efcch' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_NqPYK' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_A73tw' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_nI2xB' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_uAOPu' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_rbqQF' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_yO7XH' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_Trhbd' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_bbHaJ' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_u0XWL' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_vv5xG' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_TeaV9' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_5fyme' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_2PP5S' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_Q4l1d' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_gEj2E' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_3E9LW' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:233 server 'console-system-console-sidecar/SRV_qnbkD' changed status to ready
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_MplJ2' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_P52a3' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_hK0nZ' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_KfRtd' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_kOmVU' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_CaHAG' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_y8ZU0' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_eXeeY' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_5PiSN' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_IOGtd' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_94IRB' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_2CfnQ' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_EWH1j' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_3KCZx' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_tfl9q' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_SWOQc' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_nHuCu' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_LR7Nk' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_halWl' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_YBgPt' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_6mOUz' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_CGPSB' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_NERUd' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_Jcg2N' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_yhSsm' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_D3gX5' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_PrGRh' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_7Qtqy' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_wZk2A' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_XPDNg' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_aTt85' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_DYWUE' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-sidecar/SRV_ZVlwi' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_fdnA1' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_SWl8j' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_moJCX' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_wCdsI' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_cS7x5' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_B9ssd' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_s3uWF' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_O6aE4' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_uLWVx' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_iL6W6' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_SoNTm' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_nyfnD' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_duRYu' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_41Ipo' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_4Byv1' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_tUK5V' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_w7TKM' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_wpAJz' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_ZUGuT' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_FRUJB' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_gCllB' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_NIckV' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_Mhmp6' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_sKdQm' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_pvV6c' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_RpLZt' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_PgpOY' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_Qdiuw' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_vWZuR' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_A73tw' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_nI2xB' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_uAOPu' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_tvme4' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_Efcch' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_NqPYK' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_bbHaJ' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_rbqQF' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_yO7XH' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_Trhbd' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_YWuAn' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_1Rbn2' changed status to maint
2020/12/16 16:40:27 DEBUG   endpoints.go:229 server 'console-system-console-console/SRV_VyDUN' changed status to maint
2020/12/16 16:40:32 TRACE   controller.go:106 HAProxy config sync started
2020/12/16 16:40:32 ERROR   service.go:170 service 'reply-server' does not exist
2020/12/16 16:40:32 ERROR   service.go:170 service 'jenkins' does not exist
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_Qdiuw': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_Qdiuw': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_Qdiuw'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_vWZuR': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_vWZuR': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_vWZuR'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_NqPYK': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_NqPYK': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_NqPYK'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_A73tw': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_A73tw': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_A73tw'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_nI2xB': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_nI2xB': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_nI2xB'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_uAOPu': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_uAOPu': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_uAOPu'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_tvme4': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_tvme4': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_tvme4'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_Efcch': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_Efcch': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_Efcch'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_Trhbd': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_Trhbd': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_Trhbd'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_bbHaJ': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_bbHaJ': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_bbHaJ'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_rbqQF': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_rbqQF': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_rbqQF'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_yO7XH': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_yO7XH': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_yO7XH'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_VyDUN': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_VyDUN': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_VyDUN'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_YWuAn': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_YWuAn': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_YWuAn'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_1Rbn2': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_1Rbn2': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_1Rbn2'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_O6aE4': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_O6aE4': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_O6aE4'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_fdnA1': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_fdnA1': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_fdnA1'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_SWl8j': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_SWl8j': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_SWl8j'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_moJCX': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_moJCX': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_moJCX'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_wCdsI': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_wCdsI': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_wCdsI'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_cS7x5': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_cS7x5': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_cS7x5'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_B9ssd': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_B9ssd': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_B9ssd'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_s3uWF': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_s3uWF': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_s3uWF'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_uLWVx': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_uLWVx': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_uLWVx'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_nyfnD': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_nyfnD': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_nyfnD'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_iL6W6': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_iL6W6': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_iL6W6'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_SoNTm': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_SoNTm': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_SoNTm'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_FRUJB': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_FRUJB': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_FRUJB'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_duRYu': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_duRYu': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_duRYu'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_41Ipo': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_41Ipo': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_41Ipo'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_4Byv1': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_4Byv1': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_4Byv1'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_tUK5V': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_tUK5V': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_tUK5V'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_w7TKM': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_w7TKM': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_w7TKM'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_wpAJz': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_wpAJz': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_wpAJz'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_ZUGuT': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_ZUGuT': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_ZUGuT'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_PgpOY': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_PgpOY': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_PgpOY'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_gCllB': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_gCllB': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_gCllB'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_NIckV': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_NIckV': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_NIckV'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_Mhmp6': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_Mhmp6': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_Mhmp6'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_sKdQm': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_sKdQm': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_sKdQm'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_pvV6c': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_pvV6c': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_pvV6c'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_RpLZt': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_RpLZt': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-console/SRV_RpLZt'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_5fyme': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_5fyme': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_5fyme'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_2PP5S': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_2PP5S': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_2PP5S'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_Q4l1d': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_Q4l1d': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_Q4l1d'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_gEj2E': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_gEj2E': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_gEj2E'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_KfRtd': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_KfRtd': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_KfRtd'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_kOmVU': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_kOmVU': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_kOmVU'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_CaHAG': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_CaHAG': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_CaHAG'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_3E9LW': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_3E9LW': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_3E9LW'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_qnbkD': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_qnbkD': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_qnbkD'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_MplJ2': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_MplJ2': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_MplJ2'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_P52a3': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_P52a3': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_P52a3'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_hK0nZ': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_hK0nZ': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_hK0nZ'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_94IRB': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_94IRB': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_94IRB'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_y8ZU0': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_y8ZU0': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_y8ZU0'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_eXeeY': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_eXeeY': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_eXeeY'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_5PiSN': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_5PiSN': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_5PiSN'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_IOGtd': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_IOGtd': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_IOGtd'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_2CfnQ': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_2CfnQ': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_2CfnQ'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_EWH1j': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_EWH1j': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_EWH1j'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_3KCZx': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_3KCZx': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_3KCZx'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_tfl9q': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_tfl9q': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_tfl9q'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_YBgPt': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_YBgPt': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_YBgPt'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_6mOUz': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_6mOUz': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_6mOUz'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_SWOQc': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_SWOQc': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_SWOQc'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_nHuCu': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_nHuCu': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_nHuCu'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_LR7Nk': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_LR7Nk': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_LR7Nk'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_halWl': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_halWl': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_halWl'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_D3gX5': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_D3gX5': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_D3gX5'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_PrGRh': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_PrGRh': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_PrGRh'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_CGPSB': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_CGPSB': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_CGPSB'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_NERUd': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_NERUd': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_NERUd'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_Jcg2N': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_Jcg2N': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_Jcg2N'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_yhSsm': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_yhSsm': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_yhSsm'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_DYWUE': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_DYWUE': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_DYWUE'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_ZVlwi': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_ZVlwi': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_ZVlwi'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_7Qtqy': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_7Qtqy': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_7Qtqy'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_wZk2A': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_wZk2A': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_wZk2A'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_XPDNg': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_XPDNg': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_XPDNg'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_aTt85': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_aTt85': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_aTt85'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_u0XWL': Configuring 'check' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_u0XWL': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_u0XWL'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_vv5xG': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_vv5xG': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_vv5xG'
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_TeaV9': Configuring 'server-ssl' annotation
2020/12/16 16:40:32 TRACE   backend-annotations.go:138 Server 'SRV_TeaV9': Configuring 'check' annotation
2020/12/16 16:40:32 DEBUG   endpoints.go:120 Updating server 'console-system-console-sidecar/SRV_TeaV9'
2020/12/16 16:40:32 TRACE   controller.go:208 HAProxy config sync terminated
bsctl commented 3 years ago

An this is the frontend configuration of HAProxy, hope this help.

/ # cat /etc/haproxy/haproxy.cfg 
# _version=3
# HAProxy Technologies
# https://www.haproxy.com/
# this file is not meant to be changed directly
# it is under haproxy ingress controller management

global 
  daemon
  master-worker
  pidfile /var/run/haproxy.pid
  stats socket /var/run/haproxy-runtime-api.sock level admin expose-fd listeners
  stats timeout 1m
  tune.ssl.default-dh-param 2048
  ssl-default-bind-options no-sslv3 no-tls-tickets no-tlsv10
  ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK
  hard-stop-after 1h
  server-state-file global
  server-state-base /var/state/haproxy/

defaults 
  log global
  log-format '%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs "%HM %[var(txn.base)] %HV"'
  option redispatch
  option dontlognull
  option http-keep-alive
  timeout http-request 5s
  timeout connect 5s
  timeout client 50s
  timeout queue 5s
  timeout server 50s
  timeout tunnel 1h
  timeout http-keep-alive 1m
  load-server-state-from-file global

frontend healthz 
  mode http
  bind 0.0.0.0:1042 name healtz_1
  monitor-uri /healthz
  option dontlog-normal

frontend http 
  mode http
  bind 0.0.0.0:80 name bind_1
  bind :::80 v4v6 name bind_2
  http-request set-var(txn.host) req.hdr(Host),field(1,:),lower
  http-request set-var(txn.path) path,lower
  http-request set-var(txn.base) base
  http-request deny deny_status 403 if { var(txn.host),concat(,txn.path) -m beg -f /etc/haproxy/maps/13277975235645142560.lst } !{ src -f /etc/haproxy/maps/16962593408631331576.lst } or { var(txn.host) -f /etc/haproxy/maps/13277975235645142560.lst } !{ src -f /etc/haproxy/maps/16962593408631331576.lst } or { var(txn.path) -m beg -f /etc/haproxy/maps/13277975235645142560.lst } !{ src -f /etc/haproxy/maps/16962593408631331576.lst }
  http-request redirect scheme https code 302 if { var(txn.host),concat(,txn.path) -m beg -f /etc/haproxy/maps/16657147599902430202.lst } !{ ssl_fc } or { var(txn.host) -f /etc/haproxy/maps/16657147599902430202.lst } !{ ssl_fc } or { var(txn.path) -m beg -f /etc/haproxy/maps/16657147599902430202.lst } !{ ssl_fc }
  use_backend console-system-console-console if { var(txn.host) gui.lab } { var(txn.path) -m beg / }
  use_backend console-system-console-sidecar if { var(txn.host) cli.lab } { var(txn.path) -m beg / }
  default_backend ingress-system-ingress-kubernetes-ingress-default-backend-8080

frontend https 
  mode http
  bind 127.0.0.1:8443 name bind_1 crt /etc/haproxy/certs ssl alpn h2,http/1.1 accept-proxy
  bind 127.0.0.1:8443 name bind_2 crt /etc/haproxy/certs ssl v4v6 alpn h2,http/1.1 accept-proxy
  http-request set-var(txn.host) req.hdr(Host),field(1,:),lower
  http-request set-var(txn.path) path,lower
  http-request set-var(txn.base) base
  http-request deny deny_status 403 if { var(txn.host),concat(,txn.path) -m beg -f /etc/haproxy/maps/13277975235645142560.lst } !{ src -f /etc/haproxy/maps/16962593408631331576.lst } or { var(txn.host) -f /etc/haproxy/maps/13277975235645142560.lst } !{ src -f /etc/haproxy/maps/16962593408631331576.lst } or { var(txn.path) -m beg -f /etc/haproxy/maps/13277975235645142560.lst } !{ src -f /etc/haproxy/maps/16962593408631331576.lst }
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  use_backend console-system-console-console if { var(txn.host) gui.lab } { var(txn.path) -m beg / }
  use_backend console-system-console-sidecar if { var(txn.host) cli.lab } { var(txn.path) -m beg / }
  default_backend ingress-system-ingress-kubernetes-ingress-default-backend-8080

frontend ssl 
  mode tcp
  bind 0.0.0.0:443 name bind_1
  bind :::443 name bind_2 v4v6
  log-format '%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %[var(sess.sni)]'
  tcp-request content reject if { req_ssl_sni -f /etc/haproxy/maps/13277975235645142560.lst } !{ src -f /etc/haproxy/maps/16962593408631331576.lst }
  tcp-request inspect-delay 50000
  tcp-request content set-var(sess.sni) req_ssl_sni
  tcp-request content accept if { req_ssl_hello_type 1 }
  use_backend auth-system-keycloak-8443 if { req_ssl_sni -i sso.lab }
  default_backend https

frontend stats 
  mode http
  bind *:1024
  stats enable
  stats uri /
  stats refresh 10s
  http-request use-service prometheus-exporter if { path /metrics }
Mo3m3n commented 3 years ago

Many thanks @bsctl I am finally able to reproduce this. I am working on providing a hotfix

bsctl commented 3 years ago

Please, consider on the same setup, we have an additional SSL passthrough ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/ssl-passthrough: "true"
    kubernetes.io/ingress.class: caas
    meta.helm.sh/release-name: keycloak
    meta.helm.sh/release-namespace: auth-system
  labels:
    app.kubernetes.io/instance: keycloak
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: keycloak
    app.kubernetes.io/version: 11.0.0
    helm.sh/chart: keycloak-0.1.1
  name: keycloak
  namespace: auth-system
spec:
  rules:
  - host: sso.lab
    http:
      paths:
      - backend:
          serviceName: keycloak
          servicePort: 8443
        path: /

and a couple of subpath ingresses:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: caas
    meta.helm.sh/release-name: grafana
    meta.helm.sh/release-namespace: monitoring-system
  labels:
    app.kubernetes.io/instance: grafana
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: grafana
    app.kubernetes.io/version: 7.2.0
    helm.sh/chart: grafana-5.7.0
  name: grafana
  namespace: monitoring-system
spec:
  rules:
  - host: gui.lab
    http:
      paths:
      - backend:
          serviceName: grafana
          servicePort: 80
        path: /grafana
Mo3m3n commented 3 years ago

@bsctl I think this is now fixed with https://github.com/haproxytech/kubernetes-ingress/commit/bf271af22beb35deef0059bb10fcd3e0965b1cb5 can you give haproxytech/kubernetes-ingress:1.4-dev and let us know if it fixes your problem ?

bsctl commented 3 years ago

@Mo3m3n we tested the fix on the same setup and following the same steps. It works! Thanks for the fantastic job!

Wonder to understand the root cause of the issue. This looks a pretty simple use case for an Ingress Controller and we're surprised why it has not been reported before. Is this issue triggered by some our specific configuration or setup? Just my curiosity.

Mo3m3n commented 3 years ago

The problem happens when endpoints scales down to zero and then up again. When receiving empty addresses the controller used not to sync/clear out correctly the config which triggered this bug. It seems more likely that this is a regression introduced at some point rather than a problem occurring for all versions. The fix is available on 1.4.11