Any configuration error breaks synchronization loop

[rochr01]

NOTE: GitHub issues are reserved for bug reports only. For anything else, please join the conversation in Kong Nation https://discuss.konghq.com/c/kubernetes.

---

Summary

When synching configuration any configuration error stops ingress-controller from processing rest of queued changes. This is PITA when running a cluster with multiple applications and multiple development teams working on them: one configuration mistake breaks ingress-controller for everyone.

Kong Ingress controller version 0.3.0

Kong or Kong Enterprise version 1.1.2-centos

Kubernetes version

lient Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"202 0-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.7-rancher1", GitCommit:"96573de545affa0a5db720a830b5d2bdf6e80b3b", GitTreeState:"clean", Buil dDate:"2019-03-27T02:08:41Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

Environment

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened

Error in one Ingress configuration prevents ingress-controller from processing other Ingresses.

Expected behavior

Error in an Ingress should not affect other Ingresses.

Steps To Reproduce

1. Create a faulty Ingress object (kong-status-ingress). For example put double slash in "path"field, resulting in "path cannot have empty segments" error

apiVersion: configuration.konghq.com/v1
kind: KongIngress
metadata:
  name: kong-http-config
  namespace: kong
route:
  protocols:
  - http
  strip_path: false
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    configuration.konghq.com: kong-http-config
    kubernetes.io/ingress.class: kong
  name: kong-status-ingress
  namespace: kong
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: kong-admin
          servicePort: kong-admin
        path: //status

2. Create a second Ingress object (kong-metrics-ingress), with correct configuration. Since ingress controller is stuck on creating kong-status-ingress, kong-metrics-ingress never gets created

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    configuration.konghq.com: kong-http-config
    kubernetes.io/ingress.class: kong
  name: kong-metrics-ingress
  namespace: kong
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: kong-admin
          servicePort: kong-admin
        path: /metrics

[hbagdi]

This has been improved quite a bit in the newer versions. 0.3.0 is quite old, I'd highly recommend upgrading to a newer version to solve this and many other synchronization issues.

[rochr01]

@hbagdi I tried the latest version 0.7.1 and I have about a dozen potential bug reports already :) But maybe just two questions for now:

1. why does your ingress controller pick up all possible resources in cluster? I mean Secrets, even though they are not referenced by any KongConsumer, or Services, even those not referenced by any Ingress, and not marked with any KongIngress annotation. This generates a lot of noise in logs.

2. While "normal" services are created just fine, I have an issue with exposing "/status" endpoint from Kong admin api (which in previous version I used as a heathcheck). No matter what I do, even when I create this ingress on a fresh Kong and Ingress Controller installation, I always get "err inserting target into state: entity already exists".

Sorry if those comments seem unrelated to the bug I filed, but I'm just following your suggestion and trying to upgrade :)

UPDATE: it is confirmed - this issue also affects the latest version, 0.7.1

[hbagdi]

I tried the latest version 0.7.1 and I have about a dozen potential bug reports already :)

Please open Github issues around whatever you see and we will try to respond by the earliest possible.

why does your ingress controller pick up all possible resources in cluster? I mean Secrets, even though they are not referenced by any KongConsumer, or Services, even those not referenced by any Ingress, and not marked with any KongIngress annotation. This generates a lot of noise in logs.

The controller gets notified about changes to secrets but doesn't do anything if the secret isn't referenced anywhere. There will be a log emitted but no configuration change is performed.

While "normal" services are created just fine, I have an issue with exposing "/status" endpoint from Kong admin api (which in previous version I used as a heathcheck). No matter what I do, even when I create this ingress on a fresh Kong and Ingress Controller installation, I always get "err inserting target into state: entity already exists".

This is new. Can you share reproducible steps please (in a separate (new) issue)?

Sorry if those comments seem unrelated to the bug I filed, but I'm just following your suggestion and trying to upgrade :)

Please open separate Github issues for these topics in future We want to avoid a single issue with many unrelated things in it.

[hbagdi]

Ping.

[rochr01]

Hi, Sorry, I've been busy recently. So, the original issue reported here seems to be fixed in version 0.8.0 - I've just tested it, and the "schema violation (paths.1: must not have empty segments)" error does not break sync loop anymore, all the other Ingresses get created just fine.

As per "err inserting target into state: entity already exists" I will try to reproduce it and create a separate issue.

Thanks