Bug when creating Consumer

heitorganzeli commented 6 years ago

Is this a request for help?: yes

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kong Ingress controller version: 0.0.5 Kong version: 0.14.0

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:14:26Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration:on premise kubernetes
OS (e.g. from /etc/os-release): Oracle Linux Server 7.5
Kernel (e.g. uname -a): 4.1.12-112.16.7.el7uek.x86_64

What happened:

I'm trying to create simple Kong configuration based on custom types tutorial

But whenever i try to create a new consumer, the controller mess up resource creation/synchronization between Kubernetes and Kong.

I'm using the following configuration for the consumer creation:

apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: consumer-team-x
  namespace: kong
username: team-X

both entities get created:

kubernetes: kubectl describe kongconsumer consumer-team-x

Name:         consumer-team-x
Namespace:    kong
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"configuration.konghq.com/v1","kind":"KongConsumer","metadata":{"annotations":{},"name":"consumer-team-x","namespace":"kong"},"username":...
API Version:  configuration.konghq.com/v1
Kind:         KongConsumer
Metadata:
Cluster Name:        
Creation Timestamp:  2018-07-20T18:00:31Z
Generation:          1
Resource Version:    11580589
Self Link:           /apis/configuration.konghq.com/v1/namespaces/kong/kongconsumers/consumer-team-x
UID:                 cb2a35d1-8c46-11e8-a4ba-90b11c1dd55e
Username:              team-X
Events:                <none>

admin-api: curl http://<host>:<port>/consumers

{
"next":null,
"data":[
  {
     "custom_id":null,
     "created_at":1532109545,
     "username":"team-X",
     "id":"8f81bc04-df4e-4afe-b165-3be99e67dcf6"
  }
]
}

although, both entities are there, ingress-controller keeps trying to recreate consumer object in kong using kubernetes UUID: cb2a35d1-8c46-11e8-a4ba-90b11c1dd55e, and do complete other configuration objects setup. Controller log:

W0720 18:27:32.851926       7 queue.go:113] requeuing simet/lmap-collector, err creating a Kong consumer: the server reported a conflict (post consumers.meta.k8s.io)
I0720 18:27:36.181051       7 controller.go:127] syncing Ingress configuration...
I0720 18:27:36.181075       7 kong.go:465] checking if Kong consumer consumer-team-x exists
I0720 18:27:36.182976       7 kong.go:469] Creating Kong consumer cb2a35d1-8c46-11e8-a4ba-90b11c1dd55e
E0720 18:27:36.185011       7 controller.go:130] unexpected failure updating Kong configuration: 
creating a Kong consumer: the server reported a conflict (post consumers.meta.k8s.io)
W0720 18:27:36.185035       7 queue.go:113] requeuing kong/kong-ingress-controller, err creating a Kong consumer: the server reported a conflict (post consumers.meta.k8s.io)

and looking at admin-api logs, we can see that the controller is trying to recreate the consumer with the wrong id:

::1 - - [20/Jul/2018:18:49:32 +0000] "POST /consumers HTTP/1.1" 409 144 "-" "Go-http-client/1.1"
::1 - - [20/Jul/2018:18:49:36 +0000] "GET /consumers/cb2a35d1-8c46-11e8-a4ba-90b11c1dd55e HTTP/1.1" 404 24 "-" "Go-http-client/1.1"
2018/07/20 18:49:36 [notice] 88#0: *245958 [lua] init.lua:391: insert(): ERROR: duplicate key value violates unique constraint "consumers_username_key"
Key (username)=(team-X) already exists., client: ::1, server: kong_admin, request: "POST /consumers HTTP/1.1", host: "localhost:8001"
::1 - - [20/Jul/2018:18:49:36 +0000] "POST /consumers HTTP/1.1" 409 144 "-" "Go-http-client/1.1"

note: Kong consumer uuid is: 8f81bc04-df4e-4afe-b165-3be99e67dcf6

I've tried to recreate all pods and configurations a couple a of times and with some modifications, but things always end up like that.

What you expected to happen:

I'd expect Kong object to be created with Kubernetes object UUID so controller can work properly

How to reproduce it (as minimally and precisely as possible):

i'm using a namespace similar to the one described by the controller tutorial: https://raw.githubusercontent.com/Kong/kubernetes-ingress-controller/master/deploy/single/all-in-one-postgres.yaml

thx in advance

AtzeDeVries commented 6 years ago

i can confirm this bug with kong-0.13.1-centos and k8s v1.11.0

hbagdi commented 6 years ago

Hi @heitorganzeli

Thank you for opening this issue. Kong CE 0.14.0 is currently not supported with the Ingress controller. The bug you're seeing is because Kong 0.14.0 introduces a breaking change in how consumers are handled on Admin API.

hbagdi commented 6 years ago

Hi @AtzeDeVries I cannot reproduce this issue with kong-0.13.1-centos and k8sv1.10.0. Since the k8s version changed, are you sure you're seeing exact those errors? Could you share some error logs?

heitorganzeli commented 6 years ago

Hi @hbagdi

we could probably add and maintain KONG supported versions on Readme so less people end up having this same error.

hbagdi commented 6 years ago

Hi @heitorganzeli Just opened up https://github.com/Kong/kubernetes-ingress-controller/pull/77

Thank you for your patience on this. Let us know if you encounter any other issue.

jaygorrell commented 6 years ago

I am also seeing this in kong:0.13.1-centos like @AtzeDeVries. This is a new kong ingress deployment as of today so there's not many variables here. The KongConsumer objects are created in k8s but the logs are failing to synchronize that to kong very similar to what @heitorganzeli stated.

hbagdi commented 6 years ago

@jaygorrell What is your k8s version? Could you post some logs that you're seeing?

jaygorrell commented 6 years ago

Sure, I'm on k8s v1.9.3 and it's essentially looping ove rthis in the logs

W0811 02:21:18.923490       7 queue.go:113] requeuing default/token-service-public, err creating a Kong consumer: the server rejected our request for an unknown reason (post consumers.meta.k8s.io)
I0811 02:21:21.919563       7 controller.go:127] syncing Ingress configuration...
I0811 02:21:22.252486       7 kong.go:454] checking if Kong consumer auth-service exists
I0811 02:21:22.257413       7 kong.go:458] Creating Kong consumer 4344b063-9d0c-11e8-9eb0-06c02f17e102
E0811 02:21:22.258096       7 controller.go:130] unexpected failure updating Kong configuration: 
creating a Kong consumer: the server rejected our request for an unknown reason (post consumers.meta.k8s.io)

The default/token-service-public changes each time and the odd thing is that it seems to be coming from the names of my different Kong custom resource types... the names match KongConsumer, KongIngress, and KongPlugin resources.

jaygorrell commented 6 years ago

Ah, this seems to be isolated to specifying only the customId field, which should work as it's semi-optional with username.

This will fail to create in Kong:

apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: customid-test
customId: auth-service

Another noteworthy trait: it seems to block other Kong changes from applying. My ingress changes weren't going through to test #56 so I tried deleting the consumer resource and that cleared it right up.

hbagdi commented 6 years ago

@jaygorrell Thanks for the logs and narrowing it down to customId issue. As you said, this should work since customId is a semi-optional field. We welcome and value contributions from the community so if you feel like doing it, we will be happy to assist you!

Another noteworthy trait: it seems to block other Kong changes from applying. My ingress changes weren't going through to test #56 so I tried deleting the consumer resource and that cleared it right up.

The ingress controller breaks out of the synchronization loop as soon as any operation has an error, hence the behavior.

jaygorrell commented 6 years ago

I do wish I had time to learn enough Go to contribute but I'll have to let someone else have the fun here :)

hbagdi commented 6 years ago

@jaygorrell I think the issue you're facing here is because of incorrect customId property in your yaml. Please use custom_id.

The following should work fine:

apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: foo
custom_id: foo

jaygorrell commented 6 years ago

Hah, of course. Thanks for pointing that out, and opened a tiny PR to address the docs.

megastallman commented 6 years ago

Guys, when are you going to support Kong 0.14?

hbagdi commented 6 years ago

Support for 0.14 has landed in the next Git branch of this repository. There are a few other enhancement in progress which will be included in our next release.

rochr01 commented 4 years ago

The ingress controller breaks out of the synchronization loop as soon as any operation has an error, hence the behavior.

@hbagdi: You did fix that in later versions, I hope? I mean, why would single configuration issue for one object prevent the whole ingress controller from working?

Kong / kubernetes-ingress-controller

Bug when creating Consumer #70