search

kubeshop / kusk-gateway

Kusk-gateway is an OpenAPI-driven API Gateway for Kubernetes
https://kubeshop.github.io/kusk-gateway/
MIT License
253 stars 21 forks source link

Bug: Cloudentity fails for a user #1009

Closed jasmingacic closed 1 year ago

jasmingacic commented 1 year ago

Reported by a user in Slack

Problem with Kusk + Cloudentity Authorizer

​ I have included steps to install the Cloudentity authorizer, and the upstream service 'httpbin', but since the error occurs when we apply the API spec, I doubt that either of these components are necessary to reproduce the fault. ​ With a Kind cluster v1.24.0 running Cloudentity's ACP 2.8.1, install kusk:

brew install kubeshop/kusk/kusk
kusk cluster install --no-dashboard --no-api

​ Confirm kusk version:

$ kusk version
Kusk version 1.5.0
https://github.com/kubeshop/kusk-gateway/releases/tag/v1.5.0
​
docker.io/envoyproxy/envoy:v1.23.1
kubeshop/kusk-gateway:v1.5.0
gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0

​ Install the Cloudentity authorizer:

helm repo add acp https://charts.cloudentity.io
helm repo update
helm upgrade --install cloudentity-authorizer acp/standalone-authorizer \
    --set clientCredentials.clientID=******************************** \
    --set clientCredentials.clientSecret=**************************************** \
    --set issuerURL=https://acp.acp-system:8443/default/system \
    --namespace kusk-system

​ Deploy the httpbin service:

kubectl apply -f examples/httpbin/deployment.yaml

​ Create a yaml spec:

openapi: 3.0.0
info:
  title: httpbin-api
  version: 0.1.0
x-kusk:
  upstream:
    service:
      name: httpbin
      namespace: default
      port: 80
  auth:
    cloudentity:
      host:
        hostname: cloudentity-authorizer-standalone-authorizer.kusk-system
        port: 9004
paths:
  /anything:
    get:
      produces:
        - application/json
      responses:
        "200":
          description: Anything passed in request
      summary: Returns anything passed in request data.
      tags:
        - Anything

​ And apply the spec:

kusk api generate -i api-httpbin.yaml | kubectl apply -f -

​ Logs from the kusk-gateway-manager pod:

manager METRICS_BIND_ADDR=127.0.0.1:8080
manager HEALTH_PROBE_BIND_ADDR=:8081
manager ENVOY_CONTROL_PLANE_BIND_ADDR=:18000
manager ENABLE_LEADER_ELECTION=false
manager LOG_LEVEL=INFO
manager WEBHOOK_CERTS_DIR=/tmp/k8s-webhook-server/serving-certs
manager ANALYTICS_ENABLED=true
manager 
manager {"level":"info","ts":1670346354.5379133,"logger":"controller-runtime.metrics","caller":"logr@v1.2.3/logr.go:261","msg":"Metrics server is starting to listen","addr":"127.0.0.1:8080"}
manager {"level":"info","ts":1670346354.5385242,"caller":"authz/authz.go:53","msg":"authz listening on","address":":19000"}
manager {"level":"dpanic","ts":1670346354.5385497,"caller":"validation/extension.go:54","msg":"odd number of arguments passed as key-value pairs for logging","ignored key":":17000","stacktrace":"github.com/kubeshop/kusk-gateway/internal/validation.(*Server).Start\n\t/workspace/internal/validation/extension.go:54\nmain.main.func2\n\t/workspace/cmd/manager/main.go:254"}
manager {"level":"info","ts":1670346354.538471,"caller":"validation/extension.go:54","msg":"validation server listening at"}
manager {"level":"info","ts":1670346354.5386918,"logger":"setup","caller":"manager/main.go:244","msg":"Starting Envoy xDS API Server"}
manager {"level":"info","ts":1670346354.5389154,"logger":"EnvoyConfigManager","caller":"manager/envoy_config_manager.go:79","msg":"control plane server listening","address":":18000"}
manager {"level":"info","ts":1670346354.741382,"logger":"setup","caller":"manager/main.go:289","msg":"Starting K8s secrets watch for the TLS certificates renewal events"}
manager {"level":"info","ts":1670346358.3996499,"logger":"setup","caller":"manager/main.go:309","msg":"Created admission webhook server certificates and updated K8s Manager's Admission configs with the generated CA certificate"}
manager {"level":"info","ts":1670346358.3997037,"logger":"setup","caller":"manager/main.go:311","msg":"Registering EnvoyFleet mutating and validating webhooks to the webhook server"}
manager {"level":"info","ts":1670346358.3999333,"logger":"controller-runtime.webhook","caller":"webhook/server.go:148","msg":"Registering webhook","path":"/mutate-gateway-kusk-io-v1alpha1-envoyfleet"}
manager {"level":"info","ts":1670346358.400146,"logger":"controller-runtime.webhook","caller":"webhook/server.go:148","msg":"Registering webhook","path":"/validate-gateway-kusk-io-v1alpha1-envoyfleet"}
manager {"level":"info","ts":1670346358.400373,"logger":"setup","caller":"manager/main.go:327","msg":"Registering API mutating and validating webhooks to the webhook server"}
manager {"level":"info","ts":1670346358.4005182,"logger":"controller-runtime.webhook","caller":"webhook/server.go:148","msg":"Registering webhook","path":"/mutate-gateway-kusk-io-v1alpha1-api"}
manager {"level":"info","ts":1670346358.4006188,"logger":"controller-runtime.webhook","caller":"webhook/server.go:148","msg":"Registering webhook","path":"/validate-gateway-kusk-io-v1alpha1-api"}
manager {"level":"info","ts":1670346358.4008603,"logger":"setup","caller":"manager/main.go:343","msg":"Registering StaticRoute mutating and validating webhooks to the webhook server"}
manager {"level":"info","ts":1670346358.4009366,"logger":"controller-runtime.webhook","caller":"webhook/server.go:148","msg":"Registering webhook","path":"/mutate-gateway-kusk-io-v1alpha1-staticroute"}
manager {"level":"info","ts":1670346358.4012203,"logger":"controller-runtime.webhook","caller":"webhook/server.go:148","msg":"Registering webhook","path":"/validate-gateway-kusk-io-v1alpha1-staticroute"}
manager {"level":"info","ts":1670346358.4012673,"logger":"setup","caller":"manager/main.go:357","msg":"Starting manager"}
manager {"level":"info","ts":1670346358.401613,"caller":"manager/internal.go:362","msg":"Starting server","path":"/metrics","kind":"metrics","addr":"127.0.0.1:8080"}
manager {"level":"info","ts":1670346358.4016016,"logger":"controller-runtime.webhook.webhooks","caller":"webhook/server.go:216","msg":"Starting webhook server"}
manager {"level":"info","ts":1670346358.4016736,"caller":"manager/internal.go:362","msg":"Starting server","kind":"health probe","addr":":8081"}
manager {"level":"info","ts":1670346358.4019914,"logger":"controller-runtime.certwatcher","caller":"logr@v1.2.3/logr.go:261","msg":"Updated current TLS certificate"}
manager {"level":"info","ts":1670346358.4021823,"logger":"controller-runtime.webhook","caller":"logr@v1.2.3/logr.go:261","msg":"Serving webhook server","host":"","port":9443}
manager {"level":"info","ts":1670346358.4025552,"logger":"controller-runtime.certwatcher","caller":"logr@v1.2.3/logr.go:261","msg":"Starting certificate watcher"}
manager {"level":"info","ts":1670346358.5028522,"caller":"controller/controller.go:185","msg":"Starting EventSource","controller":"envoyfleet","controllerGroup":"gateway.kusk.io","controllerKind":"EnvoyFleet","source":"kind source: *v1alpha1.EnvoyFleet"}
manager {"level":"info","ts":1670346358.5029304,"caller":"controller/controller.go:185","msg":"Starting EventSource","controller":"api","controllerGroup":"gateway.kusk.io","controllerKind":"API","source":"kind source: *v1alpha1.API"}
manager {"level":"info","ts":1670346358.5029583,"caller":"controller/controller.go:193","msg":"Starting Controller","controller":"api","controllerGroup":"gateway.kusk.io","controllerKind":"API"}
manager {"level":"info","ts":1670346358.5029557,"caller":"controller/controller.go:193","msg":"Starting Controller","controller":"envoyfleet","controllerGroup":"gateway.kusk.io","controllerKind":"EnvoyFleet"}
manager {"level":"info","ts":1670346358.5029604,"caller":"controller/controller.go:185","msg":"Starting EventSource","controller":"staticroute","controllerGroup":"gateway.kusk.io","controllerKind":"StaticRoute","source":"kind source: *v1alpha1.StaticRoute"}
manager {"level":"info","ts":1670346358.5029826,"caller":"controller/controller.go:193","msg":"Starting Controller","controller":"staticroute","controllerGroup":"gateway.kusk.io","controllerKind":"StaticRoute"}
manager {"level":"info","ts":1670346358.6042345,"caller":"controller/controller.go:227","msg":"Starting workers","controller":"staticroute","controllerGroup":"gateway.kusk.io","controllerKind":"StaticRoute","worker count":1}
manager {"level":"info","ts":1670346358.6043148,"caller":"controller/controller.go:227","msg":"Starting workers","controller":"envoyfleet","controllerGroup":"gateway.kusk.io","controllerKind":"EnvoyFleet","worker count":1}
manager {"level":"info","ts":1670346358.604335,"caller":"controller/controller.go:227","msg":"Starting workers","controller":"api","controllerGroup":"gateway.kusk.io","controllerKind":"API","worker count":1}
manager {"level":"info","ts":1670346358.60439,"logger":"envoy-fleet-controller","caller":"controllers/envoyfleet_controller.go:68","msg":"EnvoyFleet changed","controller":"envoyfleet","controllerGroup":"gateway.kusk.io","controllerKind":"EnvoyFleet","envoyFleet":{"name":"kusk-gateway-envoy-fleet","namespace":"kusk-system"},"namespace":"kusk-system","name":"kusk-gateway-envoy-fleet","reconcileID":"02ac7449-4ab1-4606-b0c3-c937260ab2b1","changed":"kusk-system/kusk-gateway-envoy-fleet"}
manager {"level":"info","ts":1670346358.6728199,"logger":"api-controller","caller":"controllers/api_controller.go:65","msg":"Reconciling changed API resource","controller":"api","controllerGroup":"gateway.kusk.io","controllerKind":"API","aPI":{"name":"httpbin-api","namespace":"default"},"namespace":"default","name":"httpbin-api","reconcileID":"4300e1e2-0315-4432-83cd-2dfa77a11df0","changed":"default/httpbin-api"}
manager {"level":"info","ts":1670346358.6729412,"logger":"controller.config-manager","caller":"logr@v1.2.3/logr.go:261","msg":"Started updating configuration","fleet":"kusk-gateway-envoy-fleet.kusk-system"}
manager {"level":"info","ts":1670346358.6729608,"logger":"controller.config-manager","caller":"logr@v1.2.3/logr.go:261","msg":"Getting APIs for the fleet","fleet":"kusk-gateway-envoy-fleet.kusk-system"}
manager {"level":"info","ts":1670346358.6755006,"logger":"controller.config-manager","caller":"logr@v1.2.3/logr.go:261","msg":"Processing API configuration","fleet":"kusk-gateway-envoy-fleet.kusk-system","api":"httpbin-api"}
manager {"level":"info","ts":1670346358.676585,"logger":"internal/controllers/parser.go:UpdateConfigFromAPIOpts","caller":"controllers/parser.go:176","msg":"parsing `auth` options","finalOpts.Auth":"&options.AuthOptions{OAuth2:(*options.OAuth2)(nil), Custom:(*options.Custom)(nil), Cloudentity:(*options.Cloudentity)(0xc00084e240), JWT:(*options.JWT)(nil)}"}
manager {"level":"info","ts":1670346358.6766367,"logger":"controller.config-manager","caller":"logr@v1.2.3/logr.go:261","msg":"Finished updating configuration","fleet":"kusk-gateway-envoy-fleet.kusk-system"}
manager {"level":"info","ts":1670346358.676646,"logger":"api-controller","caller":"runtime/panic.go:884","msg":"Finished reconciling changed API resource","controller":"api","controllerGroup":"gateway.kusk.io","controllerKind":"API","aPI":{"name":"httpbin-api","namespace":"default"},"namespace":"default","name":"httpbin-api","reconcileID":"4300e1e2-0315-4432-83cd-2dfa77a11df0","changed":"default/httpbin-api"}
manager {"level":"info","ts":1670346358.6766565,"caller":"controller/controller.go:117","msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"api","controllerGroup":"gateway.kusk.io","controllerKind":"API","aPI":{"name":"httpbin-api","namespace":"default"},"namespace":"default","name":"httpbin-api","reconcileID":"4300e1e2-0315-4432-83cd-2dfa77a11df0"}
manager panic: runtime error: invalid memory address or nil pointer dereference [recovered]
manager     panic: runtime error: invalid memory address or nil pointer dereference
manager [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x181ecdb]
manager 
manager goroutine 310 [running]:
manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
manager     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:118 +0x1f4
manager panic({0x1a118e0, 0x2e7be20})
manager     /usr/local/go/src/runtime/panic.go:884 +0x212
manager github.com/kubeshop/kusk-gateway/internal/envoy/auth.ParseAuthOptions(0xc0004cfd80, 0xc000b70b20)
manager     /workspace/internal/envoy/auth/parser.go:85 +0x17b
manager github.com/kubeshop/kusk-gateway/internal/controllers.UpdateConfigFromAPIOpts(0xc00040ab40, {0x1f95a60, 0xc0008dacc0}, 0xc000a2b050, 0xc000182540, 0xc000696350, 0xc000696358, {0xc0000c8700, 0xb}, {0x1fb9f50, ...})
manager     /workspace/internal/controllers/parser.go:192 +0x12bc
manager github.com/kubeshop/kusk-gateway/internal/controllers.(*KubeEnvoyConfigManager).UpdateConfiguration(0xc0003cd620, {0x1fb24b0, 0xc000882a80}, {{0xc0000ca1b0?, 0x20?}, {0xc0000c8720?, 0x2?}})
manager     /workspace/internal/controllers/config_manager.go:114 +0xac6
manager github.com/kubeshop/kusk-gateway/internal/controllers.(*APIReconciler).Reconcile(0xc0002178a0, {0x1fb24b0, 0xc000882a80}, {{{0xc0000c8710, 0x7}, {0xc0000c8700, 0xb}}})
manager     /workspace/internal/controllers/api_controller.go:114 +0x82f
manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1fb2408?, {0x1fb24b0?, 0xc000882a80?}, {{{0xc0000c8710?, 0x1bf75e0?}, {0xc0000c8700?, 0x4045d4?}}})
manager     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121 +0xc8
manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000798640, {0x1fb2408, 0xc0008dab80}, {0x1aa8d60?, 0xc00038ca60?})
manager     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320 +0x33c
manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000798640, {0x1fb2408, 0xc0008dab80})
manager     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273 +0x1d9
manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
manager     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234 +0x85
manager created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
manager     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:230 +0x333
Stream closed EOF for kusk-system/kusk-gateway-manager-6c494bb78-9kpf7 (manager)