search

knative / eventing

Event-driven application platform for Kubernetes
https://knative.dev/docs/eventing
Apache License 2.0
1.42k stars 598 forks source link

inmemorychannel-webhook-certs secret doesn't have any data #6857

Closed ysk24ok closed 1 year ago

ysk24ok commented 1 year ago

Expected Behavior

imc-controller works.

Actual Behavior

imc-controller keeps failing.

% kubectl -n knative-eventing get pods | grep imc-controller
imc-controller-794f5589df-skzrp         0/1     CrashLoopBackOff   71 (3m56s ago)   4h25m

imc-controller logs are like this.

{"level":"info","ts":"2023-04-12T10:14:26.879Z","logger":"inmemorychannel-webhook","caller":"webhook/webhook.go:242","msg":"Starting to fail readiness probes...","commit":"03f6f3b","knative.dev/pod":"imc-controller-794f5589df-skzrp"}
98
{"level":"info","ts":"2023-04-12T10:14:26.879Z","logger":"inmemorychannel-webhook.DefaultingWebhook","caller":"controller/controller.go:485","msg":"Shutting down workers","commit":"03f6f3b","knative.dev/pod":"imc-controller-794f5589df-skzrp"}
{"level":"info","ts":"2023-04-12T10:14:26.879Z","logger":"inmemorychannel-webhook","caller":"leaderelection/context.go:160","msg":"\"imc-controller-794f5589df-skzrp_ceed9d58-5424-49ee-8ec9-1af3f1d68269\" has stopped leading \"inmemorychannel-webhook.defaultingwebhook.00-of-01\"","commit":"03f6f3b","knative.dev/pod":"imc-controller-794f5589df-skzrp"}
{"level":"info","ts":"2023-04-12T10:14:26.880Z","logger":"inmemorychannel-webhook","caller":"controller/controller.go:485","msg":"Shutting down workers","commit":"03f6f3b","knative.dev/pod":"imc-controller-794f5589df-skzrp","knative.dev/controller":"knative.dev.eventing.pkg.reconciler.inmemorychannel.controller.Reconciler","knative.dev/kind":"messaging.knative.dev.InMemoryChannel"}
{"level":"info","ts":"2023-04-12T10:14:26.880Z","logger":"inmemorychannel-webhook.WebhookCertificates","caller":"controller/controller.go:485","msg":"Shutting down workers","commit":"03f6f3b","knative.dev/pod":"imc-controller-794f5589df-skzrp"}
{"level":"info","ts":"2023-04-12T10:14:26.881Z","logger":"inmemorychannel-webhook","caller":"leaderelection/context.go:160","msg":"\"imc-controller-794f5589df-skzrp_d056ac7c-3a20-42ed-8bb1-bcd95294192f\" has stopped leading \"inmemorychannel-webhook.validationwebhook.00-of-01\"","commit":"03f6f3b","knative.dev/pod":"imc-controller-794f5589df-skzrp"}
{"level":"info","ts":"2023-04-12T10:14:26.881Z","logger":"inmemorychannel-webhook","caller":"leaderelection/context.go:160","msg":"\"imc-controller-794f5589df-skzrp_0974e793-f70c-4de0-98c0-55d8c709ad2a\" has stopped leading \"inmemorychannel-webhook.knative.dev.eventing.pkg.reconciler.inmemorychannel.controller.reconciler.00-of-01\"","commit":"03f6f3b","knative.dev/pod":"imc-controller-794f5589df-skzrp"}
{"level":"info","ts":"2023-04-12T10:14:26.879Z","logger":"inmemorychannel-webhook.ValidationWebhook","caller":"controller/controller.go:485","msg":"Shutting down workers","commit":"03f6f3b","knative.dev/pod":"imc-controller-794f5589df-skzrp"}
{"level":"info","ts":"2023-04-12T10:14:26.881Z","logger":"inmemorychannel-webhook","caller":"leaderelection/context.go:160","msg":"\"imc-controller-794f5589df-skzrp_f54aaae7-a842-4135-bafd-fed290bf4578\" has stopped leading \"inmemorychannel-webhook.webhookcertificates.00-of-01\"","commit":"03f6f3b","knative.dev/pod":"imc-controller-794f5589df-skzrp"}
2023/04/12 10:14:27 http: TLS handshake error from 10.36.17.65:60248: tls: no certificates configured
{"level":"warn","ts":"2023-04-12T10:14:27.859Z","logger":"inmemorychannel-webhook","caller":"webhook/webhook.go:156","msg":"server key missing","commit":"03f6f3b","knative.dev/pod":"imc-controller-794f5589df-skzrp"}
{"level":"warn","ts":"2023-04-12T10:14:28.859Z","logger":"inmemorychannel-webhook","caller":"webhook/webhook.go:156","msg":"server key missing","commit":"03f6f3b","knative.dev/pod":"imc-controller-794f5589df-skzrp"}
2023/04/12 10:14:28 http: TLS handshake error from 10.36.17.65:60250: tls: no certificates configured
2023/04/12 10:14:29 http: TLS handshake error from 10.36.17.65:60256: tls: no certificates configured

It seems like inmemorychannel-webhook-certs doesn't have any data, so I suspect this is the cause.

% kubectl -n knative-eventing get secret inmemorychannel-webhook-certs
NAME                            TYPE     DATA   AGE
inmemorychannel-webhook-certs   Opaque   0      4h28m

Steps to Reproduce the Problem

  1. Install eventing-core.yaml of knative-eventing 1.2.4
    • The reason why we use 1.2.4 is it's bundled with Kubeflow 1.6. But you don't need to install Kubeflow to reproduce this.
  2. Install in-memory-channel.yaml of knative-eventing 1.2.4
    • Note that there are some duplicated resources in this file so you should remove them. The duplicated resources are as follows:
      • knative-eventing namespace
      • config-observability configmap
      • config-tracing configmap
  3. Delete all resources in in-memory-channel.yaml by running kubectl delete -f in-memory-channel.yaml
  4. Install in-memory-channel.yaml again then you'll see no data in inmemorychannel-webhook-certs secret.
    • The reason why we reapplied the file is that we wanted to use a different ArgoCD application to deploy them. We first deployed the resources using an old application, but moved the resources to a new application, pruned all the resources in the old app, and deployed the new application.

Additional Info

pierDipi commented 1 year ago

@ysk24ok do you have the link to docs and artifacts where kubeflow is bundling Knative at 1.2, that's a very old version that is very unlikely we would be able to patch and release it, so we should find a way for kubeflow users to provide their own Knative so that you could use a different Knative version.

I'm not that familiar with the why they bundle Knative instead of pointing to our docs for installation, maybe you know?

ysk24ok commented 1 year ago

@pierDipi Thank you for your reply.

do you have the link to docs and artifacts where kubeflow is bundling Knative at 1.2

https://github.com/kubeflow/manifests/tree/v1.6.0/common/knative/knative-eventing/base

I'm not that familiar with the why they bundle Knative instead of pointing to our docs for installation,

I don't know either, but I guess it's just the way they do when it comes to 3rd party libraries. The at-that-time latest version of 3rd party libraries are bundled In the contrib directory.

I already resolved the issue by recreating the namespace, so I think it's OK to close this issue. But before closing, is the problem solved with the latest knative-eventing?

Cali0707 commented 1 year ago

@ysk24ok I've followed the procedure you outlined of creating, deleting, and then recreating the imc resources on the latest release of knative-eventing and it seems to have had no issues:

kubectl get pods -n knative-eventing
NAME                                   READY   STATUS    RESTARTS   AGE
eventing-controller-866f7c5bb7-g629n   1/1     Running   0          3m55s
eventing-webhook-9b696bc78-4wtrb       1/1     Running   0          3m55s
imc-controller-57c655795-bcbvz         1/1     Running   0          92s
imc-dispatcher-67f646f684-s2ld2        1/1     Running   0          92s

Do you feel like we can close this issue?

github-actions[bot] commented 1 year ago

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.