Non HTTPs custom cluster interceptor broken in v0.20.1

[afrittoli]

Expected Behavior

After installing v0.20.1 I can continue to use my non-HTTPs custom cluster interceptor

Actual Behavior

It looks like triggers reconciles a caBundle in the interceptor definition, which results in:

{"level":"error","ts":"2022-07-05T09:31:46.920Z","logger":"eventlistener","caller":"sink/sink.go:289","msg":"Post \"https://build-id-interceptor.tekton-ci.svc:80/buildid\": http: server gave HTTP response to HTTPS client","eventlistener":"tekton-ci","namespace":"tekton-ci","/triggers-eventid":"fdb22324-7af5-4af3-ab26-3955539b5baf","eventlistenerUID":"52ff3801-1e0b-4a68-b0af-3d9113cf345d","/triggers-eventid":"fdb22324-7af5-4af3-ab26-3955539b5baf","/triggergroup":"github-tektoncd-issue-comment-group","stacktrace":"github.com/tektoncd/triggers/pkg/sink.Sink.processTriggerGroups\n\tgithub.com/tektoncd/triggers/pkg/sink/sink.go:289\ngithub.com/tektoncd/triggers/pkg/sink.Sink.HandleEvent.func2\n\tgithub.com/tektoncd/triggers/pkg/sink/sink.go:206"}

I manually delete the caBundle from my interceptor and things started working again for some time. After a while things stopped working again, and I realised that the caBundle was back in. I edited the interceptor again. Things started working again and then stopped working. This time no caBundle in the interceptor, but the event listener pod is in crash loop:

{"level":"fatal","ts":"2022-07-06T14:48:08.228Z","logger":"eventlistener","caller":"v2/main.go:205","msg":"Start returned an error","error":"Timed out waiting on CaBundle to available for clusterInterceptor: empty caBundle in clusterInterceptor spec","stacktrace":"github.com/tektoncd/triggers/vendor/knative.dev/eventing/pkg/adapter/v2.MainWithInformers\n\tgithub.com/tektoncd/triggers/vendor/knative.dev/eventing/pkg/adapter/v2/main.go:205\ngithub.com/tektoncd/triggers/vendor/knative.dev/eventing/pkg/adapter/v2.MainWithEnv\n\tgithub.com/tektoncd/triggers/vendor/knative.dev/eventing/pkg/adapter/v2/main.go:105\ngithub.com/tektoncd/triggers/vendor/knative.dev/eventing/pkg/adapter/v2.MainWithContext\n\tgithub.com/tektoncd/triggers/vendor/knative.dev/eventing/pkg/adapter/v2/main.go:80\nmain.main\n\tgithub.com/tektoncd/triggers/cmd/eventlistenersink/main.go:69\nruntime.main\n\truntime/proc.go:255"}

Steps to Reproduce the Problem

Interceptor YAML:

apiVersion: triggers.tekton.dev/v1alpha1
kind: ClusterInterceptor
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"triggers.tekton.dev/v1alpha1","kind":"ClusterInterceptor","metadata":{"annotations":{},"name":"build-id"},"spec":{"clientConfig":{"service":{"name":"build-id-interceptor","namespace":"tekton-ci","path":"build-id"}}}}
  creationTimestamp: "2022-07-01T14:55:26Z"
  generation: 14
  name: build-id
  resourceVersion: "1625790284"
  uid: f4a58140-dc3f-4821-840c-3af59dacd08e
spec:
  clientConfig:
    service:
      name: build-id-interceptor
      namespace: tekton-ci
      path: buildid
      port: 80
status:
  address:
    url: http://build-id-interceptor.tekton-ci.svc:80/buildid

Additional Info

• Kubernetes version:

Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.11-gke.1900", GitCommit:"1f775cfbf564daf2c8f0ba548eff09f58045c4a4", GitTreeState:"clean", BuildDate:"2022-04-13T09:36:54Z", GoVersion:"go1.16.15b7", Compiler:"gc", Platform:"linux/amd64"}

• Tekton Pipeline version:

Client version: 0.23.1
Chains version: v0.8.0
Pipeline version: v0.37.1
Triggers version: v0.20.1
Dashboard version: v0.27.0

[dibyom]

Manually specifying spec.clientConfig.url might be a workaround for now based on https://github.com/tektoncd/triggers/blob/main/pkg/apis/triggers/v1alpha1/cluster_interceptor_types.go#L71

[afrittoli]

Deleting the cluster interceptor, restarting the event listener and then recreating the cluster interceptor seems to work, the service is up and running now - I'll keep an eye on it. Once the patch release is ready I will try the upgrade from v0.20.1 to v0.20.2.

[afrittoli]

Interceptor crashed again:

{
  "level": "fatal",
  "ts": "2022-07-06T17:06:52.980Z",
  "logger": "eventlistener",
  "caller": "v2/main.go:205",
  "msg": "Start returned an error",
  "error": "Timed out waiting on CaBundle to available for clusterInterceptor: empty caBundle in clusterInterceptor spec",
  "stacktrace": "github.com/tektoncd/triggers/vendor/knative.dev/eventing/pkg/adapter/v2.MainWithInformers\n\tgithub.com/tektoncd/triggers/vendor/knative.dev/eventing/pkg/adapter/v2/main.go:205\ngithub.com/tektoncd/triggers/vendor/knative.dev/eventing/pkg/adapter/v2.MainWithEnv\n\tgithub.com/tektoncd/triggers/vendor/knative.dev/eventing/pkg/adapter/v2/main.go:105\ngithub.com/tektoncd/triggers/vendor/knative.dev/eventing/pkg/adapter/v2.MainWithContext\n\tgithub.com/tektoncd/triggers/vendor/knative.dev/eventing/pkg/adapter/v2/main.go:80\nmain.main\n\tgithub.com/tektoncd/triggers/cmd/eventlistenersink/main.go:69\nruntime.main\n\truntime/proc.go:255"
}

[afrittoli]

Tried the following:

• delete CI
• delete EL deployment
• create the CI with URL only

apiVersion: triggers.tekton.dev/v1alpha1
kind: ClusterInterceptor
metadata:
  name: build-id
spec:
  clientConfig:
    url: http://build-id-interceptor.tekton-ci.svc:80/buildid

We'll see if this setup lasts

[afrittoli]

Sometimes later, the caBundle is back:

$ ky get clusterinterceptor/build-id
apiVersion: triggers.tekton.dev/v1alpha1
kind: ClusterInterceptor
metadata:
  creationTimestamp: "2022-07-06T17:12:26Z"
  generation: 2
  name: build-id
  resourceVersion: "1628385105"
  uid: d69534fb-bfcc-4f05-af67-b6aa300aae84
spec:
  clientConfig:
    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM5akNDQXB5Z0F3SUJBZ0lRUHN4dko5SzBHMndVZmJDS2pOdVVvakFLQmdncWhrak9QUVFEQWpCWE1SUXcKRWdZRFZRUUtFd3RyYm1GMGFYWmxMbVJsZGpFL01EMEdBMVVFQXhNMmRHVnJkRzl1TFhSeWFXZG5aWEp6TFdOdgpjbVV0YVc1MFpYSmpaWEIwYjNKekxuUmxhM1J2Ymkxd2FYQmxiR2x1WlhNdWMzWmpNQ0FYRFRJeU1EY3dOVEE0Ck16TTFORm9ZRHpJeE1qSXdOakV4TURnek16VTBXakJYTVJRd0VnWURWUVFLRXd0cmJtRjBhWFpsTG1SbGRqRS8KTUQwR0ExVUVBeE0yZEdWcmRHOXVMWFJ5YVdkblpYSnpMV052Y21VdGFXNTBaWEpqWlhCMGIzSnpMblJsYTNSdgpiaTF3YVhCbGJHbHVaWE11YzNaak1Ga3dFd1lIS29aSXpqMENBUVlJS29aSXpqMERBUWNEUWdBRTY2bmNheW5PCnNvc1Z5cTFCNjFBRTVqbDRJYTZpak52RS9VM0d2THhzcUVUbDNVOU5LSnpoTXlnNk1OaG9IN1lUK2g3cWxlZ3kKK3E4d28vSDNCY2lGcjZPQ0FVWXdnZ0ZDTUE0R0ExVWREd0VCL3dRRUF3SUNoREFkQmdOVkhTVUVGakFVQmdncgpCZ0VGQlFjREFRWUlLd1lCQlFVSEF3SXdEd1lEVlIwVEFRSC9CQVV3QXdFQi96QWRCZ05WSFE0RUZnUVVPTTVQCjhDWUtkN1RSUlBubzh2MzBaanNBbFE0d2dlQUdBMVVkRVFTQjJEQ0IxWUloZEdWcmRHOXVMWFJ5YVdkblpYSnoKTFdOdmNtVXRhVzUwWlhKalpYQjBiM0p6Z2pKMFpXdDBiMjR0ZEhKcFoyZGxjbk10WTI5eVpTMXBiblJsY21ObApjSFJ2Y25NdWRHVnJkRzl1TFhCcGNHVnNhVzVsYzRJMmRHVnJkRzl1TFhSeWFXZG5aWEp6TFdOdmNtVXRhVzUwClpYSmpaWEIwYjNKekxuUmxhM1J2Ymkxd2FYQmxiR2x1WlhNdWMzWmpna1IwWld0MGIyNHRkSEpwWjJkbGNuTXQKWTI5eVpTMXBiblJsY21ObGNIUnZjbk11ZEdWcmRHOXVMWEJwY0dWc2FXNWxjeTV6ZG1NdVkyeDFjM1JsY2k1cwpiMk5oYkRBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlFQWxvdSttQVVGQU5yTDAvRzdmMEhTRWg0U0FBU25LaDUxClpUZTA3ZHJkQWV3Q0lCaUFjczlhL2NWaEtJbEd6Mmo0WkF1UnNNNUgzZnpLL0NBQU5Ec2ZnSUIvCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    url: http://build-id-interceptor.tekton-ci.svc:80/buildid
status:
  address:
    url: http://build-id-interceptor.tekton-ci.svc:80/buildid

For some reason the core interceptor was restarted:

tekton-triggers-controller-7b5fbd8886-thx5k          1/1     Running   0          28h
tekton-triggers-core-interceptors-6d44899696-n8tr5   1/1     Running   0          26m
tekton-triggers-webhook-54c696b579-6zhh6             1/1     Running   0          3h20m

I wonder if the process becomes overloaded and is kiled by k8s? Just a wild guess

[afrittoli]

The good news is that, regardless of the caBundle, the event listener keeps working