Closed vaikas closed 3 years ago
Failed again here: https://github.com/knative/eventing/pull/4729/checks?check_run_id=1694274951
Run kubectl -v 9 wait pod --for=condition=Ready -n ${SYSTEM_NAMESPACE} -l '!job-name'
kubectl -v 9 wait pod --for=condition=Ready -n ${SYSTEM_NAMESPACE} -l '!job-name'
shell: /bin/bash --noprofile --norc -e -o pipefail {0}
env:
GOPATH: /home/runner/work/eventing/eventing
GO111MODULE: off
KO_DOCKER_REPO: kind.local
SYSTEM_NAMESPACE: knative-eventing
CLUSTER_SUFFIX: c482504125.local
ARTIFACTS: /home/runner/work/eventing/eventing/artifacts
GOROOT: /opt/hostedtoolcache/go/1.15.6/x64
I0113 10:36:52.886381 25628 loader.go:375] Config loaded from file: /home/runner/.kube/config
I0113 10:36:52.892363 25628 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?labelSelector=%21job-name'
I0113 10:36:52.914279 25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?labelSelector=%21job-name 200 OK in 21 milliseconds
I0113 10:36:52.914645 25628 round_trippers.go:449] Response Headers:
I0113 10:36:52.914658 25628 round_trippers.go:452] Cache-Control: no-cache, private
I0113 10:36:52.914662 25628 round_trippers.go:452] Content-Type: application/json
I0113 10:36:52.914665 25628 round_trippers.go:452] Date: Wed, 13 Jan 2021 10:36:52 GMT
I0113 10:36:52.915130 25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"2902"},"items":[{"metadata":{"name":"chaosduck-f665c9dc7-zkdbf","generateName":"chaosduck-f665c9dc7-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/chaosduck-f665c9dc7-zkdbf","uid":"c2c2d30a-a755-4a67-9289-beb684e8fbce","resourceVersion":"1892","creationTimestamp":"2021-01-13T10:35:41Z","deletionTimestamp":"2021-01-13T10:36:14Z","deletionGracePeriodSeconds":30,"labels":{"app":"chaosduck","pod-template-hash":"f665c9dc7"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"chaosduck-f665c9dc7","uid":"be93d508-83a9-428f-94a5-a218a6835edc","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"chaosduck-token-lwvbb","secret":{"secretName":"chaosduck-token-lwvbb","defaultMode":420}}],"containers":[{"name":"chaosduck","image":"kind.local/knative.dev/pkg/leaderelection/chaosduck:967fb52fb64b86752de93a2833a15b705c07660c858b0b27a4fd4fd89094663a","args":["-disable=imc-controller","-disable=imc-dispatcher"],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}}],"resources":{},"volumeMounts":[{"name":"chaosduck-token-lwvbb","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"chaosduck","serviceAccount":"chaosduck","nodeName":"kind-worker","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:42Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:45Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:45Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:41Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.10","podIPs":[{"ip":"10.244.1.10"}],"startTime":"2021-01-13T10:35:42Z","containerStatuses":[{"name":"chaosduck","state":{"running":{"startedAt":"2021-01-13T10:35:44Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/pkg/leaderelection/chaosduck:967fb52fb64b86752de93a2833a15b705c07660c858b0b27a4fd4fd89094663a","imageID":"sha256:9ff310726fe092fe547795899132afed3195fd2651a9202a863738a1682934dd","containerID":"containerd://7d90d940150d5ce8ebb702417f38e4c0f8a4b9dc578f1e374de3c83313cf90dd","started":true}],"qosClass":"BestEffort"}},{"metadata":{"name":"eventing-controller-c895f6756-cf6dg","generateName":"eventing-controller-c895f6756-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/eventing-controller-c895f6756-cf6dg","uid":"3827b98b-f6ac-4551-b24e-f6d336cd8e53","resourceVersion":"2645","creationTimestamp":"2021-01-13T10:36:28Z","labels":{"app":"eventing-controller","eventing.knative.dev/release":"devel","pod-template-hash":"c895f6756"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"eventing-controller-c895f6756","uid":"93116175-b845-4b5c-94fe-9c4668bd0ef5","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"eventing-controller-token-6hxjg","secret":{"secretName":"eventing-controller-token-6hxjg","defaultMode":420}}],"containers":[{"name":"eventing-controller","image":"kind.local/knative.dev/eventing/cmd/controller:a695ea7ccc558e8ab6434dcdd1a7c5cb4eabb84c3cd596235473b2e59ef18158","ports":[{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/eventing"},{"name":"APISERVER_RA_IMAGE","value":"kind.local/knative.dev/eventing/cmd/apiserver_receive_adapter:8d4f85c32cc62b2731fd86a859d7305d261f94d6471e2f8f6db165a02c70c888"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{"requests":{"cpu":"100m","memory":"100Mi"}},"volumeMounts":[{"name":"eventing-controller-token-6hxjg","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"eventing-controller","serviceAccount":"eventing-controller","nodeName":"kind-worker","securityContext":{},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"app":"eventing-controller"}},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:30Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:35Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:35Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:28Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.18","podIPs":[{"ip":"10.244.1.18"}],"startTime":"2021-01-13T10:36:30Z","containerStatuses":[{"name":"eventing-controller","state":{"running":{"startedAt":"2021-01-13T10:36:34Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/controller:a695ea7ccc558e8ab6434dcdd1a7c5cb4eabb84c3cd596235473b2e59ef18158","imageID":"sha256:c9b859f4ecfcd11881660bcb48e03d44ad10991335bc4aee9d53a4894f148bfc","containerID":"containerd://50c64ef119ae94f0de6df42a6dc16ea384bd1eb5111b95d183d3389ce46016ff","started":true}],"qosClass":"Burstable"}},{"metadata":{"name":"eventing-webhook-bc5f465f4-mv6r6","generateName":"eventing-webhook-bc5f465f4-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/eventing-webhook-bc5f465f4-mv6r6","uid":"f0aed95c-93c4-4ae0-b1a4-add0bd06aac5","resourceVersion":"2595","creationTimestamp":"2021-01-13T10:35:44Z","deletionTimestamp":"2021-01-13T10:41:28Z","deletionGracePeriodSeconds":300,"labels":{"app":"eventing-webhook","pod-template-hash":"bc5f465f4","role":"eventing-webhook"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"eventing-webhook-bc5f465f4","uid":"a3fb1319-83e9-4a2c-927b-339c71893048","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"eventing-webhook-token-cmvdm","secret":{"secretName":"eventing-webhook-token-cmvdm","defaultMode":420}}],"containers":[{"name":"eventing-webhook","image":"kind.local/knative.dev/eventing/cmd/webhook:169708841d7a6371b096e69b3015117a3ea94aa52f33a23861151bb0d7d229a7","ports":[{"name":"https-webhook","containerPort":8443,"protocol":"TCP"},{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"METRICS_DOMAIN","value":"knative.dev/eventing"},{"name":"WEBHOOK_NAME","value":"eventing-webhook"},{"name":"WEBHOOK_PORT","value":"8443"},{"name":"SINK_BINDING_SELECTION_MODE","value":"exclusion"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{"limits":{"cpu":"200m","memory":"200Mi"},"requests":{"cpu":"20m","memory":"20Mi"}},"volumeMounts":[{"name":"eventing-webhook-token-cmvdm","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/","port":8443,"scheme":"HTTPS","httpHeaders":[{"name":"k-kubelet-probe","value":"webhook"}]},"initialDelaySeconds":20,"timeoutSeconds":1,"periodSeconds":1,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/","port":8443,"scheme":"HTTPS","httpHeaders":[{"name":"k-kubelet-probe","value":"webhook"}]},"timeoutSeconds":1,"periodSeconds":1,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":300,"dnsPolicy":"ClusterFirst","serviceAccountName":"eventing-webhook","serviceAccount":"eventing-webhook","nodeName":"kind-worker","securityContext":{},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"app":"eventing-webhook"}},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null, [truncated 30407 chars]
I0113 10:36:52.918471 25628 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dchaosduck-f665c9dc7-zkdbf'
I0113 10:36:52.925233 25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dchaosduck-f665c9dc7-zkdbf 200 OK in 6 milliseconds
I0113 10:36:52.925247 25628 round_trippers.go:449] Response Headers:
I0113 10:36:52.925252 25628 round_trippers.go:452] Cache-Control: no-cache, private
I0113 10:36:52.925255 25628 round_trippers.go:452] Content-Type: application/json
I0113 10:36:52.925258 25628 round_trippers.go:452] Content-Length: 3092
I0113 10:36:52.925261 25628 round_trippers.go:452] Date: Wed, 13 Jan 2021 10:36:52 GMT
I0113 10:36:52.925394 25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"2902"},"items":[{"metadata":{"name":"chaosduck-f665c9dc7-zkdbf","generateName":"chaosduck-f665c9dc7-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/chaosduck-f665c9dc7-zkdbf","uid":"c2c2d30a-a755-4a67-9289-beb684e8fbce","resourceVersion":"1892","creationTimestamp":"2021-01-13T10:35:41Z","deletionTimestamp":"2021-01-13T10:36:14Z","deletionGracePeriodSeconds":30,"labels":{"app":"chaosduck","pod-template-hash":"f665c9dc7"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"chaosduck-f665c9dc7","uid":"be93d508-83a9-428f-94a5-a218a6835edc","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"chaosduck-token-lwvbb","secret":{"secretName":"chaosduck-token-lwvbb","defaultMode":420}}],"containers":[{"name":"chaosduck","image":"kind.local/knative.dev/pkg/leaderelection/chaosduck:967fb52fb64b86752de93a2833a15b705c07660c858b0b27a4fd4fd89094663a","args":["-disable=imc-controller","-disable=imc-dispatcher"],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}}],"resources":{},"volumeMounts":[{"name":"chaosduck-token-lwvbb","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"chaosduck","serviceAccount":"chaosduck","nodeName":"kind-worker","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:42Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:45Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:45Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:41Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.10","podIPs":[{"ip":"10.244.1.10"}],"startTime":"2021-01-13T10:35:42Z","containerStatuses":[{"name":"chaosduck","state":{"running":{"startedAt":"2021-01-13T10:35:44Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/pkg/leaderelection/chaosduck:967fb52fb64b86752de93a2833a15b705c07660c858b0b27a4fd4fd89094663a","imageID":"sha256:9ff310726fe092fe547795899132afed3195fd2651a9202a863738a1682934dd","containerID":"containerd://7d90d940150d5ce8ebb702417f38e4c0f8a4b9dc578f1e374de3c83313cf90dd","started":true}],"qosClass":"BestEffort"}}]}
I0113 10:36:52.926378 25628 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-controller-c895f6756-cf6dg'
I0113 10:36:52.928973 25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-controller-c895f6756-cf6dg 200 OK in 2 milliseconds
I0113 10:36:52.929023 25628 round_trippers.go:449] Response Headers:
I0113 10:36:52.929072 25628 round_trippers.go:452] Cache-Control: no-cache, private
I0113 10:36:52.929083 25628 round_trippers.go:452] Content-Type: application/json
I0113 10:36:52.929146 25628 round_trippers.go:452] Content-Length: 3952
I0113 10:36:52.929159 25628 round_trippers.go:452] Date: Wed, 13 Jan 2021 10:36:52 GMT
I0113 10:36:52.929242 25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"2902"},"items":[{"metadata":{"name":"eventing-controller-c895f6756-cf6dg","generateName":"eventing-controller-c895f6756-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/eventing-controller-c895f6756-cf6dg","uid":"3827b98b-f6ac-4551-b24e-f6d336cd8e53","resourceVersion":"2645","creationTimestamp":"2021-01-13T10:36:28Z","labels":{"app":"eventing-controller","eventing.knative.dev/release":"devel","pod-template-hash":"c895f6756"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"eventing-controller-c895f6756","uid":"93116175-b845-4b5c-94fe-9c4668bd0ef5","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"eventing-controller-token-6hxjg","secret":{"secretName":"eventing-controller-token-6hxjg","defaultMode":420}}],"containers":[{"name":"eventing-controller","image":"kind.local/knative.dev/eventing/cmd/controller:a695ea7ccc558e8ab6434dcdd1a7c5cb4eabb84c3cd596235473b2e59ef18158","ports":[{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/eventing"},{"name":"APISERVER_RA_IMAGE","value":"kind.local/knative.dev/eventing/cmd/apiserver_receive_adapter:8d4f85c32cc62b2731fd86a859d7305d261f94d6471e2f8f6db165a02c70c888"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{"requests":{"cpu":"100m","memory":"100Mi"}},"volumeMounts":[{"name":"eventing-controller-token-6hxjg","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"eventing-controller","serviceAccount":"eventing-controller","nodeName":"kind-worker","securityContext":{},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"app":"eventing-controller"}},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:30Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:35Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:35Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:28Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.18","podIPs":[{"ip":"10.244.1.18"}],"startTime":"2021-01-13T10:36:30Z","containerStatuses":[{"name":"eventing-controller","state":{"running":{"startedAt":"2021-01-13T10:36:34Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/controller:a695ea7ccc558e8ab6434dcdd1a7c5cb4eabb84c3cd596235473b2e59ef18158","imageID":"sha256:c9b859f4ecfcd11881660bcb48e03d44ad10991335bc4aee9d53a4894f148bfc","containerID":"containerd://50c64ef119ae94f0de6df42a6dc16ea384bd1eb5111b95d183d3389ce46016ff","started":true}],"qosClass":"Burstable"}}]}
I0113 10:36:52.930211 25628 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-bc5f465f4-mv6r6'
pod/chaosduck-f665c9dc7-zkdbf condition met
pod/eventing-controller-c895f6756-cf6dg condition met
I0113 10:36:52.932381 25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-bc5f465f4-mv6r6 200 OK in 2 milliseconds
I0113 10:36:52.932399 25628 round_trippers.go:449] Response Headers:
I0113 10:36:52.932403 25628 round_trippers.go:452] Cache-Control: no-cache, private
I0113 10:36:52.932407 25628 round_trippers.go:452] Content-Type: application/json
I0113 10:36:52.932410 25628 round_trippers.go:452] Date: Wed, 13 Jan 2021 10:36:52 GMT
I0113 10:36:52.932555 25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"2902"},"items":[{"metadata":{"name":"eventing-webhook-bc5f465f4-mv6r6","generateName":"eventing-webhook-bc5f465f4-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/eventing-webhook-bc5f465f4-mv6r6","uid":"f0aed95c-93c4-4ae0-b1a4-add0bd06aac5","resourceVersion":"2595","creationTimestamp":"2021-01-13T10:35:44Z","deletionTimestamp":"2021-01-13T10:41:28Z","deletionGracePeriodSeconds":300,"labels":{"app":"eventing-webhook","pod-template-hash":"bc5f465f4","role":"eventing-webhook"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"eventing-webhook-bc5f465f4","uid":"a3fb1319-83e9-4a2c-927b-339c71893048","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"eventing-webhook-token-cmvdm","secret":{"secretName":"eventing-webhook-token-cmvdm","defaultMode":420}}],"containers":[{"name":"eventing-webhook","image":"kind.local/knative.dev/eventing/cmd/webhook:169708841d7a6371b096e69b3015117a3ea94aa52f33a23861151bb0d7d229a7","ports":[{"name":"https-webhook","containerPort":8443,"protocol":"TCP"},{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"METRICS_DOMAIN","value":"knative.dev/eventing"},{"name":"WEBHOOK_NAME","value":"eventing-webhook"},{"name":"WEBHOOK_PORT","value":"8443"},{"name":"SINK_BINDING_SELECTION_MODE","value":"exclusion"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{"limits":{"cpu":"200m","memory":"200Mi"},"requests":{"cpu":"20m","memory":"20Mi"}},"volumeMounts":[{"name":"eventing-webhook-token-cmvdm","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/","port":8443,"scheme":"HTTPS","httpHeaders":[{"name":"k-kubelet-probe","value":"webhook"}]},"initialDelaySeconds":20,"timeoutSeconds":1,"periodSeconds":1,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/","port":8443,"scheme":"HTTPS","httpHeaders":[{"name":"k-kubelet-probe","value":"webhook"}]},"timeoutSeconds":1,"periodSeconds":1,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":300,"dnsPolicy":"ClusterFirst","serviceAccountName":"eventing-webhook","serviceAccount":"eventing-webhook","nodeName":"kind-worker","securityContext":{},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"app":"eventing-webhook"}},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:44Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:31Z","reason":"ContainersNotReady","message":"containers with unready status: [eventing-webhook]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:31Z","reason":"ContainersNotReady","message":"containers with unready status: [eventing-webhook]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:44Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.15","podIPs":[{"ip":"10.244.1.15"}],"startTime":"2021-01-13T10:35:44Z","containerStatuses":[{"name":"eventing-webhook","state":{"running":{"startedAt":"2021-01-13T10:35:46Z"}},"lastState":{},"ready":false,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/webhook:169708841d7a6371b096e69b3015117a3ea94aa52f33a23861151bb0d7d229a7","imageID":"sha256:3ca196d729701392f6b0dd1adc6838612cf64e20cbadfa3d3bac820622342de5","containerID":"containerd://95b7a4b38426f0c6762988a6ea9abc39f68db70e4c4667326ebb1488a5d58f41","started":true}],"qosClass":"Burstable"}}]}
I0113 10:36:52.933010 25628 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-bc5f465f4-mv6r6&resourceVersion=2902&watch=true'
I0113 10:36:52.934812 25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-bc5f465f4-mv6r6&resourceVersion=2902&watch=true 200 OK in 1 milliseconds
I0113 10:36:52.934825 25628 round_trippers.go:449] Response Headers:
I0113 10:36:52.934829 25628 round_trippers.go:452] Cache-Control: no-cache, private
I0113 10:36:52.934833 25628 round_trippers.go:452] Content-Type: application/json
I0113 10:36:52.934836 25628 round_trippers.go:452] Date: Wed, 13 Jan 2021 10:36:52 GMT
I0113 10:37:22.935488 25628 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-bc5f465f4-zrq9g'
I0113 10:37:22.937585 25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-bc5f465f4-zrq9g 200 OK in 2 milliseconds
I0113 10:37:22.937604 25628 round_trippers.go:449] Response Headers:
I0113 10:37:22.937609 25628 round_trippers.go:452] Cache-Control: no-cache, private
I0113 10:37:22.937612 25628 round_trippers.go:452] Content-Type: application/json
I0113 10:37:22.937616 25628 round_trippers.go:452] Date: Wed, 13 Jan 2021 10:37:22 GMT
I0113 10:37:22.937773 25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3326"},"items":[{"metadata":{"name":"eventing-webhook-bc5f465f4-zrq9g","generateName":"eventing-webhook-bc5f465f4-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/eventing-webhook-bc5f465f4-zrq9g","uid":"6acec832-0948-4356-b8bc-9e0f64f9427e","resourceVersion":"2670","creationTimestamp":"2021-01-13T10:36:28Z","labels":{"app":"eventing-webhook","pod-template-hash":"bc5f465f4","role":"eventing-webhook"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"eventing-webhook-bc5f465f4","uid":"a3fb1319-83e9-4a2c-927b-339c71893048","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"eventing-webhook-token-cmvdm","secret":{"secretName":"eventing-webhook-token-cmvdm","defaultMode":420}}],"containers":[{"name":"eventing-webhook","image":"kind.local/knative.dev/eventing/cmd/webhook:169708841d7a6371b096e69b3015117a3ea94aa52f33a23861151bb0d7d229a7","ports":[{"name":"https-webhook","containerPort":8443,"protocol":"TCP"},{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"METRICS_DOMAIN","value":"knative.dev/eventing"},{"name":"WEBHOOK_NAME","value":"eventing-webhook"},{"name":"WEBHOOK_PORT","value":"8443"},{"name":"SINK_BINDING_SELECTION_MODE","value":"exclusion"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{"limits":{"cpu":"200m","memory":"200Mi"},"requests":{"cpu":"20m","memory":"20Mi"}},"volumeMounts":[{"name":"eventing-webhook-token-cmvdm","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/","port":8443,"scheme":"HTTPS","httpHeaders":[{"name":"k-kubelet-probe","value":"webhook"}]},"initialDelaySeconds":20,"timeoutSeconds":1,"periodSeconds":1,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/","port":8443,"scheme":"HTTPS","httpHeaders":[{"name":"k-kubelet-probe","value":"webhook"}]},"timeoutSeconds":1,"periodSeconds":1,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":300,"dnsPolicy":"ClusterFirst","serviceAccountName":"eventing-webhook","serviceAccount":"eventing-webhook","nodeName":"kind-worker","securityContext":{},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"app":"eventing-webhook"}},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:31Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:36Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:36Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:29Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.19","podIPs":[{"ip":"10.244.1.19"}],"startTime":"2021-01-13T10:36:31Z","containerStatuses":[{"name":"eventing-webhook","state":{"running":{"startedAt":"2021-01-13T10:36:35Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/webhook:169708841d7a6371b096e69b3015117a3ea94aa52f33a23861151bb0d7d229a7","imageID":"sha256:3ca196d729701392f6b0dd1adc6838612cf64e20cbadfa3d3bac820622342de5","containerID":"containerd://1b05f6547a8c6073eeff892307a053e2c3b3bed2d94b1bc9453dde45796d332a","started":true}],"qosClass":"Burstable"}}]}
I0113 10:37:22.938466 25628 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dimc-controller-676bd4f69-856x5'
pod/eventing-webhook-bc5f465f4-zrq9g condition met
I0113 10:37:22.944534 25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dimc-controller-676bd4f69-856x5 200 OK in 6 milliseconds
pod/imc-controller-676bd4f69-856x5 condition met
I0113 10:37:22.944553 25628 round_trippers.go:449] Response Headers:
I0113 10:37:22.944557 25628 round_trippers.go:452] Cache-Control: no-cache, private
I0113 10:37:22.944561 25628 round_trippers.go:452] Content-Type: application/json
I0113 10:37:22.944564 25628 round_trippers.go:452] Content-Length: 3632
I0113 10:37:22.944568 25628 round_trippers.go:452] Date: Wed, 13 Jan 2021 10:37:22 GMT
I0113 10:37:22.944594 25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3326"},"items":[{"metadata":{"name":"imc-controller-676bd4f69-856x5","generateName":"imc-controller-676bd4f69-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/imc-controller-676bd4f69-856x5","uid":"c9f03481-08c5-4283-858a-a2919365613d","resourceVersion":"1526","creationTimestamp":"2021-01-13T10:35:28Z","labels":{"messaging.knative.dev/channel":"in-memory-channel","messaging.knative.dev/role":"controller","pod-template-hash":"676bd4f69"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"imc-controller-676bd4f69","uid":"21816286-b917-4b9b-83d4-d934758322ee","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"imc-controller-token-5m8c8","secret":{"secretName":"imc-controller-token-5m8c8","defaultMode":420}}],"containers":[{"name":"controller","image":"kind.local/knative.dev/eventing/cmd/in_memory/channel_controller:a6d5565bf562e93a1f0911a82e7a513ee77535879002b5b2d72303fe1eb59b12","ports":[{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/inmemorychannel-controller"},{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"DISPATCHER_IMAGE","value":"kind.local/knative.dev/eventing/cmd/in_memory/channel_dispatcher:32b708f1d26d514c6459873cff70a513aca349e7d775709b058a11ffdc3a5147"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{},"volumeMounts":[{"name":"imc-controller-token-5m8c8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"imc-controller","serviceAccount":"imc-controller","nodeName":"kind-worker","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:28Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:31Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:31Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:28Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.9","podIPs":[{"ip":"10.244.1.9"}],"startTime":"2021-01-13T10:35:28Z","containerStatuses":[{"name":"controller","state":{"running":{"startedAt":"2021-01-13T10:35:31Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/in_memory/channel_controller:latest","imageID":"sha256:b28daf1512a5c35ee6c6ba69b18a86e3b3c2d8a824f5f5828994fa84b0a9afee","containerID":"containerd://62df33837bbd4e833be11ef1b529256ef5a7c03fd8a3be301a85ea9f36a0b811","started":true}],"qosClass":"BestEffort"}}]}
I0113 10:37:22.945167 25628 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dimc-dispatcher-576559d88f-j58p9'
pod/imc-dispatcher-576559d88f-j58p9 condition met
I0113 10:37:22.950680 25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dimc-dispatcher-576559d88f-j58p9 200 OK in 5 milliseconds
pod/mt-broker-controller-84fc78474-jzbdg condition met
I0113 10:37:22.950689 25628 round_trippers.go:449] Response Headers:
pod/mt-broker-filter-69f97b6df5-v6x6v condition met
I0113 10:37:22.950693 25628 round_trippers.go:452] Cache-Control: no-cache, private
pod/mt-broker-ingress-5d6df454bf-pnpm6 condition met
I0113 10:37:22.950697 25628 round_trippers.go:452] Content-Type: application/json
pod/sugar-controller-59f989cc7-ldn8w condition met
I0113 10:37:22.950700 25628 round_trippers.go:452] Content-Length: 4080
pod/zipkin-7fcb9b4ff8-xljj9 condition met
I0113 10:37:22.950703 25628 round_trippers.go:452] Date: Wed, 13 Jan 2021 10:37:22 GMT
I0113 10:37:22.950731 25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3326"},"items":[{"metadata":{"name":"imc-dispatcher-576559d88f-j58p9","generateName":"imc-dispatcher-576559d88f-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/imc-dispatcher-576559d88f-j58p9","uid":"b3798024-dd26-4a01-afdc-be8c4d0fa755","resourceVersion":"1584","creationTimestamp":"2021-01-13T10:35:28Z","labels":{"messaging.knative.dev/channel":"in-memory-channel","messaging.knative.dev/role":"dispatcher","pod-template-hash":"576559d88f"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"imc-dispatcher-576559d88f","uid":"608cd40f-4f67-45a5-b2d4-69e6a16448cf","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"imc-dispatcher-token-blmn7","secret":{"secretName":"imc-dispatcher-token-blmn7","defaultMode":420}}],"containers":[{"name":"dispatcher","image":"kind.local/knative.dev/eventing/cmd/in_memory/channel_dispatcher:32b708f1d26d514c6459873cff70a513aca349e7d775709b058a11ffdc3a5147","ports":[{"name":"http","containerPort":8080,"protocol":"TCP"},{"name":"metrics","containerPort":9090,"protocol":"TCP"}],"env":[{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/inmemorychannel-dispatcher"},{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"CONTAINER_NAME","value":"dispatcher"}],"resources":{},"volumeMounts":[{"name":"imc-dispatcher-token-blmn7","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},"initialDelaySeconds":5,"timeoutSeconds":1,"periodSeconds":2,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},"timeoutSeconds":1,"periodSeconds":2,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"imc-dispatcher","serviceAccount":"imc-dispatcher","nodeName":"kind-worker","securityContext":{},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"messaging.knative.dev/channel":"in-memory-channel","messaging.knative.dev/role":"dispatcher"}},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:28Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:33Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:33Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:28Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.8","podIPs":[{"ip":"10.244.1.8"}],"startTime":"2021-01-13T10:35:28Z","containerStatuses":[{"name":"dispatcher","state":{"running":{"startedAt":"2021-01-13T10:35:31Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/in_memory/channel_dispatcher:latest","imageID":"sha256:b4d2446c83a01b0762c1c907e5337f83d77e2137c6d0d560467d1bb0b0a68888","containerID":"containerd://e29c0fcb2dea4a6bb02513408230e200bf674cff4720a1669287d23ecabedd1a","started":true}],"qosClass":"BestEffort"}}]}
I0113 10:37:22.951375 25628 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dmt-broker-controller-84fc78474-jzbdg'
I0113 10:37:22.952825 25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dmt-broker-controller-84fc78474-jzbdg 200 OK in 1 milliseconds
I0113 10:37:22.952835 25628 round_trippers.go:449] Response Headers:
I0113 10:37:22.952839 25628 round_trippers.go:452] Date: Wed, 13 Jan 2021 10:37:22 GMT
I0113 10:37:22.952842 25628 round_trippers.go:452] Cache-Control: no-cache, private
I0113 10:37:22.952845 25628 round_trippers.go:452] Content-Type: application/json
I0113 10:37:22.952848 25628 round_trippers.go:452] Content-Length: 3857
I0113 10:37:22.952873 25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3326"},"items":[{"metadata":{"name":"mt-broker-controller-84fc78474-jzbdg","generateName":"mt-broker-controller-84fc78474-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/mt-broker-controller-84fc78474-jzbdg","uid":"42cd9a3c-9a43-4a50-873e-885764bc289f","resourceVersion":"2612","creationTimestamp":"2021-01-13T10:36:28Z","labels":{"app":"mt-broker-controller","eventing.knative.dev/release":"devel","pod-template-hash":"84fc78474"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"mt-broker-controller-84fc78474","uid":"eb855823-b150-4f3b-b8b1-412cb896f24f","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"eventing-controller-token-6hxjg","secret":{"secretName":"eventing-controller-token-6hxjg","defaultMode":420}}],"containers":[{"name":"mt-broker-controller","image":"kind.local/knative.dev/eventing/cmd/mtchannel_broker:371a2ce1db2b6af3b71f1bb6248035c293e7ca5f247f674c362398f1ec49868a","ports":[{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/eventing"},{"name":"BROKER_INJECTION_DEFAULT","value":"false"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{"requests":{"cpu":"100m","memory":"100Mi"}},"volumeMounts":[{"name":"eventing-controller-token-6hxjg","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"eventing-controller","serviceAccount":"eventing-controller","nodeName":"kind-worker","securityContext":{},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"app":"mt-broker-controller"}},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:29Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:34Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:34Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:28Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.16","podIPs":[{"ip":"10.244.1.16"}],"startTime":"2021-01-13T10:36:29Z","containerStatuses":[{"name":"mt-broker-controller","state":{"running":{"startedAt":"2021-01-13T10:36:34Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/mtchannel_broker:371a2ce1db2b6af3b71f1bb6248035c293e7ca5f247f674c362398f1ec49868a","imageID":"sha256:137b256f9100db6159c8906661c029814bd4aadcb53cf99fee4875f61644011f","containerID":"containerd://44a2d4c111f9292f0c52c49fa3d63b566c6cfa4ce0a6a7a3864a70ffa886e813","started":true}],"qosClass":"Burstable"}}]}
I0113 10:37:22.953442 25628 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dmt-broker-filter-69f97b6df5-v6x6v'
I0113 10:37:22.954856 25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dmt-broker-filter-69f97b6df5-v6x6v 200 OK in 1 milliseconds
I0113 10:37:22.954864 25628 round_trippers.go:449] Response Headers:
I0113 10:37:22.954880 25628 round_trippers.go:452] Cache-Control: no-cache, private
I0113 10:37:22.954883 25628 round_trippers.go:452] Content-Type: application/json
I0113 10:37:22.954889 25628 round_trippers.go:452] Content-Length: 4058
I0113 10:37:22.954892 25628 round_trippers.go:452] Date: Wed, 13 Jan 2021 10:37:22 GMT
I0113 10:37:22.955052 25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3326"},"items":[{"metadata":{"name":"mt-broker-filter-69f97b6df5-v6x6v","generateName":"mt-broker-filter-69f97b6df5-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/mt-broker-filter-69f97b6df5-v6x6v","uid":"cdd763a1-a5b0-437d-9742-16be293a82e1","resourceVersion":"1517","creationTimestamp":"2021-01-13T10:35:23Z","labels":{"eventing.knative.dev/brokerRole":"filter","eventing.knative.dev/release":"devel","pod-template-hash":"69f97b6df5"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"mt-broker-filter-69f97b6df5","uid":"cba052aa-09a4-4ed4-8a89-3a897d933412","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"mt-broker-filter-token-vsvrp","secret":{"secretName":"mt-broker-filter-token-vsvrp","defaultMode":420}}],"containers":[{"name":"filter","image":"kind.local/knative.dev/eventing/cmd/mtbroker/filter:1215b68433d1ab8979e55e5ea38c5f78c37818ce9f3873d0ec659ebe76bcddc8","ports":[{"name":"http","containerPort":8080,"protocol":"TCP"},{"name":"metrics","containerPort":9092,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"CONTAINER_NAME","value":"filter"},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/internal/eventing"},{"name":"FILTER_PORT","value":"8080"}],"resources":{"requests":{"cpu":"100m","memory":"100Mi"}},"volumeMounts":[{"name":"mt-broker-filter-token-vsvrp","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},"initialDelaySeconds":5,"timeoutSeconds":1,"periodSeconds":2,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},"timeoutSeconds":1,"periodSeconds":2,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"mt-broker-filter","serviceAccount":"mt-broker-filter","nodeName":"kind-worker","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:23Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:31Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:31Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:23Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.4","podIPs":[{"ip":"10.244.1.4"}],"startTime":"2021-01-13T10:35:23Z","containerStatuses":[{"name":"filter","state":{"running":{"startedAt":"2021-01-13T10:35:28Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/mtbroker/filter:1215b68433d1ab8979e55e5ea38c5f78c37818ce9f3873d0ec659ebe76bcddc8","imageID":"sha256:60f93ac8439bca5e71de1c154db2368dea719b63425ae43d6691cd212a825705","containerID":"containerd://5fe88480ab2ed8384e10246857aefd98756a821eb40126db54ff2d76df5a6a8b","started":true}],"qosClass":"Burstable"}}]}
I0113 10:37:22.955843 25628 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dmt-broker-ingress-5d6df454bf-pnpm6'
I0113 10:37:22.958044 25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dmt-broker-ingress-5d6df454bf-pnpm6 200 OK in 2 milliseconds
I0113 10:37:22.958052 25628 round_trippers.go:449] Response Headers:
I0113 10:37:22.958056 25628 round_trippers.go:452] Content-Length: 4016
I0113 10:37:22.958060 25628 round_trippers.go:452] Date: Wed, 13 Jan 2021 10:37:22 GMT
I0113 10:37:22.958063 25628 round_trippers.go:452] Cache-Control: no-cache, private
I0113 10:37:22.958066 25628 round_trippers.go:452] Content-Type: application/json
I0113 10:37:22.958093 25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3326"},"items":[{"metadata":{"name":"mt-broker-ingress-5d6df454bf-pnpm6","generateName":"mt-broker-ingress-5d6df454bf-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/mt-broker-ingress-5d6df454bf-pnpm6","uid":"0c2f4674-3da3-4e4a-8fa1-4dafd2eb855b","resourceVersion":"1579","creationTimestamp":"2021-01-13T10:35:23Z","labels":{"eventing.knative.dev/brokerRole":"ingress","eventing.knative.dev/release":"devel","pod-template-hash":"5d6df454bf"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"mt-broker-ingress-5d6df454bf","uid":"2f3dee37-c0f8-494a-8448-857b467dc315","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"mt-broker-ingress-token-4k524","secret":{"secretName":"mt-broker-ingress-token-4k524","defaultMode":420}}],"containers":[{"name":"ingress","image":"kind.local/knative.dev/eventing/cmd/mtbroker/ingress:52b8032f8f0ccabcc25b8a9a526a338d7e50a173e4b480097e278dda8fa59871","ports":[{"name":"http","containerPort":8080,"protocol":"TCP"},{"name":"metrics","containerPort":9092,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"CONTAINER_NAME","value":"ingress"},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/internal/eventing"},{"name":"INGRESS_PORT","value":"8080"}],"resources":{"requests":{"cpu":"100m","memory":"100Mi"}},"volumeMounts":[{"name":"mt-broker-ingress-token-4k524","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},"initialDelaySeconds":5,"timeoutSeconds":1,"periodSeconds":2,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},"timeoutSeconds":1,"periodSeconds":2,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"mt-broker-ingress","serviceAccount":"mt-broker-ingress","nodeName":"kind-worker","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:25Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:33Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:33Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:23Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.5","podIPs":[{"ip":"10.244.1.5"}],"startTime":"2021-01-13T10:35:25Z","containerStatuses":[{"name":"ingress","state":{"running":{"startedAt":"2021-01-13T10:35:29Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/mtbroker/ingress:latest","imageID":"sha256:538f35844d92c5172fcdbf148cd0a2b6bbd6adea6d43d043452eb013d2814a79","containerID":"containerd://a9f5f06fbeccfb51d7f002962fe8c7eec7711731ed7b8fea851733bb0106d25c","started":true}],"qosClass":"Burstable"}}]}
I0113 10:37:22.958710 25628 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dsugar-controller-59f989cc7-ldn8w'
I0113 10:37:22.960182 25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dsugar-controller-59f989cc7-ldn8w 200 OK in 1 milliseconds
I0113 10:37:22.960190 25628 round_trippers.go:449] Response Headers:
I0113 10:37:22.960194 25628 round_trippers.go:452] Cache-Control: no-cache, private
I0113 10:37:22.960197 25628 round_trippers.go:452] Content-Type: application/json
I0113 10:37:22.960278 25628 round_trippers.go:452] Content-Length: 3476
I0113 10:37:22.960282 25628 round_trippers.go:452] Date: Wed, 13 Jan 2021 10:37:22 GMT
I0113 10:37:22.960352 25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3326"},"items":[{"metadata":{"name":"sugar-controller-59f989cc7-ldn8w","generateName":"sugar-controller-59f989cc7-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/sugar-controller-59f989cc7-ldn8w","uid":"82a96098-e9fd-48e4-b61e-685c73196bf0","resourceVersion":"2609","creationTimestamp":"2021-01-13T10:36:28Z","labels":{"eventing.knative.dev/role":"sugar-controller","pod-template-hash":"59f989cc7"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"sugar-controller-59f989cc7","uid":"6ba17fce-4163-41e8-90dd-4584e2278069","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"eventing-controller-token-6hxjg","secret":{"secretName":"eventing-controller-token-6hxjg","defaultMode":420}}],"containers":[{"name":"controller","image":"kind.local/knative.dev/eventing/cmd/sugar_controller:a34f3e24a6d56caf6fffa73e954f4e5b77682aa0f021d8503d25c0c9bea418e8","ports":[{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/sugar-controller"},{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{},"volumeMounts":[{"name":"eventing-controller-token-6hxjg","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"eventing-controller","serviceAccount":"eventing-controller","nodeName":"kind-worker","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:30Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:34Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:34Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:29Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.17","podIPs":[{"ip":"10.244.1.17"}],"startTime":"2021-01-13T10:36:30Z","containerStatuses":[{"name":"controller","state":{"running":{"startedAt":"2021-01-13T10:36:34Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/sugar_controller:a34f3e24a6d56caf6fffa73e954f4e5b77682aa0f021d8503d25c0c9bea418e8","imageID":"sha256:68097895c55555c0beb9844f3e0ca4abca3f79d4f23218b372066bcc2fc8fe84","containerID":"containerd://86b4971a6963515dc77a2a2a9c5054681939ec3a4aa0ab06a114d0a745019db6","started":true}],"qosClass":"BestEffort"}}]}
I0113 10:37:22.960858 25628 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dzipkin-7fcb9b4ff8-xljj9'
I0113 10:37:22.962181 25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dzipkin-7fcb9b4ff8-xljj9 200 OK in 1 milliseconds
I0113 10:37:22.962206 25628 round_trippers.go:449] Response Headers:
I0113 10:37:22.962213 25628 round_trippers.go:452] Cache-Control: no-cache, private
I0113 10:37:22.962217 25628 round_trippers.go:452] Content-Type: application/json
I0113 10:37:22.962220 25628 round_trippers.go:452] Content-Length: 2895
I0113 10:37:22.962223 25628 round_trippers.go:452] Date: Wed, 13 Jan 2021 10:37:22 GMT
I0113 10:37:22.962429 25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3326"},"items":[{"metadata":{"name":"zipkin-7fcb9b4ff8-xljj9","generateName":"zipkin-7fcb9b4ff8-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/zipkin-7fcb9b4ff8-xljj9","uid":"62d7e52c-bb16-414b-9154-4d45655c1d04","resourceVersion":"2174","creationTimestamp":"2021-01-13T10:35:42Z","labels":{"app":"zipkin","pod-template-hash":"7fcb9b4ff8"},"annotations":{"sidecar.istio.io/inject":"false"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"zipkin-7fcb9b4ff8","uid":"2afd3171-0986-4051-9b40-3defa5be7a13","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-c4sfx","secret":{"secretName":"default-token-c4sfx","defaultMode":420}}],"containers":[{"name":"zipkin","image":"docker.io/openzipkin/zipkin:2.13.0","ports":[{"containerPort":9411,"protocol":"TCP"}],"env":[{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}}],"resources":{"limits":{"memory":"1000Mi"},"requests":{"memory":"256Mi"}},"volumeMounts":[{"name":"default-token-c4sfx","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"kind-worker","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:42Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:04Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:04Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:42Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.11","podIPs":[{"ip":"10.244.1.11"}],"startTime":"2021-01-13T10:35:42Z","containerStatuses":[{"name":"zipkin","state":{"running":{"startedAt":"2021-01-13T10:36:03Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/openzipkin/zipkin:2.13.0","imageID":"docker.io/openzipkin/zipkin@sha256:2f3139c6c2975ce4e53717fc07da628efdf2c0d2b3dbd7de890369b9450fd21d","containerID":"containerd://54aa896f65831a6f59da3923ce4c49a271d6cc6dfd610bc9d8eb7eeb7417bcf4","started":true}],"qosClass":"Burstable"}}]}
F0113 10:37:22.962774 25628 helpers.go:115] error: timed out waiting for the condition on pods/eventing-webhook-bc5f465f4-mv6r6
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0xc000010001, 0xc0000fc7e0, 0x82, 0xd3)
/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:996 +0xb9
k8s.io/klog/v2.(*loggingT).output(0x4d2aec0, 0xc000000003, 0x0, 0x0, 0xc000b18690, 0x48a9af4, 0xa, 0x73, 0x41d400)
/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:945 +0x191
k8s.io/klog/v2.(*loggingT).printDepth(0x4d2aec0, 0x3, 0x0, 0x0, 0x2, 0xc0013b7a10, 0x1, 0x1)
/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:718 +0x165
k8s.io/klog/v2.FatalDepth(...)
/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:1449
k8s.io/kubectl/pkg/cmd/util.fatal(0xc00134d620, 0x53, 0x1)
/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:93 +0x1f0
k8s.io/kubectl/pkg/cmd/util.checkErr(0x347a520, 0xc0015d2180, 0x31c65e8)
/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:188 +0x945
k8s.io/kubectl/pkg/cmd/util.CheckErr(...)
/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:115
k8s.io/kubectl/pkg/cmd/wait.NewCmdWait.func1(0xc000bdadc0, 0xc000b0bf80, 0x1, 0x8)
/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/wait/wait.go:120 +0xad
github.com/spf13/cobra.(*Command).execute(0xc000bdadc0, 0xc000b0bf00, 0x8, 0x8, 0xc000bdadc0, 0xc000b0bf00)
/go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:846 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xc001242b00, 0x7, 0xc001242b00, 0x7)
/go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
/go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:887
main.main()
/go/src/github.com/openshift/oc/cmd/oc/oc.go:110 +0x885
goroutine 4 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x4d2aec0)
/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:1131 +0x8b
created by k8s.io/klog/v2.init.0
/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:416 +0xd8
goroutine 7 [chan receive]:
k8s.io/klog.(*loggingT).flushDaemon(0x4d2ade0)
/go/src/github.com/openshift/oc/vendor/k8s.io/klog/klog.go:1010 +0x8b
created by k8s.io/klog.init.0
/go/src/github.com/openshift/oc/vendor/k8s.io/klog/klog.go:411 +0xd8
goroutine 24 [select]:
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x31c6510, 0x347e280, 0xc0011bf8f0, 0x1, 0xc0000a0360)
/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x149
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x31c6510, 0x12a05f200, 0x0, 0xc0002d6201, 0xc0000a0360)
/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/apimachinery/pkg/util/wait.Forever(0x31c6510, 0x12a05f200)
/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:81 +0x4f
created by k8s.io/component-base/logs.InitLogs
/go/src/github.com/openshift/oc/vendor/k8s.io/component-base/logs/logs.go:58 +0x8a
goroutine 125 [IO wait]:
internal/poll.runtime_pollWait(0x7f7b5407e0d8, 0x72, 0x3482f20)
/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/runtime/netpoll.go:220 +0x55
internal/poll.(*pollDesc).wait(0xc00102ee18, 0x72, 0xc001150000, 0x49b1, 0x49b1)
/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00102ee00, 0xc001150000, 0x49b1, 0x49b1, 0x0, 0x0, 0x0)
/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/internal/poll/fd_unix.go:159 +0x1b1
net.(*netFD).Read(0xc00102ee00, 0xc001150000, 0x49b1, 0x49b1, 0x203000, 0x78699b, 0xc0010d59e0)
/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0000108d0, 0xc001150000, 0x49b1, 0x49b1, 0x0, 0x0, 0x0)
/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/net/net.go:182 +0x8e
crypto/tls.(*atLeastReader).Read(0xc00166a760, 0xc001150000, 0x49b1, 0x49b1, 0xb69, 0x49ac, 0xc001139710)
/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/crypto/tls/conn.go:779 +0x62
bytes.(*Buffer).ReadFrom(0xc0010d5b00, 0x347a260, 0xc00166a760, 0x41d785, 0x2bdcba0, 0x2ed2dc0)
/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc0010d5880, 0x347f7c0, 0xc0000108d0, 0x5, 0xc0000108d0, 0xb58)
/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/crypto/tls/conn.go:801 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc0010d5880, 0x0, 0x0, 0xc001139d18)
/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/crypto/tls/conn.go:608 +0x115
crypto/tls.(*Conn).readRecord(...)
/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/crypto/tls/conn.go:576
crypto/tls.(*Conn).Read(0xc0010d5880, 0xc000fdf000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/crypto/tls/conn.go:1252 +0x15f
bufio.(*Reader).Read(0xc000246840, 0xc0005ded58, 0x9, 0x9, 0xc001139d18, 0x31c9f00, 0x9bbceb)
/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/bufio/bufio.go:227 +0x222
io.ReadAtLeast(0x347a000, 0xc000246840, 0xc0005ded58, 0x9, 0x9, 0x9, 0xc000062070, 0x0, 0x347a520)
/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/io/io.go:314 +0x87
io.ReadFull(...)
/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/io/io.go:333
golang.org/x/net/http2.readFrameHeader(0xc0005ded58, 0x9, 0x9, 0x347a000, 0xc000246840, 0x0, 0x0, 0xc00165ce10, 0x0)
/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/frame.go:237 +0x89
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0005ded20, 0xc00165ce10, 0x0, 0x0, 0x0)
/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/frame.go:492 +0xa5
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc001139fa8, 0x0, 0x0)
/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:1794 +0xd8
golang.org/x/net/http2.(*ClientConn).readLoop(0xc0009da780)
/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:1716 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:695 +0x66e
Error: Process completed with exit code 255.
Ok, so moar logs:
b8769ee1e7671481fe8bbef0b44c51276bf4924898ed0b28aef3fe7c87","containerID":"containerd://4b91885600e42139dab1940c34f59052c14a23b40f1cd61095581a7a18407d5c","started":true}],"qosClass":"Burstable"}}]}
2021-01-12T10:00:59.2281962Z I0112 10:00:59.219514 26156 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-9d555fbd4-hnnbm'
2021-01-12T10:00:59.2286072Z I0112 10:00:59.221086 26156 round_trippers.go:443] GET https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-9d555fbd4-hnnbm 200 OK in 1 milliseconds
2021-01-12T10:00:59.2287553Z I0112 10:00:59.221110 26156 round_trippers.go:449] Response Headers:
2021-01-12T10:00:59.2288508Z I0112 10:00:59.221114 26156 round_trippers.go:452] Content-Length: 140
2021-01-12T10:00:59.2289124Z I0112 10:00:59.221135 26156 round_trippers.go:452] Date: Tue, 12 Jan 2021 10:00:59 GMT
2021-01-12T10:00:59.2289934Z I0112 10:00:59.221141 26156 round_trippers.go:452] Cache-Control: no-cache, private
2021-01-12T10:00:59.2291195Z I0112 10:00:59.221144 26156 round_trippers.go:452] Content-Type: application/json
2021-01-12T10:00:59.2293471Z I0112 10:00:59.221156 26156 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3522"},"items":[]}
2021-01-12T10:00:59.2295809Z I0112 10:00:59.221278 26156 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-9d555fbd4-hnnbm&resourceVersion=3522&watch=true'
2021-01-12T10:00:59.2298840Z I0112 10:00:59.221971 26156 round_trippers.go:443] GET https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-9d555fbd4-hnnbm&resourceVersion=3522&watch=true 200 OK in 0 milliseconds
2021-01-12T10:00:59.2300203Z I0112 10:00:59.221985 26156 round_trippers.go:449] Response Headers:
2021-01-12T10:00:59.2301048Z I0112 10:00:59.221989 26156 round_trippers.go:452] Content-Type: application/json
2021-01-12T10:00:59.2302207Z I0112 10:00:59.221993 26156 round_trippers.go:452] Date: Tue, 12 Jan 2021 10:00:59 GMT
2021-01-12T10:00:59.2303262Z I0112 10:00:59.221996 26156 round_trippers.go:452] Cache-Control: no-cache, private
2021-01-12T10:00:59.2305010Z pod/eventing-controller-67d57bbcb9-c7n6n condition met
2021-01-12T10:01:29.2231482Z I0112 10:01:29.222736 26156 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-9d555fbd4-st8fw'
2021-01-12T10:01:29.2261284Z I0112 10:01:29.225821 26156 round_trippers.go:443] GET https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-9d555fbd4-st8fw 200 OK in 3 milliseconds
2021-01-12T10:01:29.2263069Z I0112 10:01:29.225836 26156 round_trippers.go:449] Response Headers:
2021-01-12T10:01:29.2264303Z I0112 10:01:29.225841 26156 round_trippers.go:452] Cache-Control: no-cache, private
2021-01-12T10:01:29.2265563Z I0112 10:01:29.225845 26156 round_trippers.go:452] Content-Type: application/json
2021-01-12T10:01:29.2266867Z I0112 10:01:29.225848 26156 round_trippers.go:452] Date: Tue, 12 Jan 2021 10:01:29 GMT
2021-01-12T10:01:29.2311596Z I0112 10:01:29.227030 26156 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3944"},"items":[{"metadata":{"name":"eventing-webhook-9d555fbd4-st8fw","generateName":"eventing-webhook-9d555fbd4-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/eventing-webhook-9d555fbd4-st8fw","uid":"57a04376-a85c-4f30-9e0f-c5e032260288","resourceVersion":"2743","creationTimestamp":"2021-01-12T09:59:57Z","labels":{"app":"eventing-webhook","pod-template-hash":"9d555fbd4","role":"eventing-webhook"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"eventing-webhook-9d555fbd4","uid":"c2be7162-0148-475b-ab0a-ae0226596635","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2021-01-12T09:59:57Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:app":{},"f:pod-template-hash":{},"f:role":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"c2be7162-0148-475b-ab0a-ae0226596635\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:affinity":{".":{},"f:podAntiAffinity":{".":{},"f:preferredDuringSchedulingIgnoredDuringExecution":{}}},"f:containers":{"k:{\"name\":\"eventing-webhook\"}":{".":{},"f:env":{".":{},"k:{\"name\":\"CONFIG_LOGGING_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"METRICS_DOMAIN\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"POD_NAME\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}},"k:{\"name\":\"SINK_BINDING_SELECTION_MODE\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"SYSTEM_NAMESPACE\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}},"k:{\"name\":\"WEBHOOK_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"WEBHOOK_PORT\"}":{".":{},"f:name":{},"f:value":{}}},"f:image":{},"f:imagePullPolicy":{},"f:livenessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:httpHeaders":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:initialDelaySeconds":{},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":8008,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{\"containerPort\":8443,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}}},"f:readinessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:httpHeaders":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:resources":{".":{},"f:limits":{".":{},"f:cpu":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:securityContext":{".":{},"f:allowPrivilegeEscalation":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-01-12T10:00:04Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.17\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"eventing-webhook-token-ljck4","secret":{"secretName":"eventing-webhook-token-ljck4","defaultMode":420}}],"containers":[{"name":"eventing-webhook","image":"kind.local/knative.dev/eventing/cmd/webhook:b5a5737377c81d19b91ebf347e1a853232f74273063e4fa8669b2e5b7babfa55","ports":[{"name":"https-webhook","containerPort":8443,"protocol":"TCP"},{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"METRICS_DOMAIN","value":"knative.dev/eventing"},{"name":"WEBHOOK_NAME","value":"eventing-webhook"},{"name":"WEBHOOK_PORT","value":"8443"},{"name":"SINK_BINDING_SELECTION_MODE","value":"exclusion"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{"limits":{"cpu":"200m","memory":"200Mi"},"requests":{"cpu":"20m","memory":"20Mi"}},"volumeMounts":[{"name":"eventing-webhook-token-ljck4","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/","port":8443,"scheme":"HTTPS","httpHeaders":[{"name":"k-kubelet-probe","value":"webhook"}]},"initialDelaySeconds":20,"timeoutSeconds":1,"periodSeconds":1,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/","port":8443,"scheme":"HTTPS","httpHeaders":[{"name":"k-kubelet-probe","value":"webhook"}]},"timeoutSeconds":1,"periodSeconds":1,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":300,"dnsPolicy":"ClusterFirst","serviceAccountName":"eventing-webhook","serviceAccount":"eventing-webhook","nodeName":"kind-worker","securityContext":{},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"app":"eventing-webhook"}},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:58Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T10:00:04Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T10:00:04Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:57Z"}],"hostIP":"172.18.0.2","podIP":"10.244.1.17","podIPs":[{"ip":"10.244.1.17"}],"startTime":"2021-01-12T09:59:58Z","containerStatuses":[{"name":"eventing-webhook","state":{"running":{"startedAt":"2021-01-12T10:00:02Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/webhook:latest","imageID":"sha256:ac72d6366971fda96b76839e59ef67b3f31eb66bc34f6225ec6bbe0a3433f81b","containerID":"containerd://0c8891f22e9e43092d29227f2c6e226147b8fb5beca5c56f20256986655fd7c1","started":true}],"qosClass":"Burstable"}}]}
2021-01-12T10:01:29.2349822Z pod/eventing-webhook-9d555fbd4-st8fw condition met
We're trying to fetch two different webhooks, one is found, other is not.
It fails because the first is not found there:
2021-01-12T10:01:30.2204077Z F0112 10:01:29.266090 26156 helpers.go:115] timed out waiting for the condition on pods/chaosduck-5d8694d86b-7qf9q
2021-01-12T10:01:30.2205303Z timed out waiting for the condition on pods/eventing-webhook-9d555fbd4-hnnbm
And the st8fw pod is found there to be running:
2021-01-12T10:01:30.5300104Z NAME READY STATUS RESTARTS AGE
2021-01-12T10:01:30.5302609Z eventing-controller-67d57bbcb9-c7n6n 1/1 Running 0 93s
2021-01-12T10:01:30.5304221Z eventing-webhook-9d555fbd4-st8fw 1/1 Running 0 93s
2021-01-12T10:01:30.5305729Z imc-controller-7586dff887-t68d6 1/1 Running 0 2m33s
2021-01-12T10:01:30.5307784Z imc-dispatcher-6d95d7b95f-dlkgb 1/1 Running 0 2m33s
2021-01-12T10:01:30.5309366Z mt-broker-controller-5b755c9445-n5gcp 1/1 Running 0 93s
2021-01-12T10:01:30.5311139Z mt-broker-filter-56864974c6-mhndh 1/1 Running 0 2m37s
2021-01-12T10:01:30.5312478Z mt-broker-ingress-6db4d947c6-2kpxk 1/1 Running 0 2m34s
2021-01-12T10:01:30.5313840Z sugar-controller-84b77b5675-hkdsj 1/1 Running 0 93s
2021-01-12T10:01:30.5315209Z zipkin-764f9875c6-w5rg8 1/1 Running 0 2m20s
From timestamps, I'd guess chaos killed them, but I can't seem to find the logs for chaos. Will look into it more. But furthermore will look at the waiting code to see if there's something there that we can look. I'd think we only care that the webhook is up, not that a particular webhook pod with a specific name is up.
Yeah, so the step that does this is:
- name: Wait for things to be up
run: |
kubectl -v 9 wait pod --for=condition=Ready -n ${SYSTEM_NAMESPACE} -l '!job-name'
Which seems to turn into list calls where it is looking for specific pod names in the lists.
I can't seem to find the logs for chaos
chaos is disabled on KinD tests. https://github.com/knative/eventing/blob/d74dc25193d0724f55b2f9c9e14efb5e5b0b56af/.github/workflows/kind-e2e.yaml#L161-L162
I'd think we only care that the webhook is up, not that a particular webhook pod with a specific name is up
Do we have more than one replica?
I'm testing on KinD eventing-kafka-broker and failures I see are never related to the eventing webhook not coming up.
However, since creating a Kafka cluster is slow and contains some dynamic resources to be created by the operator we added a sleep
to prevent, "yes resources are ready but not all are there tho", so I guess eventually the webhook recovers.
@pierDipi hehe, that explains the no logs and disproves my theory :)
Describe the bug
Here's one example https://github.com/knative/eventing/pull/4724/checks?check_run_id=1687176531
Every now and then we observe a failure where the pods are supposedly not coming up even though they are. Working theory is that things get started, chaos duck kills something and somehow the list gets screwed because of this:
Yet, after collecting the system diagnostics we see this:
A clear and concise description of what the bug is.
Expected behavior A clear and concise description of what you expected to happen.
To Reproduce Steps to reproduce the behavior.
Knative release version HEAD
Additional context Add any other context about the problem here such as proposed priority