Closed grantr closed 3 years ago
/priority important-soon
/milestone v0.8.0
in-memory-channel-dispatcher
is using the same logic to manage tracing and it has the same problem. Without zipkin-endpoint
configured, in-memory-channel-dispatcher
pod will get error
when starting up.
$ kubectl get pods -n knative-eventing
NAME READY STATUS RESTARTS AGE
eventing-controller-bd99fd4c6-h7hbd 1/1 Running 0 6h38m
eventing-webhook-589b7cbf6-m5f9q 1/1 Running 0 6h38m
in-memory-channel-controller-67fb687b6d-5q9b2 1/1 Running 0 30s
in-memory-channel-dispatcher-755fc5dd9c-h546d 0/1 Error 2 26s
sources-controller-ffc8988b-kb4wh 1/1 Running 0 6h38m
So the logic to manage tracing with Zipkin shall be reviewed and updated.
Several pods with dependency on Zipkin are affected by this issue, not only broker
and in-memory-channel-dispatcher
. They are:
The root cause is vendor/knative.dev/pkg/tracing/config/trace.go: NewTracingConfigFromMap
returning nil when tracing configmap is malformed. The activator
pod in serving is affected by this issue too.
So we'd better create an issue in serving and fix it there.
/milestone v0.9.0
/project Observability To do
/milestone v0.10.0
This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen
. Mark the issue as
fresh by adding the comment /remove-lifecycle stale
.
Describe the bug If the tracing configmap is updated to include the key
enabled: "true"
but nozipkin-endpoint
key, the Broker pods log an error but keep operating. They will fail to start upon the next restart.Expected behavior The pod should use the default value, or disable tracing and continue. Tracing is an observability feature and its misconfiguration shouldn't cause the Broker to stop working. The fact that the bug is triggered by a restart means the failure will be difficult to associate with the root cause.
To Reproduce
config-tracing
configmap in the Broker's namespace with the keyenabled: "true"
.The Broker pods fail to start.
Knative release version Eventing HEAD, Serving 0.6
Additional context