TestBrokerTracing test fails

vaikas commented 3 years ago

Describe the bug

TestBrokerTracing failed to get a trace here: https://github.com/mattmoor/mink/runs/1701843155?check_suite_focus=true#step:11:42

        tracing_test_helper.go:90: Unable to get trace "7dc09e59099b1a78e430e9364e8877f0": timeout getting JSONTrace, most recent error:<nil>. Trace so far [{"timestamp":1610628967596933,"duration":404,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"6acb6f4b64b20e18","parentId":"e560f9e026ddbd20","name":"trigger:logger.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp","localEndpoint":{"serviceName":"broker-filter.mink-system","ipv4":"10.96.235.174"},"tags":{"cloudevents.datacontenttype":"application/json","cloudevents.id":"event-1","cloudevents.source":"sender","cloudevents.specversion":"1.0","cloudevents.type":"transformer","messaging.destination":"trigger:logger.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp","messaging.message_id":"event-1","messaging.protocol":"HTTP","messaging.system":"knative"}},{"timestamp":1610628967596852,"duration":500,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"e560f9e026ddbd20","parentId":"a6a7aa384a5528af","name":"/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/logger/21ca5ec1-d52b-403c-a2b7-2027a8c668e8","kind":"SERVER","localEndpoint":{"serviceName":"broker-filter.mink-system","ipv4":"10.96.235.174"},"annotations":[{"timestamp":1610628967596868,"value":"Received 35 bytes"}],"tags":{"http.host":"broker-filter.mink-system.svc.c485363944.local","http.method":"POST","http.path":"/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/logger/21ca5ec1-d52b-403c-a2b7-2027a8c668e8","http.status_code":"200","http.url":"/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/logger/21ca5ec1-d52b-403c-a2b7-2027a8c668e8","http.user_agent":"Go-http-client/1.1","opencensus.status_description":"OK"}},{"timestamp":1610628967708079,"duration":111,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"74a05c20df5cb006","parentId":"ef35e6b5a1875f0f","name":"trigger:transformer.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp","localEndpoint":{"serviceName":"broker-filter.mink-system","ipv4":"10.96.235.174"},"tags":{"cloudevents.datacontenttype":"application/json","cloudevents.id":"event-1","cloudevents.source":"sender","cloudevents.specversion":"1.0","cloudevents.type":"logger","messaging.destination":"trigger:transformer.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp","messaging.message_id":"event-1","messaging.protocol":"HTTP","messaging.system":"knative"}},{"timestamp":1610628967708009,"duration":195,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"ef35e6b5a1875f0f","parentId":"f52e5dae1c0ea8ce","name":"/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/transformer/4764700e-5e4f-45ea-8866-c504440d6669","kind":"SERVER","localEndpoint":{"serviceName":"broker-filter.mink-system","ipv4":"10.96.235.174"},"annotations":[{"timestamp":1610628967708020,"value":"Received 35 bytes"}],"tags":{"http.host":"broker-filter.mink-system.svc.c485363944.local","http.method":"POST","http.path":"/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/transformer/4764700e-5e4f-45ea-8866-c504440d6669","http.status_code":"200","http.url":"/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/transformer/4764700e-5e4f-45ea-8866-c504440d6669","http.user_agent":"Go-http-client/1.1","opencensus.status_description":"OK"}},{"timestamp":1610628967584913,"duration":170,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"c23569f5d1349678","parentId":"1811f72a53074821","name":"/","kind":"SERVER","localEndpoint":{"serviceName":"imc-dispatcher","ipv4":"10.96.145.20"},"annotations":[{"timestamp":1610628967584923,"value":"Received 35 bytes"}],"tags":{"http.host":"br-kne-trigger-kn-channel.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp.svc.c485363944.local","http.method":"POST","http.path":"/","http.status_code":"202","http.url":"/","http.user_agent":"Go-http-client/1.1","opencensus.status_description":"OK"}},{"timestamp":1610628967596221,"duration":4935,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"a6a7aa384a5528af","parentId":"2d4bda272ccd8361","name":"/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/logger/21ca5ec1-d52b-403c-a2b7-2027a8c668e8","kind":"CLIENT","localEndpoint":{"serviceName":"imc-dispatcher","ipv4":"10.96.145.20"},"tags":{"http.host":"broker-filter.mink-system.svc.c485363944.local","http.method":"POST","http.path":"/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/logger/21ca5ec1-d52b-403c-a2b7-2027a8c668e8","http.status_code":"200","http.url":"http://broker-filter.mink-system.svc.c485363944.local/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/logger/21ca5ec1-d52b-403c-a2b7-2027a8c668e8","opencensus.status_description":"OK"}},{"timestamp":1610628967596085,"duration":5132,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"2d4bda272ccd8361","parentId":"c23569f5d1349678","name":"knative.dev","kind":"CLIENT","localEndpoint":{"serviceName":"imc-dispatcher","ipv4":"10.96.145.20"},"tags":{"cloudevents.id":"event-1","cloudevents.source":"sender","cloudevents.specversion":"1.0","cloudevents.type":"transformer"}},{"timestamp":1610628967586068,"duration":88587,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"3b923906f0bc3ac6","parentId":"c23569f5d1349678","name":"knative.dev","kind":"CLIENT","localEndpoint":{"serviceName":"imc-dispatcher","ipv4":"10.96.145.20"},"tags":{"cloudevents.id":"event-1","cloudevents.source":"sender","cloudevents.specversion":"1.0","cloudevents.type":"transformer"}},{"timestamp":1610628967586110,"duration":88808,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"b4ee09170e45df13","parentId":"3b923906f0bc3ac6","name":"/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/transformer/4764700e-5e4f-45ea-8866-c504440d6669","kind":"CLIENT","localEndpoint":{"serviceName":"imc-dispatcher","ipv4":"10.96.145.20"},"tags":{"http.host":"broker-filter.mink-system.svc.c485363944.local","http.method":"POST","http.path":"/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/transformer/4764700e-5e4f-45ea-8866-c504440d6669","http.status_code":"200","http.url":"http://broker-filter.mink-system.svc.c485363944.local/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/transformer/4764700e-5e4f-45ea-8866-c504440d6669","opencensus.status_description":"OK"}},{"timestamp":1610628967704276,"duration":163,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"11bd1b6ba4ed1598","parentId":"65a462075a303452","name":"/","kind":"SERVER","localEndpoint":{"serviceName":"imc-dispatcher","ipv4":"10.96.145.20"},"annotations":[{"timestamp":1610628967704286,"value":"Received 35 bytes"}],"tags":{"http.host":"br-kne-trigger-kn-channel.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp.svc.c485363944.local","http.method":"POST","http.path":"/","http.status_code":"202","http.url":"/","http.user_agent":"Go-http-client/1.1","opencensus.status_description":"OK"}},{"timestamp":1610628967674774,"duration":31351,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"98604b5a8665714a","parentId":"1f047b4968ddccfc","name":"/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/br","kind":"CLIENT","localEndpoint":{"serviceName":"imc-dispatcher","ipv4":"10.96.145.20"},"tags":{"http.host":"broker-ingress.mink-system.svc.c485363944.local","http.method":"POST","http.path":"/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/br","http.status_code":"202","http.url":"http://broker-ingress.mink-system.svc.c485363944.local/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/br","opencensus.status_description":"OK"}},{"timestamp":1610628967674727,"duration":31459,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"1f047b4968ddccfc","parentId":"3b923906f0bc3ac6","name":"knative.dev","kind":"CLIENT","localEndpoint":{"serviceName":"imc-dispatcher","ipv4":"10.96.145.20"},"tags":{"cloudevents.id":"event-1","cloudevents.source":"sender","cloudevents.specversion":"1.0","cloudevents.type":"logger"}},{"timestamp":1610628967705657,"duration":2701,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"f52e5dae1c0ea8ce","parentId":"7cd28c9dfe850381","name":"/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/transformer/4764700e-5e4f-45ea-8866-c504440d6669","kind":"CLIENT","localEndpoint":{"serviceName":"imc-dispatcher","ipv4":"10.96.145.20"},"tags":{"http.host":"broker-filter.mink-system.svc.c485363944.local","http.method":"POST","http.path":"/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/transformer/4764700e-5e4f-45ea-8866-c504440d6669","http.status_code":"200","http.url":"http://broker-filter.mink-system.svc.c485363944.local/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/transformer/4764700e-5e4f-45ea-8866-c504440d6669","opencensus.status_description":"OK"}},{"timestamp":1610628967705552,"duration":2841,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"7cd28c9dfe850381","parentId":"11bd1b6ba4ed1598","name":"knative.dev","kind":"CLIENT","localEndpoint":{"serviceName":"imc-dispatcher","ipv4":"10.96.145.20"},"tags":{"cloudevents.id":"event-1","cloudevents.source":"sender","cloudevents.specversion":"1.0","cloudevents.type":"logger"}},{"timestamp":1610628967704650,"duration":92063,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"0376bc8ce0fd5e33","parentId":"8a19ec7bc275bae5","name":"/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/logger/21ca5ec1-d52b-403c-a2b7-2027a8c668e8","kind":"CLIENT","localEndpoint":{"serviceName":"imc-dispatcher","ipv4":"10.96.145.20"},"tags":{"http.host":"broker-filter.mink-system.svc.c485363944.local","http.method":"POST","http.path":"/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/logger/21ca5ec1-d52b-403c-a2b7-2027a8c668e8","http.status_code":"202","http.url":"http://broker-filter.mink-system.svc.c485363944.local/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/logger/21ca5ec1-d52b-403c-a2b7-2027a8c668e8","opencensus.status_description":"OK"}},{"timestamp":1610628967704603,"duration":92403,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"8a19ec7bc275bae5","parentId":"11bd1b6ba4ed1598","name":"knative.dev","kind":"CLIENT","localEndpoint":{"serviceName":"imc-dispatcher","ipv4":"10.96.145.20"},"tags":{"cloudevents.id":"event-1","cloudevents.source":"sender","cloudevents.specversion":"1.0","cloudevents.type":"logger"}},{"timestamp":1610628967766683,"duration":29317,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"a5c8310c08f30958","parentId":"be4475097a4bc2b5","name":"/","kind":"SERVER","localEndpoint":{"serviceName":"recordevents","ipv4":"10.244.2.25"},"annotations":[{"timestamp":1610628967766693,"value":"Received 35 bytes"}],"tags":{"http.host":"recordevents.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp.svc.c485363944.local","http.method":"POST","http.path":"/","http.status_code":"202","http.url":"/","http.user_agent":"Go-http-client/1.1","opencensus.status_description":"OK"}},{"timestamp":1610628967636711,"duration":37116,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"a368b9ce48a467a2","parentId":"8d67d13d95ea4453","name":"/","kind":"SERVER","localEndpoint":{"serviceName":"transformer","ipv4":"10.244.1.27"},"annotations":[{"timestamp":1610628967636721,"value":"Received 35 bytes"},{"timestamp":1610628967673786,"value":"Sent 35 bytes"}],"tags":{"http.host":"transformer.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp.svc.c485363944.local","http.method":"POST","http.path":"/","http.status_code":"200","http.url":"/","http.user_agent":"Go-http-client/1.1","opencensus.status_description":"OK"}},{"timestamp":1610628967508684,"duration":76628,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"1811f72a53074821","parentId":"7f1e9f102e606a4e","kind":"CLIENT","localEndpoint":{"serviceName":"broker-ingress.mink-system","ipv4":"10.96.95.252"},"tags":{"http.host":"br-kne-trigger-kn-channel.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp.svc.c485363944.local","http.method":"POST","http.path":"","http.status_code":"202","http.url":"http://br-kne-trigger-kn-channel.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp.svc.c485363944.local","opencensus.status_description":"OK"}},{"timestamp":1610628967508516,"duration":76910,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"7f1e9f102e606a4e","parentId":"e62b47f608b98c7b","name":"broker:br.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp","localEndpoint":{"serviceName":"broker-ingress.mink-system","ipv4":"10.96.95.252"},"tags":{"cloudevents.datacontenttype":"application/json","cloudevents.id":"event-1","cloudevents.source":"sender","cloudevents.specversion":"1.0","cloudevents.type":"transformer","messaging.destination":"broker:br.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp","messaging.message_id":"event-1","messaging.protocol":"HTTP","messaging.system":"knative"}},{"timestamp":1610628967508395,"duration":77040,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"e62b47f608b98c7b","parentId":"06220831e15ea802","name":"/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/br","kind":"SERVER","localEndpoint":{"serviceName":"broker-ingress.mink-system","ipv4":"10.96.95.252"},"annotations":[{"timestamp":1610628967508418,"value":"Received 35 bytes"}],"tags":{"http.host":"broker-ingress.mink-system.svc.c485363944.local","http.method":"POST","http.path":"/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/br","http.status_code":"202","http.url":"/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/br","http.user_agent":"Go-http-client/1.1","opencensus.status_description":"OK"}},{"timestamp":1610628967586937,"duration":87412,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"8d67d13d95ea4453","parentId":"d21df0f948cac587","name":"/","kind":"CLIENT","localEndpoint":{"serviceName":"broker-filter.mink-system","ipv4":"10.96.235.174"},"tags":{"http.host":"transformer.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp.svc.c485363944.local","http.method":"POST","http.path":"/","http.status_code":"200","http.url":"http://transformer.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp.svc.c485363944.local/","opencensus.status_description":"OK"}},{"timestamp":1610628967586623,"duration":87789,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"d21df0f948cac587","parentId":"17d40eb6fca946bc","name":"trigger:transformer.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp","localEndpoint":{"serviceName":"broker-filter.mink-system","ipv4":"10.96.235.174"},"tags":{"cloudevents.datacontenttype":"application/json","cloudevents.id":"event-1","cloudevents.source":"sender","cloudevents.specversion":"1.0","cloudevents.type":"transformer","messaging.destination":"trigger:transformer.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp","messaging.message_id":"event-1","messaging.protocol":"HTTP","messaging.system":"knative"}},{"timestamp":1610628967586415,"duration":88009,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"17d40eb6fca946bc","parentId":"b4ee09170e45df13","name":"/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/transformer/4764700e-5e4f-45ea-8866-c504440d6669","kind":"SERVER","localEndpoint":{"serviceName":"broker-filter.mink-system","ipv4":"10.96.235.174"},"annotations":[{"timestamp":1610628967586428,"value":"Received 35 bytes"},{"timestamp":1610628967674380,"value":"Sent 35 bytes"}],"tags":{"http.host":"broker-filter.mink-system.svc.c485363944.local","http.method":"POST","http.path":"/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/transformer/4764700e-5e4f-45ea-8866-c504440d6669","http.status_code":"200","http.url":"/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/transformer/4764700e-5e4f-45ea-8866-c504440d6669","http.user_agent":"Go-http-client/1.1","opencensus.status_description":"OK"}},{"timestamp":1610628967705111,"duration":91153,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"be4475097a4bc2b5","parentId":"03fb93c52d2b43ea","name":"/","kind":"CLIENT","localEndpoint":{"serviceName":"broker-filter.mink-system","ipv4":"10.96.235.174"},"tags":{"http.host":"recordevents.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp.svc.c485363944.local","http.method":"POST","http.path":"/","http.status_code":"202","http.url":"http://recordevents.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp.svc.c485363944.local/","opencensus.status_description":"OK"}},{"timestamp":1610628967704893,"duration":91412,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"03fb93c52d2b43ea","parentId":"48b1b281e10ac41e","name":"trigger:logger.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp","localEndpoint":{"serviceName":"broker-filter.mink-system","ipv4":"10.96.235.174"},"tags":{"cloudevents.datacontenttype":"application/json","cloudevents.id":"event-1","cloudevents.source":"sender","cloudevents.specversion":"1.0","cloudevents.type":"logger","messaging.destination":"trigger:logger.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp","messaging.message_id":"event-1","messaging.protocol":"HTTP","messaging.system":"knative"}},{"timestamp":1610628967704833,"duration":91487,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"48b1b281e10ac41e","parentId":"0376bc8ce0fd5e33","name":"/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/logger/21ca5ec1-d52b-403c-a2b7-2027a8c668e8","kind":"SERVER","localEndpoint":{"serviceName":"broker-filter.mink-system","ipv4":"10.96.235.174"},"annotations":[{"timestamp":1610628967704845,"value":"Received 35 bytes"}],"tags":{"http.host":"broker-filter.mink-system.svc.c485363944.local","http.method":"POST","http.path":"/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/logger/21ca5ec1-d52b-403c-a2b7-2027a8c668e8","http.status_code":"202","http.url":"/triggers/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/logger/21ca5ec1-d52b-403c-a2b7-2027a8c668e8","http.user_agent":"Go-http-client/1.1","opencensus.status_description":"OK"}},{"timestamp":1610628967675640,"duration":30261,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"65a462075a303452","parentId":"58a41db61e057661","kind":"CLIENT","localEndpoint":{"serviceName":"broker-ingress.mink-system","ipv4":"10.96.95.252"},"tags":{"http.host":"br-kne-trigger-kn-channel.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp.svc.c485363944.local","http.method":"POST","http.path":"","http.status_code":"202","http.url":"http://br-kne-trigger-kn-channel.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp.svc.c485363944.local","opencensus.status_description":"OK"}},{"timestamp":1610628967675546,"duration":30445,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"58a41db61e057661","parentId":"4ba4d864e3d9b770","name":"broker:br.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp","localEndpoint":{"serviceName":"broker-ingress.mink-system","ipv4":"10.96.95.252"},"tags":{"cloudevents.datacontenttype":"application/json","cloudevents.id":"event-1","cloudevents.source":"sender","cloudevents.specversion":"1.0","cloudevents.type":"logger","messaging.destination":"broker:br.test-broker-tracing-in-memory-channel-messaging-knative-dec26fp","messaging.message_id":"event-1","messaging.protocol":"HTTP","messaging.system":"knative"}},{"timestamp":1610628967675206,"duration":30803,"traceId":"7dc09e59099b1a78e430e9364e8877f0","id":"4ba4d864e3d9b770","parentId":"98604b5a8665714a","name":"/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/br","kind":"SERVER","localEndpoint":{"serviceName":"broker-ingress.mink-system","ipv4":"10.96.95.252"},"tags":{"http.host":"broker-ingress.mink-system.svc.c485363944.local","http.method":"POST","http.path":"/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/br","http.status_code":"202","http.url":"/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/br","http.user_agent":"Go-http-client/1.1","opencensus.status_description":"OK"}}]

Expected behavior Tests to pass

To Reproduce First time I've seen this fail.

Knative release version HEAD

Additional context Add any other context about the problem here such as proposed priority

vaikas commented 3 years ago

@xtreme-sameer-vohra would you mind taking a look since you were playing with traces for IMC?

xtreme-sameer-vohra commented 3 years ago

Assuming this failed due to the same reason as #4745 And is now happy in the latest run

@vaikas going to close this but feel free to re-open if I didn't address the problem.

xtreme-sameer-vohra commented 3 years ago

/close

knative-prow-robot commented 3 years ago

@xtreme-sameer-vohra: Closing this issue.

In response to [this](https://github.com/knative/eventing/issues/4744#issuecomment-760365462): >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

vaikas commented 3 years ago

Failed again: https://github.com/mattmoor/mink/runs/1706367255?check_suite_focus=true#step:11:22

xtreme-sameer-vohra commented 3 years ago

/assign

xtreme-sameer-vohra commented 3 years ago

Span references a parentID 8ac255f4ed4407e6 that isn't present in the list of spans captured in the latest failure

{
    "timestamp": 1610686222424325,
    "duration": 2045499,
    "traceId": "0aaf0da82a519fe156732f59b460a352",
    "id": "4f4471391eb85d35",
    "parentId": "8ac255f4ed4407e6",
    "name": "/test-broker-tracing-in-memory-channel-messaging-knative-demk484/br",
    "kind": "SERVER",
    "localEndpoint": {
      "serviceName": "broker-ingress.mink-system",
      "ipv4": "10.96.61.2"
    },
    "annotations": [
      {
        "timestamp": 1610686222425551,
        "value": "Received 35 bytes"
      }
    ],
    "tags": {
      "http.host": "broker-ingress.mink-system.svc.c487043867.local",
      "http.method": "POST",
      "http.path": "/test-broker-tracing-in-memory-channel-messaging-knative-demk484/br",
      "http.status_code": "202",
      "http.url": "/test-broker-tracing-in-memory-channel-messaging-knative-demk484/br",
      "http.user_agent": "Go-http-client/1.1",
      "opencensus.status_description": "OK"
    }
  },

which then caused a failure here The span above with timestamp 1610686222424325 was the oldest span captured, its parent wasn't.

It looks like the same issue in the previous failure where span with ID 06220831e15ea802 is missing

  {
    "timestamp": 1610628967508395,
    "duration": 77040,
    "traceId": "7dc09e59099b1a78e430e9364e8877f0",
    "id": "e62b47f608b98c7b",
    "parentId": "06220831e15ea802",
    "name": "/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/br",
    "kind": "SERVER",
    "localEndpoint": {
      "serviceName": "broker-ingress.mink-system",
      "ipv4": "10.96.95.252"
    },
    "annotations": [
      {
        "timestamp": 1610628967508418,
        "value": "Received 35 bytes"
      }
    ],
    "tags": {
      "http.host": "broker-ingress.mink-system.svc.c485363944.local",
      "http.method": "POST",
      "http.path": "/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/br",
      "http.status_code": "202",
      "http.url": "/test-broker-tracing-in-memory-channel-messaging-knative-dec26fp/br",
      "http.user_agent": "Go-http-client/1.1",
      "opencensus.status_description": "OK"
    }
  },

The span above with timestamp 1610628967508395 was the oldest span captured, its parent wasn't.

lberk commented 3 years ago

@xtreme-sameer-vohra looking at the related updates, is this something that might be solved/able to close?

xtreme-sameer-vohra commented 3 years ago

Hey @lberk, Unfortunately this was a flake and we don't have a reproducible failure case. It is possible bumping zipkin https://github.com/knative/eventing/pull/4755 solved the issue 🤷
I took a look through MinKinD Upstream E2Es tests and don't see the issue in the last ~5 days.

@vaikas please feel free to re-open this if you notice the explicit failure no root spans found in the trace

xtreme-sameer-vohra commented 3 years ago

/close

knative-prow-robot commented 3 years ago

@xtreme-sameer-vohra: Closing this issue.

In response to [this](https://github.com/knative/eventing/issues/4744#issuecomment-767247610): >/close > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

knative / eventing

TestBrokerTracing test fails #4744