Closed saisree95 closed 1 year ago
After submitting to dx-demo namespace events reaching all the subscribers
{"level":"debug","ts":"2022-06-27T08:45:35.605Z","logger":"kafkachannel-dispatcher","caller":"channel/message_receiver.go:164","msg":"Received request","knative.dev/pod":"kafka-ch-dispatcher-c79dfd769-d6brj","host":"default-kne-trigger-kn-channel.dx-demo.svc.cluster.local"}
{"level":"debug","ts":"2022-06-27T08:45:35.606Z","logger":"kafkachannel-dispatcher","caller":"channel/message_receiver.go:177","msg":"Request mapped to channel","knative.dev/pod":"kafka-ch-dispatcher-c79dfd769-d6brj","channel":"dx-demo/default-kne-trigger"}
{"level":"debug","ts":"2022-06-27T08:45:35.606Z","logger":"kafkachannel-dispatcher","caller":"dispatcher/dispatcher.go:112","msg":"Received a new message from MessageReceiver, dispatching to Kafka","knative.dev/pod":"kafka-ch-dispatcher-c79dfd769-d6brj","channel":{"Namespace":"dx-demo","Name":"default-kne-trigger"}}
{"level":"debug","ts":"2022-06-27T08:45:35.819Z","logger":"kafkachannel-dispatcher","caller":"dispatcher/dispatcher.go:124","msg":"message sent","knative.dev/pod":"kafka-ch-dispatcher-c79dfd769-d6brj","partition":94,"offset":1}
{"level":"debug","ts":"2022-06-27T08:45:35.825Z","logger":"kafkachannel-dispatcher","caller":"consumer/consumer_handler.go:137","msg":"Consuming Kafka Message","knative.dev/pod":"kafka-ch-dispatcher-c79dfd769-d6brj","knative.dev/controller":"knative.dev.eventing-kafka.pkg.channel.consolidated.reconciler.dispatcher.Reconciler","knative.dev/kind":"messaging.knative.dev.KafkaChannel","knative.dev/traceid":"6d75a8e5-1ba8-4726-94da-e4b63361c169","knative.dev/key":"dx-demo/default-kne-trigger","Headers":{"X-B3-Sampled":["0"],"X-B3-Spanid":["b363f9213582e6b8"],"X-B3-Traceid":["ed5f17f632819e61b363f9213582e6b8"],"X-Request-Id":["11989468-eb5f-446d-819d-351eb548732b"],"ce_gatekeeperkey":["gkr-l7vdt"],"ce_id":["6b0a5ab9-72f1-48f9-a66b-7fd20b13c17c"],"ce_knativearrivaltime":["2022-06-27T08:45:35.588514541Z"],"ce_knativebrokerttl":["255"],"ce_partitionkey":["CDRM_7663"],"ce_source":["dx"],"ce_specversion":["1.0"],"ce_subject":["party/v4/listener/individualCreateEvent"],"ce_time":["2022-06-27T08:45:35Z"],"ce_type":["dxevent"],"content-type":["application/json"],"traceparent":["00-ed5f17f632819e61b363f9213582e6b8-a50868148d833ba6-00"]},"Key":"CDRM_7663","Value":"{\"subject\":\"party/v4/listener/individualCreateEvent\",\"gatekeeperKey\":\"gkr-l7vdt\",\"partitionkey\":\"CDRM_7663\",\"time\":\"2022-06-27T08:45:35Z\",\"id\":\"6b0a5ab9-72f1-48f9-a66b-7fd20b13c17c\",\"type\":\"dxevent\",\"specversion\":\"1.0\",\"source\":\"dx\",\"data\":\"{\\"eventId\\":\\"IndividualCreateEvent_2021-05-21T07:13:04.388743Z\\",\\"eventTime\\":\\"2021-05-21T07:13:04.388743Z\\",\\"eventType\\":\\"IndividualCreateEvent\\",\\"correlationId\\":\\"FAB-8b02c68f-68a9-4b5e-b134-9bc41770fcd0\\",\\"event\\":{\\"individual\\":{\\"partyId\\":\\"300100543240668\\",\\"doNotCallFlag\\":false,\\"doNotEmailFlag\\":false,\\"lastUpdateDate\\":\\"2021-05-21T07:12:58.857Z\\",\\"creationDate\\":\\"2021-05-21T07:12:55.002Z\\",\\"createdBy\\":\\"kolisetty.sidhardha@oracle.com\\",\\"lastUpdatedBy\\":\\"kolisetty.sidhardha@oracle.com\\",\\"partyNumber\\":\\"CDRM_9311133\\",\\"id\\":\\"CDRM_7663\\",\\"href\\":\\"https://144.25.34.150/api/01/party/v4/individual/CDRM_931181\\\",\\\"familyName\\\":\\\"Kolisetty\\\",\\\"fullName\\\":\\\"Sidhardha Kolisetty\\",\\"givenName\\":\\"Sidhardha\\",\\"status\\":\\"initialized\\",\\"@type\\":\\"IndividualOracle\\"}}}\"}","Topic":"knative-messaging-kafka.dx-demo.default-kne-trigger","Partition":94,"Offset":1}
{"level":"debug","ts":"2022-06-27T08:45:35.826Z","logger":"kafkachannel-dispatcher","caller":"dispatcher/consumer_message_handler.go:69","msg":"Going to dispatch the message{topic 15 0 knative-messaging-kafka.dx-demo.default-kne-trigger
After submitting to namespace dx-dryrun events reaching only one subscriber even though there are many others subscribers
`{"level":"info","ts":"2022-06-27T08:46:17.846Z","logger":"kafkachannel-dispatcher","caller":"consumer/consumer_handler.go:162","msg":"Failure while handling a me ssage","knative.dev/pod":"kafka-ch-dispatcher-c79dfd769-d6brj","knative.dev/controller":"knative.dev.eventing-kafka.pkg.channel.consolidated.reconciler.dispatcher.Reconciler","knative.dev/kind":"messaging.knative.dev.KafkaChannel","knative.dev/traceid":"79fcc3a7-f34f-4692-8451-3cd04a70edf6","knative.dev/key":"dx-demo/default-kne-trigger","topic":"knative-messaging-kafka.dx-demo.default-kne-trigger","partition":94,"offset":1,"error":"unable to complete request to http://broker-filter.knative-eventing.svc.cluster.local/triggers/dx-demo/gen-lr-m992x-x5p85/229bd5a4-90cc-42e0-af71-df7bee5846c5: unexpected HTTP response, expected 2xx, got 502"} {"level":"debug","ts":"2022-06-27T08:46:17.847Z","logger":"kafkachannel-dispatcher","caller":"dispatcher/kafka_subscription.go:46","msg":"Setting subscription readiness","knative.dev/pod":"kafka-ch-dispatcher-c79dfd769-d6brj","subscription":"80d0f7a2-9054-4c63-a3bf-b7c52df13eb0","ready":false} {"level":"debug","ts":"2022-06-27T08:46:17.847Z","logger":"kafkachannel-dispatcher","caller":"dispatcher/kafka_subscription.go:59","msg":"Ejecting cached ready subscription","knative.dev/pod":"kafka-ch-dispatcher-c79dfd769-d6brj","subscription":"80d0f7a2-9054-4c63-a3bf-b7c52df13eb0","partition":94} {"level":"warn","ts":"2022-06-27T08:46:17.847Z","logger":"kafkachannel-dispatcher","caller":"dispatcher/dispatcher.go:315","msg":"Error in consumer group","knative.dev/pod":"kafka-ch-dispatcher-c79dfd769-d6brj","error":"unable to complete request to http://broker-filter.knative-eventing.svc.cluster.local/triggers/dx-demo/gen-lr-m992x-x5p85/229bd5a4-90cc-42e0-af71-df7bee5846c5: unexpected HTTP response, expected 2xx, got 502"}
{"level":"debug","ts":"2022-06-27T08:46:18.376Z","logger":"kafkachannel-dispatcher","caller":"channel/message_receiver.go:164","msg":"Received request","knative.dev/pod":"kafka-ch-dispatcher-c79dfd769-d6brj","host":"default-kne-trigger-kn-channel.dx-dryrun.svc.cluster.local"}
{"level":"debug","ts":"2022-06-27T08:46:18.376Z","logger":"kafkachannel-dispatcher","caller":"channel/message_receiver.go:177","msg":"Request mapped to channel","knative.dev/pod":"kafka-ch-dispatcher-c79dfd769-d6brj","channel":"dx-dryrun/default-kne-trigger"}
{"level":"debug","ts":"2022-06-27T08:46:18.376Z","logger":"kafkachannel-dispatcher","caller":"dispatcher/dispatcher.go:112","msg":"Received a new message from MessageReceiver, dispatching to Kafka","knative.dev/pod":"kafka-ch-dispatcher-c79dfd769-d6brj","channel":{"Namespace":"dx-dryrun","Name":"default-kne-trigger"}}
{"level":"debug","ts":"2022-06-27T08:46:18.383Z","logger":"kafkachannel-dispatcher","caller":"dispatcher/dispatcher.go:124","msg":"message sent","knative.dev/pod":"kafka-ch-dispatcher-c79dfd769-d6brj","partition":82,"offset":3}
{"level":"debug","ts":"2022-06-27T08:46:18.386Z","logger":"kafkachannel-dispatcher","caller":"consumer/consumer_handler.go:137","msg":"Consuming Kafka Message","knative.dev/pod":"kafka-ch-dispatcher-c79dfd769-d6brj","knative.dev/controller":"knative.dev.eventing-kafka.pkg.channel.consolidated.reconciler.dispatcher.Reconciler","knative.dev/kind":"messaging.knative.dev.KafkaChannel","knative.dev/traceid":"0180aa5f-8659-4878-be75-d85612ced9da","knative.dev/key":"dx-dryrun/default-kne-trigger","Headers":{"X-B3-Sampled":["0"],"X-B3-Spanid":["6d95e6d7f5423ddf"],"X-B3-Traceid":["5131d10f139a01a06d95e6d7f5423ddf"],"X-Request-Id":["848df6ae-eed0-43ca-843a-de046195f55b"],"ce_gatekeeperkey":["gkr-rkd7c"],"ce_id":["6e8d2066-9636-4e5a-a722-7ac8c5d9b501"],"ce_knativearrivaltime":["2022-06-27T08:46:18.320874601Z"],"ce_knativebrokerttl":["255"],"ce_partitionkey":["IndividualCreateEvent_2021-05-21T07:13:04.388743Z"],"ce_source":["dx"],"ce_specversion":["1.0"],"ce_subject":["party/v4/listener/individualCreateEvent"],"ce_time":["2022-06-27T08:46:18Z"],"ce_type":["dxevent"],"content-type":["application/json"],"traceparent":["00-5131d10f139a01a06d95e6d7f5423ddf-71c7ba7dc2367bcb-00"]},"Key":"IndividualCreateEvent_2021-05-21T07:13:04.388743Z","Value":"{\"subject\":\"party/v4/listener/individualCreateEvent\",\"gatekeeperKey\":\"gkr-rkd7c\",\"partitionkey\":\"IndividualCreateEvent_2021-05-21T07:13:04.388743Z\",\"time\":\"2022-06-27T08:46:18Z\",\"id\":\"6e8d2066-9636-4e5a-a722-7ac8c5d9b501\",\"type\":\"dxevent\",\"specversion\":\"1.0\",\"source\":\"dx\",\"data\":\"{\\"eventId\\":\\"IndividualCreateEvent_2021-05-21T07:13:04.388743Z\\",\\"eventTime\\":\\"2021-05-21T07:13:04.388743Z\\",\\"eventType\\":\\"IndividualCreateEvent\\",\\"correlationId\\":\\"FAB-8b02c68f-68a9-4b5e-b134-9bc41770fcd0\\",\\"event\\":{\\"individual\\":{\\"partyId\\":\\"300100543240668\\",\\"doNotCallFlag\\":false,\\"doNotEmailFlag\\":false,\\"lastUpdateDate\\":\\"2021-05-21T07:12:58.857Z\\",\\"creationDate\\":\\"2021-05-21T07:12:55.002Z\\",\\"createdBy\\":\\"kolisetty.sidhardha@oracle.com\\",\\"lastUpdatedBy\\":\\"kolisetty.sidhardha@oracle.com\\",\\"partyNumber\\":\\"CDRM_9311133\\",\\"id\\":\\"CDRM_7664\\",\\"href\\":\\"https://144.25.34.150/api/01/party/v4/individual/CDRM_931181\\\",\\\"familyName\\\":\\\"Kolisetty\\\",\\\"fullName\\\":\\\"Sidhardha Kolisetty\\",\\"givenName\\":\\"Sidhardha\\",\\"status\\":\\"initialized\\",\\"@type\\":\\"IndividualOracle\\"}}}\"}","Topic":"knative-messaging-kafka.dx-dryrun.default-kne-trigger","Partition":82,"Offset":3}
{"level":"debug","ts":"2022-06-27T08:46:18.387Z","logger":"kafkachannel-dispatcher","caller":"dispatcher/consumer_message_handler.go:69","msg":"Going to dispatch the message{topic 15 0 knative-messaging-kafka.dx-dryrun.default-kne-trigger
This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen
. Mark the issue as
fresh by adding the comment /remove-lifecycle stale
.
Describe the bug Intermittently we see an issue in which , When the triggers are present and configured as needed with status in TRUE state , we don't see the events reaching our subscriber URI for some of the triggers. Recreating the trigger again was the only way to make it work. Tried to restart events pods but it did not help.
Also , when we saw this intermittent issue , in one of the namespace the events were posted to all subscriber URI and working as expected. In another namespace it was posting events to only some of the subscribers.
To Reproduce Could not find an exact way to reproduce. It is intermittent.
Knative release version Version : 1.3