search

knative-extensions / eventing-natss

NATS streaming integration with Knative Eventing.
Apache License 2.0
40 stars 41 forks source link

1.3.5, 1.4.0, 1.10.2 dispatcher not send response to broker #381

Closed chris93111 closed 11 months ago

chris93111 commented 1 year ago

Describe the bug after upgrade 1.3.0 to 1.3.5, knative sequence with broker not work correctly, the dispatch controller don't accept content-type "application/cloudevents+json", it's generate error dispatcher/consumer.go:73 failed to handle message failed to forward reply to http://broker-ingress.knative-eventing.svc.cluster.local/xxxxxx, specversion: no specversion

So the response message is not send to broker

If content-type is application/json the dispatcher don't have error, but msg is not send to broker

i used the app appender https://github.com/knative/eventing/blob/main/cmd/appender/main.go same as documentation

Expected behavior response msg send to broker

To Reproduce knative service with response cloudevent

Knative release version 1.2.2

astelmashenko commented 1 year ago

I will try to reproduce and fix this issue in context of https://github.com/knative-sandbox/eventing-natss/pull/382

Please correct me if I'm worng, this tutorial https://knative.dev/docs/eventing/flows/sequence/sequence-with-broker-trigger/#create-the-trigger-targeting-the-sequence should reproduce the issue.

chris93111 commented 1 year ago

Hi @astelmashenko yes i play this tutorial

astelmashenko commented 1 year ago

This is what I received locally

{
    "caller": "dispatcher/consumer.go:73",
    "error": "failed to forward reply to http://sequence-kn-sequence-1-kn-jsm-channel.default.svc.cluster.local: Post \"http://sequence-kn-sequence-1-kn-jsm-channel.default.svc.cluster.local\": dial tcp 10.102.254.116:80: i/o timeout",
    "is_leader": true,
    "knative.dev/controller": "knative.dev.eventing-natss.pkg.channel.jetstream.dispatcher.Reconciler",
    "knative.dev/key": "default/sequence-kn-sequence-0",
    "knative.dev/kind": "messaging.knative.dev.NatsJetStreamChannel",
    "knative.dev/pod": "jetstream-ch-dispatcher-7db57f9b49-xjt5h",
    "knative.dev/traceid": "964b22d8-37d5-4655-a0a0-91577f819343",
    "level": "error",
    "logger": "jetstream-channel-dispatcher",
    "msg": "failed to handle message",
    "stacktrace": "knative.dev/eventing-natss/pkg/channel/jetstream/dispatcher.(*Consumer).MsgHandler\n\tknative.dev/eventing-natss/pkg/channel/jetstream/dispatcher/consumer.go:73\ngithub.com/nats-io/nats%2ego.(*js).subscribe.func1\n\tgithub.com/nats-io/nats.go@v1.15.0/js.go:1512\ngithub.com/nats-io/nats%2ego.(*Conn).waitForMsgs\n\tgithub.com/nats-io/nats.go@v1.15.0/nats.go:2715",
    "sub_uid": "bf2808b0-51ae-4f2b-81fa-583e615cf17d",
    "ts": "2022-12-12T10:46:30.394Z"
}

Key thing here is 

failed to forward reply to http://sequence-kn-sequence-1-kn-jsm-channel.default.svc.cluster.local: Post \"http://sequence-kn-sequence-1-kn-jsm-channel.default.svc.cluster.local\": dial tcp 10.102.254.116:80: i/o timeout

I also checked that both binary (application/json) and structured (application/cloudevents+json) events are working if I send directly to channel.

I checked documentation and there is a notice:

Sequence needs "hairpin" traffic. Please verify that your pod can reach itself via the service IP. If the "hairpin" traffic is not available, you can reach out to your cluster administrator since its a cluster level (typically CNI) setting.

Not sure if it is exactly the case of the issue.

chris93111 commented 1 year ago

@astelmashenko it's work with version 1.3.0, it's not a problem with CNI, i don't have the same error

specversion: no specversion thinks events send to broker not formated correctly

chris93111 commented 1 year ago

@astelmashenko Hi no is not work for me the sequence-display don't receive event and in nats the channel as only one event, so the last reply with change type is not send to broker

This is my manifests

apiVersion: eventing.knative.dev/v1
kind: Broker
metadata:
  annotations:
    eventing.knative.dev/broker.class: MTChannelBasedBroker
  name: test
  namespace: cfg-events
spec:
  config:
    apiVersion: v1
    kind: ConfigMap
    name: config-natjsm-channel
    namespace: knative-eventing
  delivery:
    backoffDelay: PT0.2S
    backoffPolicy: exponential
    retry: 2
---
apiVersion: flows.knative.dev/v1
kind: Sequence
metadata:
  name: test
  namespace: cfg-events
spec:
  channelTemplate:
    apiVersion: messaging.knative.dev/v1alpha1
    kind: NatsJetStreamChannel
    spec:
      consumerConfigTemplate:
        ackWait: 30s
        deliverPolicy: New
      stream:
        config:
          duplicateWindow: 0s
          maxAge: 168h
  reply:
    ref:
      apiVersion: eventing.knative.dev/v1
      kind: Broker
      name: test
      namespace: cfg-events
  steps:
    - delivery:
        backoffDelay: PT5S
        backoffPolicy: exponential
        retry: 5
      ref:
        apiVersion: serving.knative.dev/v1
        kind: Service
        name: first
        namespace: cfg-events
    - delivery:
        backoffDelay: PT5S
        backoffPolicy: exponential
        retry: 5
      ref:
        apiVersion: serving.knative.dev/v1
        kind: Service
        name: second
        namespace: cfg-events
    - delivery:
        backoffDelay: PT5S
        backoffPolicy: exponential
        retry: 5
      ref:
        apiVersion: serving.knative.dev/v1
        kind: Service
        name: third
        namespace: cfg-events
---
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: first
  namespace: cfg-events
spec:
  template:
    metadata:
      creationTimestamp: null
    spec:
      containerConcurrency: 0
      containers:
        - env:
            - name: MESSAGE
              value: ' - Handled by 0'
          image: gcr.io/knative-releases/knative.dev/eventing/cmd/appender
          name: user-container
          readinessProbe:
            successThreshold: 1
            tcpSocket:
              port: 0
          resources: {}
      enableServiceLinks: false
      timeoutSeconds: 300
  traffic:
    - latestRevision: true
      percent: 100
---
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: second
  namespace: cfg-events
spec:
  template:
    metadata:
      creationTimestamp: null
    spec:
      containerConcurrency: 0
      containers:
        - env:
            - name: MESSAGE
              value: ' - Handled by 1'
          image: gcr.io/knative-releases/knative.dev/eventing/cmd/appender
          name: user-container
          readinessProbe:
            successThreshold: 1
            tcpSocket:
              port: 0
          resources: {}
      enableServiceLinks: false
      timeoutSeconds: 300
  traffic:
    - latestRevision: true
      percent: 100
---
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: third
  namespace: cfg-events
spec:
  template:
    metadata:
      creationTimestamp: null
    spec:
      containerConcurrency: 0
      containers:
        - env:
            - name: MESSAGE
              value: ' - Handled by 2'
            - name: TYPE
              value: samples.http.mod3
          image: gcr.io/knative-releases/knative.dev/eventing/cmd/appender
          name: user-container
          readinessProbe:
            successThreshold: 1
            tcpSocket:
              port: 0
          resources: {}
      enableServiceLinks: false
      timeoutSeconds: 300
  traffic:
    - latestRevision: true
      percent: 100
---
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: sequence-display
  namespace: cfg-events
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/min-scale: '1'
      creationTimestamp: null
    spec:
      containerConcurrency: 0
      containers:
        - image: gcr.io/knative-releases/knative.dev/eventing/cmd/event_display
          name: user-container
          readinessProbe:
            successThreshold: 1
            tcpSocket:
              port: 0
          resources: {}
      enableServiceLinks: false
      timeoutSeconds: 300
  traffic:
    - latestRevision: true
      percent: 100
---
apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
  name: sequence-trigger
  namespace: cfg-events
spec:
  broker: test
  filter:
    attributes:
      source: test/a
      type: test/b
  subscriber:
    ref:
      apiVersion: flows.knative.dev/v1
      kind: Sequence
      name: test
      namespace: cfg-events
---
apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
  name: display-trigger
  namespace: cfg-events
spec:
  broker: test
  filter:
    attributes:
      type: samples.http.mod3
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: sequence-display
      namespace: cfg-events

In Nats 

│ KN_CFG_EVENTS__TEST_KN_SEQUENCE_0                          │             │ 2022-12-16 19:40:29 │ 1        │ 476 B │ 8.38s        │
│ KN_CFG_EVENTS__TEST_KN_SEQUENCE_1                          │             │ 2022-12-16 19:40:29 │ 1        │ 497 B │ 6.40s        │
│ KN_CFG_EVENTS__TEST_KNE_TRIGGER                            │             │ 2022-12-03 18:52:06 │ 1        │ 499 B │ 8.40s        │
│ KN_CFG_EVENTS__TEST_KN_SEQUENCE_2                          │             │ 2022-12-16 19:40:29 │ 1        │ 512 B │ 4.40s

All channel have one message , normaly the KN_CFG_EVENTS__TEST_KNE_TRIGGER take 2 message, the first and the last in response of last consumer in sequence with change type

~ # nats stream view KN_CFG_EVENTS__TEST_KNE_TRIGGER
[7] Subject: cfg-events.test-kne-trigger._knative Received: 2022-12-16T21:15:33Z

  Nats-Msg-Id: 59b9e9f5-38f0-4df8-a816-0738c935a14f

{"specversion":"1.0","id":"59b9e9f5-38f0-4df8-a816-0738c935a14f","source":"test/a","type":"test/b","datacontenttype":"application/json","time":"2022-12-16T21:15:33.582005568Z","data":{"data":"test"},"knativearrivaltime":"2022-12-16T21:15:33.600884463Z","knativebrokerttl":"255","traceparent":"00-c76ec2a79c286c7a036d95c4e058ab46-0bb2568669a95eb9-00","tracestate":""}

21:20:22 Reached apparent end of data
~ # nats stream view KN_CFG_EVENTS__TEST_KN_SEQUENCE_0 
[7] Subject: cfg-events.test-kn-sequence-0._knative Received: 2022-12-16T21:15:33Z

  Nats-Msg-Id: 59b9e9f5-38f0-4df8-a816-0738c935a14f

{"specversion":"1.0","id":"59b9e9f5-38f0-4df8-a816-0738c935a14f","source":"test/a","type":"test/b","datacontenttype":"application/json","time":"2022-12-16T21:15:33.582005568Z","data":{"data":"test"},"traceparent":"00-c76ec2a79c286c7a036d95c4e058ab46-e857b51d4138a837-00","tracestate":"","knativearrivaltime":"2022-12-16T21:15:33.600884463Z"}

21:20:33 Reached apparent end of data
~ # nats stream view KN_CFG_EVENTS__TEST_KN_SEQUENCE_1
[7] Subject: cfg-events.test-kn-sequence-1._knative Received: 2022-12-16T21:15:35Z

  Nats-Msg-Id: 59b9e9f5-38f0-4df8-a816-0738c935a14f

{"specversion":"1.0","id":"59b9e9f5-38f0-4df8-a816-0738c935a14f","source":"test/a","type":"test/b","datacontenttype":"application/json","time":"2022-12-16T21:15:33.582005568Z","data":{"id":0,"message":" - Handled by 0"},"knativearrivaltime":"2022-12-16T21:15:33.600884463Z","traceparent":"00-c76ec2a79c286c7a036d95c4e058ab46-4e2e57b2df8df171-00","tracestate":""}

21:20:35 Reached apparent end of data
~ # nats stream view KN_CFG_EVENTS__TEST_KN_SEQUENCE_2
[7] Subject: cfg-events.test-kn-sequence-2._knative Received: 2022-12-16T21:15:37Z

  Nats-Msg-Id: 59b9e9f5-38f0-4df8-a816-0738c935a14f

{"specversion":"1.0","id":"59b9e9f5-38f0-4df8-a816-0738c935a14f","source":"test/a","type":"test/b","datacontenttype":"application/json","time":"2022-12-16T21:15:33.582005568Z","data":{"id":0,"message":" - Handled by 0 - Handled by 1"},"knativearrivaltime":"2022-12-16T21:15:33.600884463Z","tracestate":"","traceparent":"00-c76ec2a79c286c7a036d95c4e058ab46-b404f9467ee33aac-00"}

21:20:36 Reached apparent end of data

The logs of third :

2022/12/16 21:37:19 listening on 8080, appending " - Handled by 2" to events
2022/12/16 21:37:19 Received a new event:
2022/12/16 21:37:19 [2022-12-16 21:37:13.141133544 +0000 UTC] test/a test/b: &{Sequence:0 Message: - Handled by 0 - Handled by 1}
2022/12/16 21:37:19 Transform the event to:
2022/12/16 21:37:19 [2022-12-16 21:37:13.141133544 +0000 UTC] test/a samples.http.mod3: &{Sequence:0 Message: - Handled by 0 - Handled by 1 - Handled by 2}
chris93111 commented 1 year ago

@astelmashenko do you find any information of this issue ?

astelmashenko commented 1 year ago

@chris93111 , nope, sorry, I was busy these days.

chris93111 commented 1 year ago

Hello any update ?

github-actions[bot] commented 1 year ago

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.

chris93111 commented 1 year ago

/remove-lifecycle stale

chris93111 commented 11 months ago

Hello, after update knative , i have more logs , it looks like the filter change is causing a problem -

We can see in 1.3.0 dispatcher take reply, but after 1.3.5 is always "is a non event"

{"level":"debug","ts":"2023-09-14T20:53:07.810Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:158","msg":"published","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":"cfg-events/test-kn-sequence-2"}
{"level":"debug","ts":"2023-09-14T20:53:07.810Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:304","msg":"dispatch message","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","destination":"http://third.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T20:53:07.810Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:310","msg":"dispatch message","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","reply":"http://broker-ingress.knative-eventing.svc.cluster.local/cfg-events/test"}

the trigger 

spec:
  broker: test
  filter:
    attributes:
      source: cfg/test
      type: samples.http.mod3
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: sequence-display
      namespace: cfg-events

dispatcher:

{"level":"debug","ts":"2023-09-14T19:12:16.179Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_receiver.go:164","msg":"Received request","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","host":"test-kne-trigger-kn-jsm-channel.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T19:12:16.179Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_receiver.go:177","msg":"Request mapped to channel","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","channel":"cfg-events/test-kne-trigger"}
{"level":"debug","ts":"2023-09-14T19:12:16.183Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/consumer.go:85","msg":"received message from JetStream consumer","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","knative.dev/controller":"knative.dev.eventing-natss.pkg.channel.jetstream.dispatcher.Reconciler","knative.dev/kind":"messaging.knative.dev.NatsJetStreamChannel","knative.dev/traceid":"399853f3-ff1a-4b69-b4d4-a91aba3b710b","knative.dev/key":"cfg-events/test-kne-trigger","is_leader":true,"is_leader":true,"sub_uid":"303bea73-074a-471a-ab29-9d8003015f25","msg_id":"fd39cfc1-d6c4-4e27-8b8a-9c3d8c17f364"}
{"level":"debug","ts":"2023-09-14T19:12:16.183Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:198","msg":"Dispatching event","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","url":"http://broker-filter.knative-eventing.svc.cluster.local/triggers/cfg-events/sequence-trigger/c51ae00e-0c9f-4804-9fa4-a5132c7c7dad"}
{"level":"debug","ts":"2023-09-14T19:12:16.183Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/consumer.go:85","msg":"received message from JetStream consumer","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","knative.dev/controller":"knative.dev.eventing-natss.pkg.channel.jetstream.dispatcher.Reconciler","knative.dev/kind":"messaging.knative.dev.NatsJetStreamChannel","knative.dev/traceid":"2dc11619-518c-4629-a024-2c2bda71cff3","knative.dev/key":"cfg-events/test-kne-trigger","is_leader":true,"is_leader":true,"sub_uid":"ca208c06-0862-4d52-b7e4-b07528de353c","msg_id":"fd39cfc1-d6c4-4e27-8b8a-9c3d8c17f364"}
{"level":"debug","ts":"2023-09-14T19:12:16.183Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:198","msg":"Dispatching event","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","url":"http://broker-filter.knative-eventing.svc.cluster.local/triggers/cfg-events/display-trigger/cf7de572-3cc0-4eaf-99d1-af401bd1e714"}
{"level":"debug","ts":"2023-09-14T19:12:16.190Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_receiver.go:164","msg":"Received request","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","host":"test-kn-sequence-0-kn-jsm-channel.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T19:12:16.190Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_receiver.go:177","msg":"Request mapped to channel","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","channel":"cfg-events/test-kn-sequence-0"}
{"level":"debug","ts":"2023-09-14T19:12:16.191Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:253","msg":"Response is a non event, discarding it","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","status_code":200}
{"level":"debug","ts":"2023-09-14T19:12:16.191Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/consumer.go:123","msg":"message forward to downstream subscriber","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","knative.dev/controller":"knative.dev.eventing-natss.pkg.channel.jetstream.dispatcher.Reconciler","knative.dev/kind":"messaging.knative.dev.NatsJetStreamChannel","knative.dev/traceid":"2dc11619-518c-4629-a024-2c2bda71cff3","knative.dev/key":"cfg-events/test-kne-trigger","is_leader":true,"is_leader":true,"sub_uid":"ca208c06-0862-4d52-b7e4-b07528de353c","msg_id":"fd39cfc1-d6c4-4e27-8b8a-9c3d8c17f364"}
{"level":"debug","ts":"2023-09-14T19:12:16.191Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/consumer.go:85","msg":"received message from JetStream consumer","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","knative.dev/controller":"knative.dev.eventing-natss.pkg.channel.jetstream.dispatcher.Reconciler","knative.dev/kind":"messaging.knative.dev.NatsJetStreamChannel","knative.dev/traceid":"4575ccdf-a581-4e18-8ac9-4caea56eed6a","knative.dev/key":"cfg-events/test-kn-sequence-0","is_leader":true,"is_leader":true,"sub_uid":"fc5540b7-05e9-4edc-a178-cb3697ffdb45","msg_id":"fd39cfc1-d6c4-4e27-8b8a-9c3d8c17f364"}
{"level":"debug","ts":"2023-09-14T19:12:16.191Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:198","msg":"Dispatching event","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","url":"http://first.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T19:12:16.192Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:253","msg":"Response is a non event, discarding it","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","status_code":202}
{"level":"debug","ts":"2023-09-14T19:12:16.192Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/consumer.go:123","msg":"message forward to downstream subscriber","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","knative.dev/controller":"knative.dev.eventing-natss.pkg.channel.jetstream.dispatcher.Reconciler","knative.dev/kind":"messaging.knative.dev.NatsJetStreamChannel","knative.dev/traceid":"399853f3-ff1a-4b69-b4d4-a91aba3b710b","knative.dev/key":"cfg-events/test-kne-trigger","is_leader":true,"is_leader":true,"sub_uid":"303bea73-074a-471a-ab29-9d8003015f25","msg_id":"fd39cfc1-d6c4-4e27-8b8a-9c3d8c17f364"}
{"level":"debug","ts":"2023-09-14T19:12:16.201Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:198","msg":"Dispatching event","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","url":"http://test-kn-sequence-1-kn-jsm-channel.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T19:12:16.206Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_receiver.go:164","msg":"Received request","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","host":"test-kn-sequence-1-kn-jsm-channel.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T19:12:16.206Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_receiver.go:177","msg":"Request mapped to channel","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","channel":"cfg-events/test-kn-sequence-1"}
{"level":"debug","ts":"2023-09-14T19:12:16.207Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/consumer.go:85","msg":"received message from JetStream consumer","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","knative.dev/controller":"knative.dev.eventing-natss.pkg.channel.jetstream.dispatcher.Reconciler","knative.dev/kind":"messaging.knative.dev.NatsJetStreamChannel","knative.dev/traceid":"8cf5eb25-0a6b-4e94-b071-07903d6f5686","knative.dev/key":"cfg-events/test-kn-sequence-1","is_leader":true,"is_leader":true,"sub_uid":"794454fe-8849-4bf4-a011-a83ab0159aa0","msg_id":"fd39cfc1-d6c4-4e27-8b8a-9c3d8c17f364"}
{"level":"debug","ts":"2023-09-14T19:12:16.207Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:198","msg":"Dispatching event","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","url":"http://second.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T19:12:16.208Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:253","msg":"Response is a non event, discarding it","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","status_code":202}
{"level":"debug","ts":"2023-09-14T19:12:16.208Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/consumer.go:123","msg":"message forward to downstream subscriber","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","knative.dev/controller":"knative.dev.eventing-natss.pkg.channel.jetstream.dispatcher.Reconciler","knative.dev/kind":"messaging.knative.dev.NatsJetStreamChannel","knative.dev/traceid":"4575ccdf-a581-4e18-8ac9-4caea56eed6a","knative.dev/key":"cfg-events/test-kn-sequence-0","is_leader":true,"is_leader":true,"sub_uid":"fc5540b7-05e9-4edc-a178-cb3697ffdb45","msg_id":"fd39cfc1-d6c4-4e27-8b8a-9c3d8c17f364"}
{"level":"debug","ts":"2023-09-14T19:12:16.218Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:198","msg":"Dispatching event","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","url":"http://test-kn-sequence-2-kn-jsm-channel.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T19:12:16.223Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_receiver.go:164","msg":"Received request","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","host":"test-kn-sequence-2-kn-jsm-channel.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T19:12:16.223Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_receiver.go:177","msg":"Request mapped to channel","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","channel":"cfg-events/test-kn-sequence-2"}
{"level":"debug","ts":"2023-09-14T19:12:16.224Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/consumer.go:85","msg":"received message from JetStream consumer","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","knative.dev/controller":"knative.dev.eventing-natss.pkg.channel.jetstream.dispatcher.Reconciler","knative.dev/kind":"messaging.knative.dev.NatsJetStreamChannel","knative.dev/traceid":"af527795-5fd0-40db-a687-70b0fb2da19d","knative.dev/key":"cfg-events/test-kn-sequence-2","is_leader":true,"is_leader":true,"sub_uid":"348b1a86-34ee-4072-a3a2-ef3db3c3750b","msg_id":"fd39cfc1-d6c4-4e27-8b8a-9c3d8c17f364"}
{"level":"debug","ts":"2023-09-14T19:12:16.224Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:198","msg":"Dispatching event","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","url":"http://third.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T19:12:16.225Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:253","msg":"Response is a non event, discarding it","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","status_code":202}
{"level":"debug","ts":"2023-09-14T19:12:16.225Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/consumer.go:123","msg":"message forward to downstream subscriber","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","knative.dev/controller":"knative.dev.eventing-natss.pkg.channel.jetstream.dispatcher.Reconciler","knative.dev/kind":"messaging.knative.dev.NatsJetStreamChannel","knative.dev/traceid":"8cf5eb25-0a6b-4e94-b071-07903d6f5686","knative.dev/key":"cfg-events/test-kn-sequence-1","is_leader":true,"is_leader":true,"sub_uid":"794454fe-8849-4bf4-a011-a83ab0159aa0","msg_id":"fd39cfc1-d6c4-4e27-8b8a-9c3d8c17f364"}
{"level":"debug","ts":"2023-09-14T19:12:16.234Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:198","msg":"Dispatching event","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","url":"http://broker-ingress.knative-eventing.svc.cluster.local/cfg-events/test"}
{"level":"debug","ts":"2023-09-14T19:12:16.240Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_receiver.go:164","msg":"Received request","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","host":"test-kne-trigger-kn-jsm-channel.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T19:12:16.240Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_receiver.go:177","msg":"Request mapped to channel","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","channel":"cfg-events/test-kne-trigger"}
{"level":"debug","ts":"2023-09-14T19:12:16.242Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:253","msg":"Response is a non event, discarding it","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","status_code":202}
{"level":"debug","ts":"2023-09-14T19:12:16.242Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/consumer.go:123","msg":"message forward to downstream subscriber","knative.dev/pod":"jetstream-ch-dispatcher-6cd4585dcd-cvn8j","knative.dev/controller":"knative.dev.eventing-natss.pkg.channel.jetstream.dispatcher.Reconciler","knative.dev/kind":"messaging.knative.dev.NatsJetStreamChannel","knative.dev/traceid":"af527795-5fd0-40db-a687-70b0fb2da19d","knative.dev/key":"cfg-events/test-kn-sequence-2","is_leader":true,"is_leader":true,"sub_uid":"348b1a86-34ee-4072-a3a2-ef3db3c3750b","msg_id":"fd39cfc1-d6c4-4e27-8b8a-9c3d8c17f364"}

broker-filter:

{"level":"debug","ts":"2023-09-14T19:12:16.184Z","logger":"mt_broker_filter","caller":"filter/filter_handler.go:189","msg":"Received message","commit":"4d6e1fc","triggerRef":"cfg-events/sequence-trigger"}
{"level":"debug","ts":"2023-09-14T19:12:16.184Z","logger":"mt_broker_filter","caller":"filter/filter_handler.go:422","msg":"Applying attributes filter.","commit":"4d6e1fc","trigger":"cfg-events/sequence-trigger","filter":{"attributes":{"source":"cfg/test","type":"cfg/test"}}}
{"level":"debug","ts":"2023-09-14T19:12:16.190Z","logger":"mt_broker_filter","caller":"filter/filter_handler.go:189","msg":"Received message","commit":"4d6e1fc","triggerRef":"cfg-events/display-trigger"}
{"level":"debug","ts":"2023-09-14T19:12:16.190Z","logger":"mt_broker_filter","caller":"filter/filter_handler.go:422","msg":"Applying attributes filter.","commit":"4d6e1fc","trigger":"cfg-events/display-trigger","filter":{"attributes":{"source":"cfg/test","type":"samples.http.mod3"}}}
{"level":"debug","ts":"2023-09-14T19:12:16.190Z","logger":"mt_broker_filter","caller":"attributes/filter.go:52","msg":"Attribute had non-matching value{attribute 15 0 type <nil>} {filter 15 0 samples.http.mod3 <nil>} {received 15 0 cfg/test <nil>}","commit":"4d6e1fc","trigger":"cfg-events/display-trigger"}
{"level":"debug","ts":"2023-09-14T19:12:16.192Z","logger":"mt_broker_filter","caller":"filter/filter_handler.go:263","msg":"Successfully dispatched message","commit":"4d6e1fc","target":"http://test-kn-sequence-0-kn-jsm-channel.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T19:12:16.192Z","logger":"mt_broker_filter","caller":"filter/filter_handler.go:358","msg":"Response doesn't contain a CloudEvent, replying with an empty response","commit":"4d6e1fc","target":"http://test-kn-sequence-0-kn-jsm-channel.cfg-events.svc.cluster.local"}

This is the same scenario with 1.3.0, dispatcher with subject and work correctly

dispatcher :

{"level":"debug","ts":"2023-09-14T20:53:03.762Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_receiver.go:164","msg":"Received request","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","host":"test-kne-trigger-kn-channel.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T20:53:03.762Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_receiver.go:177","msg":"Request mapped to channel","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":"cfg-events/test-kne-trigger"}
{"level":"info","ts":"2023-09-14T20:53:03.762Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:134","msg":"Received event","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":"cfg-events/test-kne-trigger"}
{"level":"debug","ts":"2023-09-14T20:53:03.764Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:158","msg":"published","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":"cfg-events/test-kne-trigger"}
{"level":"debug","ts":"2023-09-14T20:53:03.766Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:299","msg":"NATS JetStream message received","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","subject":"K-ORDERS.test-kne-trigger-cfg-events"}
{"level":"debug","ts":"2023-09-14T20:53:03.766Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:304","msg":"dispatch message","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","destination":"http://broker-filter.knative-eventing.svc.cluster.local/triggers/cfg-events/display-trigger/2c108490-4459-4cef-9a55-b1589c952e8f"}
{"level":"debug","ts":"2023-09-14T20:53:03.766Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:310","msg":"dispatch message","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","reply":"http://broker-ingress.knative-eventing.svc.cluster.local/cfg-events/test"}
{"level":"debug","ts":"2023-09-14T20:53:03.766Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:198","msg":"Dispatching event","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","url":"http://broker-filter.knative-eventing.svc.cluster.local/triggers/cfg-events/display-trigger/2c108490-4459-4cef-9a55-b1589c952e8f"}
{"level":"debug","ts":"2023-09-14T20:53:03.766Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:299","msg":"NATS JetStream message received","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","subject":"K-ORDERS.test-kne-trigger-cfg-events"}
{"level":"debug","ts":"2023-09-14T20:53:03.766Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:304","msg":"dispatch message","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","destination":"http://broker-filter.knative-eventing.svc.cluster.local/triggers/cfg-events/sequence-trigger/baba2259-29c2-4b4f-bcd1-b821df1fc48a"}
{"level":"debug","ts":"2023-09-14T20:53:03.766Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:310","msg":"dispatch message","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","reply":"http://broker-ingress.knative-eventing.svc.cluster.local/cfg-events/test"}
{"level":"debug","ts":"2023-09-14T20:53:03.766Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:198","msg":"Dispatching event","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","url":"http://broker-filter.knative-eventing.svc.cluster.local/triggers/cfg-events/sequence-trigger/baba2259-29c2-4b4f-bcd1-b821df1fc48a"}
{"level":"debug","ts":"2023-09-14T20:53:03.775Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:253","msg":"Response is a non event, discarding it","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","status_code":200}
{"level":"debug","ts":"2023-09-14T20:53:03.775Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:326","msg":"Dispatch details","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","DispatchExecutionInfo":{"Time":9525610,"ResponseCode":200,"ResponseBody":null}}
{"level":"debug","ts":"2023-09-14T20:53:03.775Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:331","msg":"message dispatched","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":{"Namespace":"cfg-events","Name":"test-kne-trigger"}}
{"level":"debug","ts":"2023-09-14T20:53:03.780Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_receiver.go:164","msg":"Received request","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","host":"test-kn-sequence-0-kn-channel.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T20:53:03.780Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_receiver.go:177","msg":"Request mapped to channel","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":"cfg-events/test-kn-sequence-0"}
{"level":"info","ts":"2023-09-14T20:53:03.780Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:134","msg":"Received event","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":"cfg-events/test-kn-sequence-0"}
{"level":"debug","ts":"2023-09-14T20:53:03.783Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:158","msg":"published","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":"cfg-events/test-kn-sequence-0"}
{"level":"debug","ts":"2023-09-14T20:53:03.783Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:299","msg":"NATS JetStream message received","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","subject":"K-ORDERS.test-kn-sequence-0-cfg-events"}
{"level":"debug","ts":"2023-09-14T20:53:03.783Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:304","msg":"dispatch message","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","destination":"http://first.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T20:53:03.783Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:310","msg":"dispatch message","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","reply":"http://test-kn-sequence-1-kn-channel.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T20:53:03.783Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:198","msg":"Dispatching event","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","url":"http://first.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T20:53:03.796Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:253","msg":"Response is a non event, discarding it","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","status_code":202}
{"level":"debug","ts":"2023-09-14T20:53:03.796Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:326","msg":"Dispatch details","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","DispatchExecutionInfo":{"Time":29830641,"ResponseCode":202,"ResponseBody":null}}
{"level":"debug","ts":"2023-09-14T20:53:03.796Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:331","msg":"message dispatched","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":{"Namespace":"cfg-events","Name":"test-kne-trigger"}}
{"level":"debug","ts":"2023-09-14T20:53:05.767Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:198","msg":"Dispatching event","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","url":"http://test-kn-sequence-1-kn-channel.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T20:53:05.775Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_receiver.go:164","msg":"Received request","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","host":"test-kn-sequence-1-kn-channel.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T20:53:05.775Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_receiver.go:177","msg":"Request mapped to channel","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":"cfg-events/test-kn-sequence-1"}
{"level":"info","ts":"2023-09-14T20:53:05.775Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:134","msg":"Received event","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":"cfg-events/test-kn-sequence-1"}
{"level":"debug","ts":"2023-09-14T20:53:05.779Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:158","msg":"published","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":"cfg-events/test-kn-sequence-1"}
{"level":"debug","ts":"2023-09-14T20:53:05.780Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:299","msg":"NATS JetStream message received","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","subject":"K-ORDERS.test-kn-sequence-1-cfg-events"}
{"level":"debug","ts":"2023-09-14T20:53:05.780Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:304","msg":"dispatch message","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","destination":"http://second.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T20:53:05.780Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:310","msg":"dispatch message","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","reply":"http://test-kn-sequence-2-kn-channel.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T20:53:05.780Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:198","msg":"Dispatching event","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","url":"http://second.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T20:53:05.791Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:253","msg":"Response is a non event, discarding it","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","status_code":202}
{"level":"debug","ts":"2023-09-14T20:53:05.791Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:326","msg":"Dispatch details","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","DispatchExecutionInfo":{"Time":24142574,"ResponseCode":202,"ResponseBody":null}}
{"level":"debug","ts":"2023-09-14T20:53:05.791Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:331","msg":"message dispatched","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":{"Namespace":"cfg-events","Name":"test-kn-sequence-0"}}
{"level":"debug","ts":"2023-09-14T20:53:07.799Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:198","msg":"Dispatching event","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","url":"http://test-kn-sequence-2-kn-channel.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T20:53:07.807Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_receiver.go:164","msg":"Received request","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","host":"test-kn-sequence-2-kn-channel.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T20:53:07.808Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_receiver.go:177","msg":"Request mapped to channel","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":"cfg-events/test-kn-sequence-2"}
{"level":"info","ts":"2023-09-14T20:53:07.808Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:134","msg":"Received event","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":"cfg-events/test-kn-sequence-2"}
{"level":"debug","ts":"2023-09-14T20:53:07.810Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:299","msg":"NATS JetStream message received","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","subject":"K-ORDERS.test-kn-sequence-2-cfg-events"}
{"level":"debug","ts":"2023-09-14T20:53:07.810Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:158","msg":"published","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":"cfg-events/test-kn-sequence-2"}
{"level":"debug","ts":"2023-09-14T20:53:07.810Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:304","msg":"dispatch message","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","destination":"http://third.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T20:53:07.810Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:310","msg":"dispatch message","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","reply":"http://broker-ingress.knative-eventing.svc.cluster.local/cfg-events/test"}
{"level":"debug","ts":"2023-09-14T20:53:07.810Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:198","msg":"Dispatching event","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","url":"http://third.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T20:53:07.821Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:253","msg":"Response is a non event, discarding it","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","status_code":202}
{"level":"debug","ts":"2023-09-14T20:53:07.821Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:326","msg":"Dispatch details","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","DispatchExecutionInfo":{"Time":22405108,"ResponseCode":202,"ResponseBody":null}}
{"level":"debug","ts":"2023-09-14T20:53:07.821Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:331","msg":"message dispatched","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":{"Namespace":"cfg-events","Name":"test-kn-sequence-1"}}
{"level":"debug","ts":"2023-09-14T20:53:10.157Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:198","msg":"Dispatching event","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","url":"http://broker-ingress.knative-eventing.svc.cluster.local/cfg-events/test"}
{"level":"debug","ts":"2023-09-14T20:53:10.169Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_receiver.go:164","msg":"Received request","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","host":"test-kne-trigger-kn-channel.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T20:53:10.169Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_receiver.go:177","msg":"Request mapped to channel","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":"cfg-events/test-kne-trigger"}
{"level":"info","ts":"2023-09-14T20:53:10.169Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:134","msg":"Received event","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":"cfg-events/test-kne-trigger"}
{"level":"debug","ts":"2023-09-14T20:53:10.171Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:158","msg":"published","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":"cfg-events/test-kne-trigger"}
{"level":"debug","ts":"2023-09-14T20:53:10.172Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:299","msg":"NATS JetStream message received","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","subject":"K-ORDERS.test-kne-trigger-cfg-events"}
{"level":"debug","ts":"2023-09-14T20:53:10.172Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:304","msg":"dispatch message","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","destination":"http://broker-filter.knative-eventing.svc.cluster.local/triggers/cfg-events/sequence-trigger/baba2259-29c2-4b4f-bcd1-b821df1fc48a"}
{"level":"debug","ts":"2023-09-14T20:53:10.172Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:299","msg":"NATS JetStream message received","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","subject":"K-ORDERS.test-kne-trigger-cfg-events"}
{"level":"debug","ts":"2023-09-14T20:53:10.172Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:304","msg":"dispatch message","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","destination":"http://broker-filter.knative-eventing.svc.cluster.local/triggers/cfg-events/display-trigger/2c108490-4459-4cef-9a55-b1589c952e8f"}
{"level":"debug","ts":"2023-09-14T20:53:10.172Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:310","msg":"dispatch message","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","reply":"http://broker-ingress.knative-eventing.svc.cluster.local/cfg-events/test"}
{"level":"debug","ts":"2023-09-14T20:53:10.172Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:198","msg":"Dispatching event","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","url":"http://broker-filter.knative-eventing.svc.cluster.local/triggers/cfg-events/display-trigger/2c108490-4459-4cef-9a55-b1589c952e8f"}
{"level":"debug","ts":"2023-09-14T20:53:10.172Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:310","msg":"dispatch message","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","reply":"http://broker-ingress.knative-eventing.svc.cluster.local/cfg-events/test"}
{"level":"debug","ts":"2023-09-14T20:53:10.172Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:198","msg":"Dispatching event","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","url":"http://broker-filter.knative-eventing.svc.cluster.local/triggers/cfg-events/sequence-trigger/baba2259-29c2-4b4f-bcd1-b821df1fc48a"}
{"level":"debug","ts":"2023-09-14T20:53:10.174Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:253","msg":"Response is a non event, discarding it","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","status_code":202}
{"level":"debug","ts":"2023-09-14T20:53:10.174Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:326","msg":"Dispatch details","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","DispatchExecutionInfo":{"Time":16699320,"ResponseCode":202,"ResponseBody":null}}
{"level":"debug","ts":"2023-09-14T20:53:10.174Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:331","msg":"message dispatched","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":{"Namespace":"cfg-events","Name":"test-kn-sequence-2"}}
{"level":"debug","ts":"2023-09-14T20:53:10.174Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:253","msg":"Response is a non event, discarding it","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","status_code":200}
{"level":"debug","ts":"2023-09-14T20:53:10.174Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:326","msg":"Dispatch details","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","DispatchExecutionInfo":{"Time":2238830,"ResponseCode":200,"ResponseBody":null}}
{"level":"debug","ts":"2023-09-14T20:53:10.174Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:331","msg":"message dispatched","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":{"Namespace":"cfg-events","Name":"test-kne-trigger"}}
{"level":"debug","ts":"2023-09-14T20:53:10.188Z","logger":"jetstream-channel-dispatcher","caller":"channel/message_dispatcher.go:253","msg":"Response is a non event, discarding it","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","status_code":200}
{"level":"debug","ts":"2023-09-14T20:53:10.188Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:326","msg":"Dispatch details","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","DispatchExecutionInfo":{"Time":16027400,"ResponseCode":200,"ResponseBody":null}}
{"level":"debug","ts":"2023-09-14T20:53:10.188Z","logger":"jetstream-channel-dispatcher","caller":"dispatcher/jetstream_dispatcher.go:331","msg":"message dispatched","knative.dev/pod":"jetstream-ch-dispatcher-6796f67954-9r2kt","channel":{"Namespace":"cfg-events","Name":"test-kne-trigger"}}

broker filter:

{"level":"debug","ts":"2023-09-14T20:53:03.774Z","logger":"mt_broker_filter","caller":"filter/filter_handler.go:189","msg":"Received message","commit":"4d6e1fc","triggerRef":"cfg-events/sequence-trigger"}
{"level":"debug","ts":"2023-09-14T20:53:03.774Z","logger":"mt_broker_filter","caller":"filter/filter_handler.go:422","msg":"Applying attributes filter.","commit":"4d6e1fc","trigger":"cfg-events/sequence-trigger","filter":{"attributes":{"source":"cfg/test","type":"cfg/test"}}}
{"level":"debug","ts":"2023-09-14T20:53:03.774Z","logger":"mt_broker_filter","caller":"filter/filter_handler.go:189","msg":"Received message","commit":"4d6e1fc","triggerRef":"cfg-events/display-trigger"}
{"level":"debug","ts":"2023-09-14T20:53:03.774Z","logger":"mt_broker_filter","caller":"filter/filter_handler.go:422","msg":"Applying attributes filter.","commit":"4d6e1fc","trigger":"cfg-events/display-trigger","filter":{"attributes":{"type":"samples.http.mod3"}}}
{"level":"debug","ts":"2023-09-14T20:53:03.774Z","logger":"mt_broker_filter","caller":"attributes/filter.go:52","msg":"Attribute had non-matching value{attribute 15 0 type <nil>} {filter 15 0 samples.http.mod3 <nil>} {received 15 0 cfg/test <nil>}","commit":"4d6e1fc","trigger":"cfg-events/display-trigger"}
{"level":"debug","ts":"2023-09-14T20:53:03.795Z","logger":"mt_broker_filter","caller":"filter/filter_handler.go:263","msg":"Successfully dispatched message","commit":"4d6e1fc","target":"http://test-kn-sequence-0-kn-channel.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T20:53:03.795Z","logger":"mt_broker_filter","caller":"filter/filter_handler.go:358","msg":"Response doesn't contain a CloudEvent, replying with an empty response","commit":"4d6e1fc","target":"http://test-kn-sequence-0-kn-channel.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T20:53:10.173Z","logger":"mt_broker_filter","caller":"filter/filter_handler.go:189","msg":"Received message","commit":"4d6e1fc","triggerRef":"cfg-events/sequence-trigger"}
{"level":"debug","ts":"2023-09-14T20:53:10.173Z","logger":"mt_broker_filter","caller":"filter/filter_handler.go:422","msg":"Applying attributes filter.","commit":"4d6e1fc","trigger":"cfg-events/sequence-trigger","filter":{"attributes":{"source":"cfg/test","type":"cfg/test"}}}
{"level":"debug","ts":"2023-09-14T20:53:10.173Z","logger":"mt_broker_filter","caller":"attributes/filter.go:52","msg":"Attribute had non-matching value{attribute 15 0 type <nil>} {filter 15 0 cfg/test <nil>} {received 15 0 samples.http.mod3 <nil>}","commit":"4d6e1fc","trigger":"cfg-events/sequence-trigger"}
{"level":"debug","ts":"2023-09-14T20:53:10.173Z","logger":"mt_broker_filter","caller":"filter/filter_handler.go:189","msg":"Received message","commit":"4d6e1fc","triggerRef":"cfg-events/display-trigger"}
{"level":"debug","ts":"2023-09-14T20:53:10.173Z","logger":"mt_broker_filter","caller":"filter/filter_handler.go:422","msg":"Applying attributes filter.","commit":"4d6e1fc","trigger":"cfg-events/display-trigger","filter":{"attributes":{"type":"samples.http.mod3"}}}
{"level":"debug","ts":"2023-09-14T20:53:10.187Z","logger":"mt_broker_filter","caller":"filter/filter_handler.go:263","msg":"Successfully dispatched message","commit":"4d6e1fc","target":"http://sequence-display.cfg-events.svc.cluster.local"}
{"level":"debug","ts":"2023-09-14T20:53:10.187Z","logger":"mt_broker_filter","caller":"filter/filter_handler.go:358","msg":"Response doesn't contain a CloudEvent, replying with an empty response","commit":"4d6e1fc","target":"http://sequence-display.cfg-events.svc.cluster.local"}
chris93111 commented 11 months ago

tested with 1.10.2 , scenario is the same

chris93111 commented 11 months ago

after search , this issue is not problem with dispatcher , https://github.com/knative/eventing/blob/release-1.10/pkg/channel/message_dispatcher.go reject message without encoding , in my exemple , i use triggermesh webhooksource with sink broker , encoding is lost .