Troubleshooting lack of messages between the event source and bus

[rossnelson]

I've configured the eventsource deployment and service per the example. Here are my specific configs:

Deployment

``` --- apiVersion: apps/v1 kind: Deployment metadata: annotations: reloader.stakater.com/auto: "true" labels: app: natsjs-eventsource name: natsjs-eventsource namespace: app spec: replicas: 1 revisionHistoryLimit: 3 selector: matchLabels: app: natsjs-eventsource strategy: {} template: metadata: labels: app: natsjs-eventsource annotations: ad.datadoghq.com/synapi.logs: '[{"source":"golang","service":"natsjs-eventsource"}]' spec: restartPolicy: Always containers: - name: natsjs-eventsource image: justinisrael/argo-natsjs-eventsource imagePullPolicy: Always ports: - name: http containerPort: 8080 resources: requests: cpu: 200m memory: 256Mi limits: cpu: 500m memory: 512Mi nodeSelector: node_name: app ```

Service

``` --- apiVersion: v1 kind: Service metadata: namespace: app name: natsjs-eventsource labels: run: natsjs-eventsource spec: ports: - port: 80 protocol: TCP targetPort: 8080 selector: app: natsjs-eventsource ```

EventSource

``` apiVersion: argoproj.io/v1alpha1 kind: EventSource metadata: name: nats-event-source namespace: app spec: eventBusName: jetstream-eventbus generic: notifications-email: url: "http://natsjs-eventsource.app.svc.cluster.local:80" insecure: true jsonBody: true config: |- url: "nats://jetstream-nats.jetstream.svc.cluster.local:4222" jsonBody: true stream: "default" subjects: - "default.notifications.email.>" tls: insecureSkipVerify: true ```

I can view logs for the deployment and source. Things seem configured properly.

EventSource Logs

``` ❯ kubectl logs -f -n app nats-event-source-eventsource-vxj55-bf55c6468-phtkp {"level":"info","ts":1724966714.1706817,"logger":"argo-events.eventsource","caller":"metrics/metrics.go:190","msg":"starting metrics server","eventSourceName":"nats-event-source"} {"level":"info","ts":1724966714.16987,"logger":"argo-events.eventsource","caller":"cmd/start.go:63","msg":"starting eventsource server","eventSourceName":"nats-event-source","version":"v1.9.2"} {"level":"info","ts":1724966714.2014341,"logger":"argo-events.eventsource","caller":"leaderelection/leaderelection.go:156","msg":"Not the LEADER, stand by ...","eventSourceName":"nats-event-source"} {"level":"info","ts":1724966715.5276387,"logger":"argo-events.eventsource","caller":"leaderelection/leaderelection.go:165","msg":"Becoming a Candidate, stand by ...","eventSourceName":"nats-event-source"} {"level":"info","ts":1724966715.5278525,"logger":"argo-events.eventsource","caller":"leaderelection/leaderelection.go:162","msg":"I'm the LEADER, starting ...","eventSourceName":"nats-event-source"} {"level":"info","ts":1724966715.5279014,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:454","msg":"Starting event source server...","eventSourceName":"nats-event-source"} {"level":"info","ts":1724966715.5303342,"logger":"argo-events.eventsource","caller":"eventbus/driver.go:35","msg":"eventBusConfig: {NATS:nil JetStream:&JetStreamConfig{URL:nats://jetstream-nats.jetstream.svc.cluster.local:4222,AccessSecret:nil,StreamConfig:,} Kafka:nil}","eventSourceName":"nats-event-source"} {"level":"info","ts":1724966715.5307024,"logger":"argo-events.eventsource","caller":"base/jetstream.go:83","msg":"NATS auth strategy: None","eventSourceName":"nats-event-source"} {"level":"info","ts":1724966715.5467536,"logger":"argo-events.eventsource","caller":"base/jetstream.go:102","msg":"Connected to NATS Jetstream server.","eventSourceName":"nats-event-source"} {"level":"info","ts":1724966715.5485764,"logger":"argo-events.eventsource","caller":"base/jetstream.go:115","msg":"No need to create Stream 'default' as it already exists","eventSourceName":"nats-event-source"} {"level":"info","ts":1724966715.5486119,"logger":"argo-events.eventsource","caller":"base/jetstream.go:83","msg":"NATS auth strategy: None","eventSourceName":"nats-event-source"} {"level":"info","ts":1724966715.5608046,"logger":"argo-events.eventsource","caller":"base/jetstream.go:102","msg":"Connected to NATS Jetstream server.","eventSourceName":"nats-event-source"} {"level":"info","ts":1724966715.5609808,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:603","msg":"Eventing server started.","eventSourceName":"nats-event-source"} {"level":"info","ts":1724966715.5610075,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:481","msg":"starting eventbus connection daemon...","eventSourceName":"nats-event-source"} {"level":"info","ts":1724966715.5617573,"logger":"argo-events.eventsource","caller":"generic/start.go:56","msg":"started processing the generic event source...","eventSourceName":"nats-event-source","eventSourceType":"generic","eventName":"notifications-email","url":"http://natsjs-eventsource.app.svc.cluster.local:80"} {"level":"info","ts":1724966715.5643768,"logger":"argo-events.eventsource","caller":"generic/start.go:59","msg":"connecting to eventsource server in 5 seconds...","eventSourceName":"nats-event-source","eventSourceType":"generic","eventName":"notifications-email","url":"http://natsjs-eventsource.app.svc.cluster.local:80"} {"level":"info","ts":1724966720.5647228,"logger":"argo-events.eventsource","caller":"generic/start.go:72","msg":"dialing eventsource server...","eventSourceName":"nats-event-source","eventSourceType":"generic","eventName":"notifications-email","url":"http://natsjs-eventsource.app.svc.cluster.local:80"} ```

Jestream Consumer Deployment Logs

``` ❯ kubectl logs -f -n app natsjs-eventsource-76b56fddfd-cq82t {"time":"2024-08-29T16:11:07.827610082Z","level":"INFO","msg":"listening","host":"0.0.0.0:8080"} ```

But I never see a message passed between the two. I can see messages in the default stream waiting to be picked up.

nats stream ls

``` ❯ nats --tlsca="$(mkcert -CAROOT)/dev-ca.pem" stream ls ╭────────────────────────────────────────────────────────────────────────────────╮ │ Streams │ ├─────────┬─────────────┬─────────────────────┬──────────┬────────┬──────────────┤ │ Name │ Description │ Created │ Messages │ Size │ Last Message │ ├─────────┼─────────────┼─────────────────────┼──────────┼────────┼──────────────┤ │ default │ │ 2024-08-07 22:18:31 │ 70 │ 19 KiB │ 31m29s │ ╰─────────┴─────────────┴─────────────────────┴──────────┴────────┴──────────────╯ ```

I'm sure I'm missing something but haven't found the issue yet. A second set of eyes on this would be much appreciated. 🥲

[justinfx]

Does it make any difference if you remove the "http://" scheme from the generic EventSource url?

notifications-email:
      url: "natsjs-eventsource.app.svc.cluster.local:80"

It doesn't appear to be documented as wanting a scheme, since its a grpc endpoint

[justinfx]

Additionally, for troubleshooting purposes, the fact that you don't see any consumers draining messages from your "default" stream implies that when you deploy the generic EventSource, it is not successfully making a connection to your argo-natsjs-eventsource service. You showed the logs from that, but did you look at the logs of your "nats-event-source"?

The workflow that happens under the hood is that you have this long running external "argo-natsjs-eventsource" deployment. Then you deploy 1 or more generic EventSources that point at the url of "argo-natsjs-eventsource", and provide the details of how it should tell that service to be a consumer and forward messages to you. If your EventSource is deployed successfully, it should be able to connect and you should start seeing "argo-natsjs-eventsource" log messages that it is servicing a new client.

[gedw99]

I am adding myself as I am using dagger , nats and Argo.

[justinfx]

@gedw99 you are seeing the same issue?

[gedw99]

@gedw99 you are seeing the same issue?

Will try it tomorrow. Am at beach today :)