Sensor fails to subscribe to subject

[ptemmer]

After a sensor pod (that was working OK for several hours) apparently experienced a connectivity problem with the NATS streaming server, the sensor reconnects, closes a subscription to the subject, but then fails to subscribe to the same subject again. The error message repeats itself over and over, and can only be mitigated by killing the sensor pod. The eventsource pod did not show any errors.

Please see the log below:

2020-09-11T10:30:26.847495245Z {"level":"info","ts":1599820226.847264,"logger":"argo-events.sensor","caller":"sensors/listener.go:266","msg":"successfully processed the trigger","sensorName":"tapp-push-se
nsor-pr-4","triggerName":"buildFromBranch"}
2020-09-11T10:57:21.2862582Z {"level":"error","ts":1599821841.2859821,"logger":"argo-events.sensor","caller":"driver/nats.go:111","msg":"NATS streaming connection lost","sensorName":"tapp-push-sensor-pr-4
","clientID":"client-2991044578","stacktrace":"github.com/argoproj/argo-events/eventbus/driver.(*natsStreaming).Connect.func3\n\t/home/runner/work/argo-events/argo-events/eventbus/driver/nats.go:111"}
2020-09-11T10:57:25.455194472Z {"level":"info","ts":1599821845.4544566,"logger":"argo-events.sensor","caller":"sensors/listener.go:189","msg":"NATS connection lost, reconnecting...","sensorName":"tapp-pus
h-sensor-pr-4"}
2020-09-11T10:57:25.455240473Z {"level":"info","ts":1599821845.4546099,"logger":"argo-events.sensor","caller":"driver/nats.go:92","msg":"NATS auth strategy: Token","sensorName":"tapp-push-sensor-pr-4","cl
ientID":"client-2991044578"}
2020-09-11T10:57:25.464658335Z {"level":"info","ts":1599821845.4643745,"logger":"argo-events.sensor","caller":"driver/nats.go:104","msg":"Connected to NATS server.","sensorName":"tapp-push-sensor-pr-4","c
lientID":"client-2991044578"}
2020-09-11T10:57:25.986326371Z {"level":"info","ts":1599821845.986085,"logger":"argo-events.sensor","caller":"driver/nats.go:117","msg":"Connected to NATS streaming server.","sensorName":"tapp-push-sensor
-pr-4","clientID":"client-2991044578"}
2020-09-11T10:57:25.986391973Z {"level":"info","ts":1599821845.986148,"logger":"argo-events.sensor","caller":"sensors/listener.go:195","msg":"reconnected to NATS streaming server.","sensorName":"tapp-push
-sensor-pr-4","clientID":"client-2991044578"}
2020-09-11T10:57:25.986403973Z {"level":"info","ts":1599821845.9861758,"logger":"argo-events.sensor","caller":"driver/nats.go:165","msg":"closing subscription...","sensorName":"tapp-push-sensor-pr-4","cli
entID":"client-2991044578"}
2020-09-11T10:57:25.986588178Z {"level":"info","ts":1599821845.9861977,"logger":"argo-events.sensor","caller":"driver/nats.go:167","msg":"subscription on subject eventbus-argo closed","sensorName":"tapp-p
ush-sensor-pr-4","clientID":"client-2991044578"}
2020-09-11T10:57:27.986664006Z {"level":"info","ts":1599821847.9864333,"logger":"argo-events.sensor","caller":"sensors/listener.go:166","msg":"started subscribing to events for triggers [buildFromBranch] 
with client client-2991044578","sensorName":"tapp-push-sensor-pr-4"}
2020-09-11T10:57:28.039589181Z {"level":"error","ts":1599821848.0392988,"logger":"argo-events.sensor","caller":"driver/nats.go:153","msg":"failed to subscribe to subject eventbus-argo","sensorName":"tapp-
push-sensor-pr-4","clientID":"client-2991044578","stacktrace":"github.com/argoproj/argo-events/eventbus/driver.(*natsStreaming).SubscribeEventSources\n\t/home/runner/work/argo-events/argo-events/eventbus/
driver/nats.go:153\ngithub.com/argoproj/argo-events/sensors.(*SensorContext).ListenEvents.func1.4.1\n\t/home/runner/work/argo-events/argo-events/sensors/listener.go:168"}
2020-09-11T10:57:28.039838388Z {"level":"error","ts":1599821848.0393965,"logger":"argo-events.sensor","caller":"sensors/listener.go:170","msg":"failed to subscribe to event bus","sensorName":"tapp-push-se
nsor-pr-4","clientID":"client-2991044578","error":"leadership lost while committing log","stacktrace":"github.com/argoproj/argo-events/sensors.(*SensorContext).ListenEvents.func1.4.1\n\t/home/runner/work/
argo-events/argo-events/sensors/listener.go:170"}

Expected behavior A clear and concise description of what you expected to happen.

Environment (please complete the following information):

• Kubernetes: v1.14.8
• Argo Events: v1.0.0

Additional context

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[whynowy]

Thanks for reporting the issue.

Could you let me know what was the event, and the the size of the message approximately? And are you able to reproduce it?

[ptemmer]

The event was a git (BitBucket) webhook event, triggered by a commit to the repository. Message size was aprox 1kb. I have not been able to reproduce.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the last 60 days. It will be closed if no further activity occurs. Thank you for your contributions.