[question]: failed to new Nats Rpc in Argo Sensor

wesbragagt commented 1 month ago

Prior Search

[X] I have already searched this project's issues to determine if a similar question has already been asked.

What is your question?

I'm noticing that the cicd sensors for Argo I was using prior to version edge.24-08-13 are displaying the following error. Has anyone seen something similar? The webhook events are being delivered but not triggering any build workflows.

{"level":"fatal","ts":1724380260.9587278,"logger":"argo-events.sensor","caller":"leaderelection/leaderelection.go:139","msg":"failed to new Nats Rpc","sensorName":"ci-sensor-reconciliation-engine","error":"read tcp 10.0.245.78:49430->10.0.114.207:4222: i/o timeout","stacktrace":"github.com/argoproj/argo-events/common/leaderelection.(*natsEventBusElector).RunOrDie\n\t/home/runner/work/argo-events/argo-events/common/leaderelection/leaderelection.go:139\ngithub.com/argoproj/argo-events/sensors.(*SensorContext).Start\n\t/home/runner/work/argo-events/argo-events/sensors/listener.go:74\ngithub.com/argoproj/argo-events/sensors/cmd.Start\n\t/home/runner/work/argo-events/argo-events/sensors/cmd/start.go:86\ngithub.com/argoproj/argo-events/cmd/commands.NewSensorCommand.func1\n\t/home/runner/work/argo-events/argo-events/cmd/commands/sensor.go:14\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039\ngithub.com/argoproj/argo-events/cmd/commands.Execute\n\t/home/runner/work/argo-events/argo-events/cmd/commands/root.go:19\nmain.main\n\t/home/runner/work/argo-events/argo-events/cmd/main.go:8\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.14/x64/src/runtime/proc.go:250"}

What primary components of the stack does this relate to?

terraform

Code of Conduct

[X] I agree to follow this project's Code of Conduct

fullykubed commented 1 month ago

Can you clarify whether you are experiencing (a) some dropped events or (b) the sensor is continuously down / is dropping all events?

fullykubed commented 1 month ago

Additionally, can you post the entire series of logs that the sensor generates +- 2 minutes from when you notice this issue occurring?

wesbragagt commented 1 month ago

@fullykubed from what I've uncovered today is that when two sensors are pointing to the same default eventName it fails. I brought it down to just one sensor and I no longer see the error.

Panfactum / stack