search

grafana / synthetic-monitoring-agent

Synthetic Monitoring Agent
https://grafana.com/docs/grafana-cloud/how-do-i/synthetic-monitoring/
Apache License 2.0
156 stars 22 forks source link

Traceroute Check Entered FAIL State #783

Open WhyAydan opened 1 month ago

WhyAydan commented 1 month ago

Hey,

Has anyone had this error before and been able to resolve it?

{"level":"info","program":"synthetic-monitoring-agent","subsystem":"updater","region_id":0,"tenantId":XXXX,"check_id":14491,"probe":"Home","target":"google.co.uk","job":"Google","check":"traceroute","time":1720440373704,"caller":"github.com/grafana/synthetic-monitoring-agent/internal/scraper/scraper.go:263","message":"check entered FAIL state"}

roobre commented 1 month ago

Hi @WhyAydan, this typically signals that the traceroute check failed to complete successfully. Could be due to a network hiccup, or some other transient error. Does this happen often? Does the check recover after a couple of executions after this failure?

WhyAydan commented 1 month ago

Hey there @roobre It never seems to get past the 2nd hop. But running it via command line it does get to the end result.

Screenshot 2024-07-08 at 14 22 26 Screenshot 2024-07-08 at 14 23 18 Screenshot 2024-07-08 at 15 08 33
roobre commented 1 month ago

That seems odd, there shouldn't be much difference between the traceroute CLI and our test, provided they are run from the same host. Where is this SM check running? Is it one of Grafana public probes, or a private one? If it is private, can you run the traceroute command from the same machine where the agent is running? (Including inside the same container, if it is running in one).

WhyAydan commented 1 month ago

Hey Roobre,

The Agent is a private one, running on the same Ubuntu VM that the first screenshot is on. It's running barebones on the VM no docker.

The traceroute on the host works just not via the agent :)

Same thing happens on the docker image.