search

grafana / synthetic-monitoring-agent

Synthetic Monitoring Agent
https://grafana.com/docs/grafana-cloud/how-do-i/synthetic-monitoring/
Apache License 2.0
167 stars 24 forks source link

Traceroute Check Entered FAIL State #783

Open WhyAydan opened 4 months ago

WhyAydan commented 4 months ago

Hey,

Has anyone had this error before and been able to resolve it?

{"level":"info","program":"synthetic-monitoring-agent","subsystem":"updater","region_id":0,"tenantId":XXXX,"check_id":14491,"probe":"Home","target":"google.co.uk","job":"Google","check":"traceroute","time":1720440373704,"caller":"github.com/grafana/synthetic-monitoring-agent/internal/scraper/scraper.go:263","message":"check entered FAIL state"}

roobre commented 4 months ago

Hi @WhyAydan, this typically signals that the traceroute check failed to complete successfully. Could be due to a network hiccup, or some other transient error. Does this happen often? Does the check recover after a couple of executions after this failure?

WhyAydan commented 4 months ago

Hey there @roobre It never seems to get past the 2nd hop. But running it via command line it does get to the end result.

Screenshot 2024-07-08 at 14 22 26 Screenshot 2024-07-08 at 14 23 18 Screenshot 2024-07-08 at 15 08 33
roobre commented 4 months ago

That seems odd, there shouldn't be much difference between the traceroute CLI and our test, provided they are run from the same host. Where is this SM check running? Is it one of Grafana public probes, or a private one? If it is private, can you run the traceroute command from the same machine where the agent is running? (Including inside the same container, if it is running in one).

WhyAydan commented 4 months ago

Hey Roobre,

The Agent is a private one, running on the same Ubuntu VM that the first screenshot is on. It's running barebones on the VM no docker.

The traceroute on the host works just not via the agent :)

Same thing happens on the docker image.