search

grafana / synthetic-monitoring-agent

Synthetic Monitoring Agent
https://grafana.com/docs/grafana-cloud/how-do-i/synthetic-monitoring/
Apache License 2.0
156 stars 22 forks source link

agent doesn't start #652

Closed jfrankiewicz closed 5 months ago

jfrankiewicz commented 5 months ago

I tried on docker and newest binary, I get the same result:

{"level":"info","program":"synthetic-monitoring-agent","time":1709992571187,"caller":"github.com/grafana/synthetic-monitoring-agent/cmd/synthetic-monitoring-agent/main.go:151","message":"disabling k6 checks"}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571189,"message":"2024/03/09 13:56:11 INFO: [core] [Channel #1] Channel created"}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571190,"message":"2024/03/09 13:56:11 INFO: [core] [Channel #1] original dial target is: \"synthetic-monitoring-api-eu-west-3.grafana.net:443\""}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571190,"message":"2024/03/09 13:56:11 INFO: [core] [Channel #1] parsed dial target is: resolver.Target{URL:url.URL{Scheme:\"synthetic-monitoring-api-eu-west-3.grafana.net\", Opaque:\"443\", User:(*url.Userinfo)(nil), Host:\"\", Path:\"\", RawPath:\"\", OmitHost:false, ForceQuery:false, RawQuery:\"\", Fragment:\"\", RawFragment:\"\"}}"}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571190,"message":"2024/03/09 13:56:11 INFO: [core] [Channel #1] fallback to scheme \"passthrough\""}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571190,"message":"2024/03/09 13:56:11 INFO: [core] [Channel #1] parsed dial target is: passthrough:///synthetic-monitoring-api-eu-west-3.grafana.net:443"}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571191,"message":"2024/03/09 13:56:11 INFO: [core] [Channel #1] Channel authority set to \"synthetic-monitoring-api-eu-west-3.grafana.net\""}
{"program":"synthetic-monitoring-agent","subsystem":"http","time":1709992571192,"caller":"log/log.go:245","message":"Starting HTTP server on localhost:4050..."}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571192,"message":"2024/03/09 13:56:11 INFO: [core] [Channel #1] Resolver state updated: {\n  \"Addresses\": [\n    {\n      \"Addr\": \"synthetic-monitoring-api-eu-west-3.grafana.net:443\",\n      \"ServerName\": \"\",\n      \"Attributes\": null,\n      \"BalancerAttributes\": null,\n      \"Metadata\": null\n    }\n  ],\n  \"Endpoints\": [\n    {\n      \"Addresses\": [\n        {\n          \"Addr\": \"synthetic-monitoring-api-eu-west-3.grafana.net:443\",\n          \"ServerName\": \"\",\n          \"Attributes\": null,\n          \"BalancerAttributes\": null,\n          \"Metadata\": null\n        }\n      ],\n      \"Attributes\": null\n    }\n  ],\n  \"ServiceConfig\": null,\n  \"Attributes\": null\n} (resolver returned new addresses)"}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571192,"message":"2024/03/09 13:56:11 INFO: [core] [Channel #1] Channel switches to new LB policy \"pick_first\""}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571193,"message":"2024/03/09 13:56:11 INFO: [core] [pick-first-lb 0xc00020e180] Received new config {\n  \"shuffleAddressList\": false\n}, resolver state {\n  \"Addresses\": [\n    {\n      \"Addr\": \"synthetic-monitoring-api-eu-west-3.grafana.net:443\",\n      \"ServerName\": \"\",\n      \"Attributes\": null,\n      \"BalancerAttributes\": null,\n      \"Metadata\": null\n    }\n  ],\n  \"Endpoints\": [\n    {\n      \"Addresses\": [\n        {\n          \"Addr\": \"synthetic-monitoring-api-eu-west-3.grafana.net:443\",\n          \"ServerName\": \"\",\n          \"Attributes\": null,\n          \"BalancerAttributes\": null,\n          \"Metadata\": null\n        }\n      ],\n      \"Attributes\": null\n    }\n  ],\n  \"ServiceConfig\": null,\n  \"Attributes\": null\n}"}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571193,"message":"2024/03/09 13:56:11 INFO: [core] [Channel #1 SubChannel #2] Subchannel created"}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571193,"message":"2024/03/09 13:56:11 INFO: [core] [Channel #1] Channel Connectivity change to CONNECTING"}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571193,"message":"2024/03/09 13:56:11 INFO: [core] [Channel #1] Channel exiting idle mode"}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571194,"message":"2024/03/09 13:56:11 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING"}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571194,"message":"2024/03/09 13:56:11 INFO: [core] [Channel #1 SubChannel #2] Subchannel picks a new address \"synthetic-monitoring-api-eu-west-3.grafana.net:443\" to connect"}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571194,"message":"2024/03/09 13:56:11 INFO: [core] [pick-first-lb 0xc00020e180] Received SubConn state update: 0xc00020e300, {ConnectivityState:CONNECTING ConnectionError:<nil>}"}
2024/03/09 13:56:11 RequireTransportSecurity
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571281,"message":"2024/03/09 13:56:11 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to READY"}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571281,"message":"2024/03/09 13:56:11 INFO: [core] [pick-first-lb 0xc00020e180] Received SubConn state update: 0xc00020e300, {ConnectivityState:READY ConnectionError:<nil>}"}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571281,"message":"2024/03/09 13:56:11 INFO: [core] [Channel #1] Channel Connectivity change to READY"}
{"level":"info","program":"synthetic-monitoring-agent","subsystem":"adhoc","time":1709992571282,"caller":"github.com/grafana/synthetic-monitoring-agent/internal/adhoc/adhoc.go:243","message":"fetching ad-hoc checks from synthetic-monitoring-api"}
{"level":"info","program":"synthetic-monitoring-agent","subsystem":"updater","time":1709992571283,"caller":"github.com/grafana/synthetic-monitoring-agent/internal/checks/checks.go:324","message":"fetching check configuration from synthetic-monitoring-api"}
{"level":"error","program":"synthetic-monitoring-agent","subsystem":"adhoc","error":"rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type \"text/plain\"","ok":true,"action":"registering ad-hoc probe with synthetic-monitoring-api","code":12,"time":1709992571307,"caller":"github.com/grafana/synthetic-monitoring-agent/internal/adhoc/adhoc.go:252","message":"unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type \"text/plain\""}
{"level":"error","program":"synthetic-monitoring-agent","subsystem":"adhoc","error":"API does not support required features","connection_state":"READY","time":1709992571307,"caller":"github.com/grafana/synthetic-monitoring-agent/internal/adhoc/adhoc.go:187","message":"cannot connect, bailing out"}
{"level":"error","program":"synthetic-monitoring-agent","subsystem":"updater","error":"rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type \"text/plain\"","action":"registering probe with synthetic-monitoring-api","code":12,"time":1709992571308,"caller":"github.com/grafana/synthetic-monitoring-agent/internal/checks/checks.go:330","message":"unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type \"text/plain\""}
{"level":"info","program":"synthetic-monitoring-agent","subsystem":"updater","error":"API does not support required features","was_connected":false,"connection_state":"READY","time":1709992571308,"caller":"github.com/grafana/synthetic-monitoring-agent/internal/checks/checks.go:259","message":"broke out of loop"}
{"level":"error","program":"synthetic-monitoring-agent","subsystem":"updater","error":"API does not support required features","connection_state":"READY","time":1709992571308,"caller":"github.com/grafana/synthetic-monitoring-agent/internal/checks/checks.go:294","message":"cannot connect, bailing out"}
{"level":"info","program":"synthetic-monitoring-agent","subsystem":"signal handler","time":1709992571308,"caller":"github.com/grafana/synthetic-monitoring-agent/cmd/synthetic-monitoring-agent/main.go:288","message":"shutting down"}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571309,"message":"2024/03/09 13:56:11 INFO: [core] [Channel #1] Channel Connectivity change to SHUTDOWN"}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571309,"message":"2024/03/09 13:56:11 INFO: [core] [Channel #1] Closing the name resolver"}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571309,"message":"2024/03/09 13:56:11 INFO: [core] [Channel #1] ccBalancerWrapper: closing"}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571309,"message":"2024/03/09 13:56:11 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to SHUTDOWN"}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571309,"message":"2024/03/09 13:56:11 INFO: [core] [Channel #1 SubChannel #2] Subchannel deleted"}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571310,"message":"2024/03/09 13:56:11 INFO: [transport] [client-transport 0xc00020a248] Closing: rpc error: code = Canceled desc = grpc: the client connection is closing"}
{"program":"synthetic-monitoring-agent","component":"grpc-go","time":1709992571310,"message":"2024/03/09 13:56:11 INFO: [core] [Channel #1] Channel deleted"}
ka3de commented 5 months ago

I believe the issue is because the value set for the --api-server-address is synthetic-monitoring-api-eu-west-3.grafana.net:443, but that points to the HTTP API, that's why the agent is complaining on received unexpected content-type \"text/plain\". Instead you have to set that argument to point to the gRPC API, which for eu-west-3 should be synthetic-monitoring-grpc-eu-west-3.grafana.net:443. You can see each region APIs URLs in the private probes docs here.

Let me know if that fixes the issue.

jfrankiewicz commented 5 months ago

yes, it was exactly this problem. Thanks.