search

Pierre-Lannoy / wp-decalog

Capture and log events, metrics and traces on your site. Make WordPress observable - finally!
https://perfops.one/
GNU General Public License v3.0
64 stars 8 forks source link

Grafana Clound Tracing with Tempo gives Error 405 Method Not Allowed #55

Closed Damcios-s closed 11 months ago

Damcios-s commented 11 months ago

Describe the bug I am unable to configure the Grafana Cloud Traces logger. The best result I could get was an Error 405: Method not allowed. The Grafana Cloud Events logger works fine.

To Reproduce Steps to reproduce the behavior:

  1. Add a new Grafana Cloud Traces logger.
  2. Configure it with: sampling = 100%; format = Zipkin V2; Service URL = https://grafana_tempo_host:port
  3. Do something on your site
  4. Check logs (You need to have a logger configured). You will have warnings like: message="Pushing traces to https://host:port => Method Not Allowed"

Expected behavior Traces are sent correctly.

Environment (please complete the following information):

Additional context At first I thought that authentication was missing as Grafana Cloud mentions using basic auth with api key as password for Tempo. I tried setting the URL as: https://id:api_key@host:port but that does not change anything. Trying out some API calls to https://: I've seen that only GET request are accepted, all other request give an Error 405 Method Not Allowed. I tried following this tutorial: https://perfops.one/a-almost-free-observability-platform-part-1/ but it omits the most crucial part - the configuration - and simply tells me the Sending Traces page has all the config info I need.

Pierre-Lannoy commented 11 months ago

Hello @Damcios-s ! Thanks for the report. I think it's because Grafana Cloud no longer authorises unauthenticated pushes… Let me investigate it and I will come with a solution - well I hope so 😉

Pierre-Lannoy commented 11 months ago

OK. Found the culprit! That's me… Let me explain. The current Grafana Cloud Traces logger needs a local agent (see description: "Traces sent to Grafana Cloud via Grafana agent."). So you need to have a Grafana local agent installed on your server (or another server) to use this logger.

I've done it like that because, at the time I developed it, it was the only way: no direct push… Nevertheless it seems it's now possible to directly push traces to Grafana Cloud Tempo. So I will develop it for next DecaLog version.

Many thanks for this "discovery".

Pierre-Lannoy commented 11 months ago

Well, not good news, sorry. Grafana Cloud still does not allow direct push. So you need to go with local agent, that the only way. Sorry…

Damcios-s commented 11 months ago

Hi Pierre! Thank you for the quick answer! The most important thing I need right now are logs, and that is working great so thank you for the plugin!

About the agent, I have no access to the server, but you mentioned it could be installed on another server ? So I could push my traces (Prometheus metrics also work this way i believe?) to that server with the agent installed and then use Grafana Cloud to visualize them?

Pierre-Lannoy commented 11 months ago

Thanks for your kind words <3 Yes, the agent can be installed on another machine over which you have control…