search

Pierre-Lannoy / wp-decalog

Capture and log events, metrics and traces on your site. Make WordPress observable - finally!
https://perfops.one/
GNU General Public License v3.0
64 stars 9 forks source link

Grafana Clound Tracing with Tempo gives Error 405 Method Not Allowed #55

Closed Damcios-s closed 1 year ago

Damcios-s commented 1 year ago

Describe the bug I am unable to configure the Grafana Cloud Traces logger. The best result I could get was an Error 405: Method not allowed. The Grafana Cloud Events logger works fine.

To Reproduce Steps to reproduce the behavior:

  1. Add a new Grafana Cloud Traces logger.
  2. Configure it with: sampling = 100%; format = Zipkin V2; Service URL = https://grafana_tempo_host:port
  3. Do something on your site
  4. Check logs (You need to have a logger configured). You will have warnings like: message="Pushing traces to https://host:port => Method Not Allowed"

Expected behavior Traces are sent correctly.

Environment (please complete the following information):

Additional context At first I thought that authentication was missing as Grafana Cloud mentions using basic auth with api key as password for Tempo. I tried setting the URL as: https://id:api_key@host:port but that does not change anything. Trying out some API calls to https://: I've seen that only GET request are accepted, all other request give an Error 405 Method Not Allowed. I tried following this tutorial: https://perfops.one/a-almost-free-observability-platform-part-1/ but it omits the most crucial part - the configuration - and simply tells me the Sending Traces page has all the config info I need.

Pierre-Lannoy commented 1 year ago

Hello @Damcios-s ! Thanks for the report. I think it's because Grafana Cloud no longer authorises unauthenticated pushes… Let me investigate it and I will come with a solution - well I hope so 😉

Pierre-Lannoy commented 1 year ago

OK. Found the culprit! That's me… Let me explain. The current Grafana Cloud Traces logger needs a local agent (see description: "Traces sent to Grafana Cloud via Grafana agent."). So you need to have a Grafana local agent installed on your server (or another server) to use this logger.

I've done it like that because, at the time I developed it, it was the only way: no direct push… Nevertheless it seems it's now possible to directly push traces to Grafana Cloud Tempo. So I will develop it for next DecaLog version.

Many thanks for this "discovery".

Pierre-Lannoy commented 1 year ago

Well, not good news, sorry. Grafana Cloud still does not allow direct push. So you need to go with local agent, that the only way. Sorry…

Damcios-s commented 1 year ago

Hi Pierre! Thank you for the quick answer! The most important thing I need right now are logs, and that is working great so thank you for the plugin!

About the agent, I have no access to the server, but you mentioned it could be installed on another server ? So I could push my traces (Prometheus metrics also work this way i believe?) to that server with the agent installed and then use Grafana Cloud to visualize them?

Pierre-Lannoy commented 1 year ago

Thanks for your kind words <3 Yes, the agent can be installed on another machine over which you have control…