Tried to connect exporter using oltp to grafana tempo error when providing authorization headers

[dimas-adriyanto]

Describe the bug I tried to connect my app to grafana tempo using collector otel here is my otel config :

exporters: otlp: endpoint: tempo-eu-west-0.grafana.net:443 headers: authorization: Basic <xxxxxxxxxxx>

always showing this error : opentelementry-collector-collector-1 | 2022-04-20T04:03:28.025Z info exporterhelper/queued_retry.go:276 Exporting failed. Will retry the request after interval{"component_kind": "exporter", "component_type": "otlp", "component_name": "otlp", "error": "failed to push trace data via OTLP exporter: rpc error: code = Unauthenticated desc = Unauthorized: HTTP status code 401; transport: received the unexpected content-type \"application/json\"", "interval": "17.923638552s"}

Steps to reproduce

https://grafana.com/blog/2021/04/13/how-to-send-traces-to-grafana-clouds-tempo-service-with-opentelemetry-collector/#running-the-collector

I'm following that blog and using my own grafana tempo credentials

What did you expect to see? connected to grafana tempo using collector config

What did you see instead? A clear and concise description of what you saw instead.

What version did you use? Version: (e.g., v0.4.0, 1eb551b, etc)

What config did you use? Config: (e.g. the yaml config file)

Environment OS: (e.g., "Ubuntu 20.04") Compiler(if manually compiled): (e.g., "go 14.2")

Additional context Add any other context about the problem here.

[mx-psi]

I think this does not belong here but rather you should ask Grafana for support; AFAICT there is no indication that the problem is in the Collector side. cc @jpkrohling in case you can help

[jpkrohling]

Could you please share your config, minus the credentials? I just tried the following and it worked for me:

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  otlp:
    endpoint: tempo-us-central1.grafana.net:443
    headers:
      authorization: Basic $DIGEST

extensions:
  zpages:

service:
  extensions: [zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

Note that $DIGEST is the output of:

echo -n "user:pass" | base64 -w 0

And user is typically your 5-digit account number, with pass being your key.

[dimas-adriyanto]

@jpkrohling here is my config :

extensions:
  health_check:
  pprof:
    endpoint: :1888
  zpages:
    endpoint: :55679

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
 otlp:
   endpoint: tempo-eu-west-0.grafana.net:443
   headers:
     authorization: Basic user <xxxxxx>

service:
  extensions: [ pprof , zpages , health_check ]
  pipelines:
    traces:
      receivers: [ otlp ]
      processors: [ batch ]
      exporters: [ otlp ]

echo -n "<198587>:" | base64

it's work using your configuration , I guess I was using the wrong encryption password but why the error said "received the unexpected content-type" it's looks like I'm sending http format .thanks for the help 🙏🏻

[jpkrohling]

Not sure why you were receiving a "unexpected content-type" message but I'm glad it's working now. I'm cc'ing @mapno, perhaps he knows what might have gone wrong there. In any case, I'm closing this as I think this is now solved.

[klochowicz]

hi! please pardon resurrecting a closed issue, but I've come across exactly same error; despite adjusting the authorization header in all possible ways with fresh API keys, I have not been able to connect to grafana tempo successfully. I have reached out to grafana support in the meantime, will post answer here if I solve the issue this way.

[jdhiro]

I just ran into this as well while assessing Grafana (Cloud) Tempo. I think Grafana introduced a bug when making changes to API Key / Cloud Access Policies.

I started by generating a password in the Tempo interface. It didn't work, no matter what I tried. I went in an inspected the Cloud Access Policies that the password was attached to and I noticed... it had no write permissions at all. I created a new policy with trace:write permissions and generated a token under that.

Furthermore, from the docs, it seems there is an easier way to use the token in your config.

extensions:
  basicauth/traces:
    client_auth:
      username: "<username>"
      password: "<password>"

exporters:
  otlp:
    endpoint: tempo-prod-15-prod-us-west-0.grafana.net:443
    auth:
      authenticator: basicauth/traces

service:
  extensions: [ basicauth/traces ]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

With that, my 401 errors are gone. But now I'm getting throttling errors... sigh. Next thing.

[jpkrohling]

@jdhiro, if you still need help, please join the Grafana community slack, either #tempo or #opentelemetry channels: https://slack.grafana.com