Open Aaronontheweb opened 2 years ago
--agent.tags
is for enriching spans (which are payload), it has nothing to do with propagation of the bearer tokens (which exists at the protocol level)
@yurishkuro I fully agree, but it's what the Elastic folks are recommending for their integration with Jaeger agents. I'm just not sure why one method via the driver code works but agent.tags
doesn't for the same content.
Here's the relevant comment from the Elastic thread I mentioned: https://github.com/elastic/apm/issues/212#issuecomment-830976883
@Aaronontheweb I think you need $(ELASTIC_APM_TOKEN)
(parentheses) rather than ${ELASTIC_APM_TOKEN}
(curly braces) in your k8s yaml. If that doesn't fix it, please head over to https://discuss.elastic.co/c/observability/apm/58 and we can help you out there. I don't think this is a problem with Jaeger.
@yurishkuro I think the use of process tags for this is not great. We needed a way to have jaeger-agent send auth tokens and couldn't find a better way. If you have any recommendations I'd be keen to hear them.
Would you be open to a PR that adds an option to send gRPC request metadata? Something like --reporter.grpc.headers "key=value"
, which would result in key=value being added to outgoing metadata. I think this could also be useful when sending to opentelemetry-collector.
sgtm. we already have a similar flag to read bearer token from file when talking to ES
@Aaronontheweb I think you need
$(ELASTIC_APM_TOKEN)
(parentheses) rather than${ELASTIC_APM_TOKEN}
(curly braces) in your k8s yaml. If that doesn't fix it, please head over to https://discuss.elastic.co/c/observability/apm/58 and we can help you out there. I don't think this is a problem with Jaeger.
FWIW I did try that as well; same outcome. I'll report an issue on the Elastic APM tracker.
any news on this issue? Having the same problems sending traces over RPC to Elastic APM..
@pentago no news. Please open a topic at https://discuss.elastic.co/c/observability/apm/58.
Describe the bug While following the instructions for exporting spans from Jaeger Agents to Elastic Cloud https://github.com/elastic/apm/issues/212, I've not been able to get the recommended
agent.tags
format for the given agent to parse correctly using CLI arguments or theJAEGER_TAGS
environment variable for the client.The agent starts up fine, but I get the following error whenever we attempt to transmit spans over gRPC:
This is because the bearer token included in the span tags was not propagated in the correct format - even if I hard code the
ELASTIC_APM_TOKEN
value into theagent.tags
value.However, if I make the following call using the Jaeger C# client:
Then that works fine and my spans arrive in Elastic Cloud no problem.
To Reproduce Create an agent using the following configuration (using Kubernetes in my case):
Have a client propagate spans to it while using the
elastic-apm-auth=Bearer ${ELASTIC_APM_TOKEN}
tag - and if connected to Elastic Cloud you'll receive an authorization error. I'm not sure why you will - passing in the tag explicitly via the programmatic client configuration works fine, but not if you do it via environment variables or via theagent.tags
property.Expected behavior The content and the format of the
elastic-apm-auth=Bearer ${ELASTIC_APM_TOKEN}
tag should be preserved.Edit: fixed my YAML - had the relevant part commented out while I was troubleshooting.