Closed shrutich91 closed 1 year ago
I believe this is a basic Docker networking issue, but I'm adding the Jaeger label to this one.
cc @frzifus
hi @shrutich91, i agree with @jpkrohling. You should be able to fix it by adding --network=host
to both commands and replace endpoint jaeger:14250
by localhost:14250
.
Even with --network=host in both docker commands, and setting the endpoint to localhost:14250, I get the same error.
2022-10-20T05:12:06.328Z warn zapgrpc/zapgrpc.go:191 [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
"Addr": "localhost:14250",
"ServerName": "localhost:14250",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp [::1]:14250: connect: connection refused" {"grpc_log": true}
2022-10-20T05:12:07.326Z info jaegerexporter@v0.59.0/exporter.go:186 State of the connection with the Jaeger Collector backend {"kind": "exporter", "data_type": "traces", "name": "jaeger", "state": "TRANSIENT_FAILURE"}
Could you please try to get it working outside of Docker first? If it works, then you know the problem is related to the network.
Even I am facing the same issue, @shrutich91 , did you find any solution ?
@navin-rai did you try to run the collector out side of Docker as @jpkrohling has proposed?
@navin-rai did you try to run the collector out side of Docker as @jpkrohling has proposed?
I have deployed OTEL-COLLECTOR-Contrib on Kubernetes (locally [docker-desktop]) and I have spinned jegaer:all-in-one
in docker, still facing the same issue.
in Kubernetes or docker-desktop? Could you verify that both container (collector and jaeger instance) run in the same docker network and the jaeger instance is reachable with the hostname jaeger
? I guess you can use docker inspect
for that.
I assume its a network issue, since connecting from a local build works fine.
Start Jaeger
docker run --rm -it --network=host jaegertracing/all-in-one:1.36.0
Run local collector
./bin/otelcontribcol_linux_amd64 --config config.yaml
Config
---
receivers:
otlp:
protocols:
grpc:
http:
exporters:
logging:
logLevel: debug
jaeger:
endpoint: localhost:14250
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
exporters: [logging, jaeger]
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers
. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.
running docker has problem I think. If you run the following command it will work
Jaeger Docker
sudo docker run -d --name jaeger \ -p 16686:16686 \ -p 14268:14268 \ -p 14250:14250 \ jaegertracing/all-in-one:1.17 --log-level=debug
OpeTelemetry Collector
Download the appropriate collector from release page
$ wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.69.0/otelcol_0.69.0_linux_amd64.deb $ sudo dpkg -i otelcol_0.69.0_linux_amd64.deb
run
/usr/bin/otelcol --config otel-collector-config.yaml
From code specify the Otel Collector endpoint it will work. The Docker container does not work. I have tried lots of network stuff still does not work.
@anilknayak I agree. Its the same thing like https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/15291#issuecomment-1303151050.
It is important that both containers (otel + jaeger) can reach each other. For this, it must be ensured that both are located in the same virtual network. As an alternative, you can use the host network.
Example
$ docker run -d --rm -it --network=host jaegertracing/all-in-one:1.36.0
$ docker run -d --rm -it --network=host -v $(pwd)/cfg.yaml:/config.yaml:z otel/opentelemetry-collector-contrib:0.59.0 --config=/config.yaml
$ tracegen -otlp-insecure -duration 2s
Is using the host network really the only solution? I have the same problem, using docker-compose and all the containers are in the same network.
I'm running Docker on Windows. The strange thing is, when I ran the same stack a couple of weeks ago with Docker switched to "Linux containers", it was working just fine. Now I'm running the containers with Docker switched to "Windows containers" and the "platform=linux" setting on the Linux containers.
Everything except pushing traces to Jaeger works. Could it be some sort of Docker Windows networking issue?
Both Jaeger OTEL-collector images were pulled today using latest tag.
Error in OTEL-collector pushing to Jaeger:
Network of the stack:
docker-copose.yml:
prometheus:
image: homemade/prometheus-win:2.41.0nano2019
container_name: prometheus
platform: windows
ports:
- "30090:9090"
environment:
- TZ=GMT+0
volumes:
- ./prom:C:/config
grafana:
image: grafana/grafana
container_name: grafana
platform: linux
environment:
- TZ=GMT+0
ports:
- "30091:3000"
volumes:
- ./grafana-data/data:/var/lib/grafana
depends_on:
- prometheus
jaeger-all-in-one:
image: jaegertracing/all-in-one:latest
container_name: jaeger-all-in-one
platform: linux
command: ["--prometheus.server-url=http://prometheus:9090"]
environment:
- METRICS_STORAGE_TYPE=prometheus
- TZ=GMT+0
ports:
- "30092:16686"
- "14268"
- "14250"
otel-collector:
image: otel/opentelemetry-collector
container_name: otel-collector
platform: linux
environment:
- TZ=GMT+0
command: ["--config=/etc/otel-collector-config.yaml"]
volumes:
- ./otel:/etc
ports:
- "31888:1888" # pprof extension
- "8888:8888" # Prometheus metrics exposed by the collector
- "8889:8889" # Prometheus exporter metrics
- "13133:13133" # health_check extension
- "4317:4317" # OTLP gRPC receiver
- "4318:4318" # OTLP http receiver
- "55679:55679" # zpages extension
depends_on:
- jaeger-all-in-one
- prometheus
It seems that Jaeger all in one is not listening to port 14250, supporting only Thrift, not GRPC. There is thus no chance the OpenTelemetry Collector will be able to export to it. We either need a docker-compose configuration file with containers for each individual Jaeger service, Jaeger all-in-one be enhanced to listen to GRPC not just Thrift, or OpenTelemetry Collector export to Thrift in addition to GRPC.
The all in one Docker image needs to be recent enough. I mistakenly used 1.6, mislead by tags on Docker Hub. You need to use something like 1.42, or latest. Alternative to using host as a network is to create a virtual network and link the containers to it.
I have been spending considerable time on this but still couldn't figure out what the issue is. Just to confirm if whatever I am doing is right, does the collector listen on the http port 4318 for telemetry data and jaeger on http port 14250?
this is my config.yaml file:
receivers:
otlp:
protocols:
http:
exporters:
jaeger:
endpoint: localhost:14250
service:
pipelines:
traces:
receivers: [otlp]
exporters: [jaeger]
I am trying to use Jaeger
as exporter in kubernetes but getting error message as below:
Error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:
* error decoding 'exporters': unknown type: "jaeger" for id: "jaeger" (valid values: [opencensus prometheusremotewrite logging otlp otlphttp file kafka debug prometheus zipkin])
2023/10/21 04:55:21 collector server run finished with error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:
* error decoding 'exporters': unknown type: "jaeger" for id: "jaeger" (valid values: [opencensus prometheusremotewrite logging otlp otlphttp file kafka debug prometheus zipkin])
Is there something I am missing here? Or is it like Jaeger
has been deprecated in the latest version of OpenTRelemetry ?
exporters:
logging:
jaeger:
endpoint: "jaeger-collector.observability.svc.cluster.local:14250"
tls:
insecure: true
@shubhamjadhav1896: Are you using opentelemetry-collector-contrib
? I'm facing the same problem on v0.87.0
. I had a look, and yup, support for the Jaeger Exporter was dropped in v0.86.0
[1].
[1] https://github.com/open-telemetry/opentelemetry-collector-contrib/releases/tag/v0.86.0
@shubhamjadhav1896: Are you using
opentelemetry-collector-contrib
? I'm facing the same problem onv0.87.0
. I had a look, and yup, support for the Jaeger Exporter was dropped inv0.86.0
[1].[1] https://github.com/open-telemetry/opentelemetry-collector-contrib/releases/tag/v0.86.0
Yes. I am using the same. And the issue has been sorted for me. In place of jaeger, you should try using OTLP protocol as this has been a standard protocol which Jaeger community has mentioned and they have removed the client library of Jaeger protocol from May 2022.
@shubhamjadhav1896: Thank you!
@shubhamjadhav1896: Thank you!
Welcome 😊
This helped me, I hadn't changed the port also
https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/26675/files
Describe your environment I am using the following docker compose
Jaeger Docker:
OTEL Collector Docker:
OTEL Collector Config
What is the expected behavior? I want to export traces from collector to jaeger but getting an error
What is the actual behavior?