search

jaegertracing / jaeger

CNCF Jaeger, a Distributed Tracing Platform
https://www.jaegertracing.io/
Apache License 2.0
20.23k stars 2.41k forks source link

No trace results in jaeger UI #468

Closed jackychee closed 6 years ago

jackychee commented 6 years ago

I run jaeger backend as an all-in-one docker image successfully. docker run -d -p6831:6831/udp -p16686:16686 jaegertracing/all-in-one:latest

netstat -anp | grep docker-proxy tcp6 0 0 :::16686 ::: LISTEN 11562/docker-proxy
udp6 0 0 :::6831 ::: 11572/docker-proxy

HotROD run successfully, logs as follow: [root@localhost hotrod]# go run ./main.go all 2017-10-12T17:25:30.975Z INFO cmd/all.go:25 Starting all services 2017-10-12T17:25:30.980Z INFO log/logger.go:37 Starting {"service": "frontend", "address": "http://127.0.0.1:8080"} 2017-10-12T17:25:30.988Z INFO log/logger.go:37 Starting {"service": "customer", "address": "http://127.0.0.1:8081"} 2017-10-12T17:25:30.991Z INFO log/logger.go:37 TChannel listening {"service": "driver", "hostPort": "127.0.0.1:8082"} 2017-10-12T17:25:30.993Z INFO log/logger.go:37 Starting {"service": "route", "address": "http://127.0.0.1:8083"}

2017-10-12T17:25:35.903Z INFO log/spanlogger.go:34 HTTP request received {"service": "frontend", "method": "GET", "url": "/dispatch?customer=392&nonse=0.5000926572517626"} 2017-10-12T17:25:35.903Z INFO log/spanlogger.go:34 Getting customer {"service": "frontend", "component": "customer_client", "customer_id": "392"} 2017-10-12T17:25:35.903Z INFO log/spanlogger.go:34 HTTP request received {"service": "customer", "method": "GET", "url": "/customer?customer=392"} 2017-10-12T17:25:35.903Z INFO log/spanlogger.go:34 Loading customer {"service": "customer", "component": "mysql", "customer_id": "392"} 2017-10-12T17:25:36.174Z INFO log/spanlogger.go:34 Found customer {"service": "frontend", "customer": {"ID":"392","Name":"Trom Chocolatier","Location":"577,322"}} 2017-10-12T17:25:36.175Z INFO log/spanlogger.go:34 Finding nearest drivers {"service": "frontend", "component": "driver_client", "location": "577,322"} 2017-10-12T17:25:36.178Z INFO log/spanlogger.go:34 Searching for nearby drivers {"service": "driver", "location": "577,322"} 2017-10-12T17:25:36.232Z ERROR log/spanlogger.go:39 redis timeout {"service": "driver", "driver_id": "T720212C", "error": "redis timeout"} github.com/uber/jaeger/examples/hotrod/vendor/go.uber.org/zap.Stack /root/go/src/github.com/uber/jaeger/examples/hotrod/vendor/go.uber.org/zap/field.go:209 github.com/uber/jaeger/examples/hotrod/vendor/go.uber.org/zap.(Logger).check /root/go/src/github.com/uber/jaeger/examples/hotrod/vendor/go.uber.org/zap/logger.go:273 github.com/uber/jaeger/examples/hotrod/vendor/go.uber.org/zap.(Logger).Error /root/go/src/github.com/uber/jaeger/examples/hotrod/vendor/go.uber.org/zap/logger.go:176 github.com/uber/jaeger/examples/hotrod/pkg/log.spanLogger.Error /root/go/src/github.com/uber/jaeger/examples/hotrod/pkg/log/spanlogger.go:39 github.com/uber/jaeger/examples/hotrod/pkg/log.(*spanLogger).Error

:1 github.com/uber/jaeger/examples/hotrod/services/driver.(*Redis).GetDriver /root/go/src/github.com/uber/jaeger/examples/hotrod/services/driver/redis.go:83 github.com/uber/jaeger/examples/hotrod/services/driver.(*Server).FindNearest /root/go/src/github.com/uber/jaeger/examples/hotrod/services/driver/server.go:85 github.com/uber/jaeger/examples/hotrod/services/driver/thrift-gen/driver.(*tchanDriverServer).handleFindNearest /root/go/src/github.com/uber/jaeger/examples/hotrod/services/driver/thrift-gen/driver/tchan-driver.go:92 2017-10-12T17:25:36.232Z ERROR log/spanlogger.go:39 Retrying GetDriver after error {"service": "driver", "retry_no": 1, "error": "redis timeout"}
jpkrohling commented 6 years ago

You need to expose the port 5775 as well:

$ docker run --rm -p5775:5775/udp -p6831:6831/udp -p16686:16686 jaegertracing/all-in-one:latest

Where did you get this command? We may need to fix the place where you got this command from.

jackychee commented 6 years ago

It's from: https://medium.com/opentracing/take-opentracing-for-a-hotrod-ride-f6e3141f7941

I use the following command but still no traces results, such as trace timeline view. docker run -d -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 -p5775:5775/udp -p6831:6831/udp -p6832:6832/udp \ -p5778:5778 -p16686:16686 -p14268:14268 -p9411:9411 jaegertracing/all-in-one:latest

response from 16686/api/services are: {"data":["mysql","customer","driver","redis","route","frontend"],"total":6,"limit":0,"offset":0,"errors":null}

should i execute spark job which derives dependency links and stores them directly to the storage?

jpkrohling commented 6 years ago

The blog post is probably outdated. I just tried here and it works for me:

In one session:

$ docker run --rm -p5775:5775/udp -p6831:6831/udp -p16686:16686 jaegertracing/all-in-one:latest

In another session:

$ cd $GOPATH/github.com/uber/jaeger/examples/hotrod
$ go run main.go all

Then click on the buttons on the UI available at http://127.0.0.1:8080/ :

image

and check for traces on Jaeger's UI:

image

For the all-in-one image, the Spark job isn't necessary:

image

jackychee commented 6 years ago

thanks, i run hotrod demo and all-in-one in centos machine, and use nginx as proxy for hotrod, there may be some mistake need carefully check.

yurishkuro commented 6 years ago

@jpkrohling p5775:5775/udp is for ZIpkin model, which is not used in Hotrod. I just tried the docker command exactly as shown on Medium, and it works fine.

jpkrohling commented 6 years ago

Interesting, I could swear I saw an error on HotRod related to this port and no traces on the UI. I added this port to the command and got the result from the screenshots above. Could we be seeing some intermittent issue, then ?

jpkrohling commented 6 years ago

Just tried again, and I indeed see no traces on the UI when I remove the mentioned port. On Hot Rod, I see the following on the logs:

2017-10-12T22:06:00.321+0200 ERROR log/logger.go:42 write udp 127.0.0.1:56530->127.0.0.1:5775: write: connection refused {"service": "route"}

yurishkuro commented 6 years ago

have you run glide install recently?

jpkrohling commented 6 years ago

have you run glide install recently?

I don't remember ever running it for Hot Rod :-) But I just got the latest master for Jaeger and updated the dependencies for both Jaeger itself and Hot Rod. Still got the same message on the logs, and no traces on the UI when the port 5775 isn't specified. Everything works as expected when the port 5775/udp is exposed.

2017-10-13T09:30:42.614+0200 ERROR log/logger.go:42 write udp 127.0.0.1:34715->127.0.0.1:5775: write: connection refused {"service": "route"}

jackychee commented 6 years ago

@jpkrohling I encountered the same situation when port 5775 isn't specified, and i have the log as follow:

2017-10-13T15:46:20.018+0800 ERROR log/logger.go:42 write udp 127.0.0.1:54652->127.0.0.1:5775: write: connection refused {"service": "frontend"} github.com/uber/jaeger/examples/hotrod/vendor/go.uber.org/zap.Stack

yurishkuro commented 6 years ago

@jpkrohling @jackychee please remove examples/hotrod/vendor and don't do make install_examples, just make install - I will update the instructions / tutorial.

jackychee commented 6 years ago

Thanks!

msn2708 commented 6 years ago

I am facing a similar issue. Here is my setup

Backend running inside Docker Toolbox for Windows, IP: 192.168.99.100 Frontend and other services running inside an Ubuntu running on a VMWare, IP: 192.168.159.129

The Jaeger UI (http://192.168.99.100:16686/search) is accesible from the Ubuntu machine running on VMWware

but none of the traces are being captured. Here is the error:

2017-12-05T19:16:07.113-0800 ERROR log/logger.go:42 write udp 127.0.0.1:40481->127.0.0.1:6831: write: connection refused {"service": "route"} github.com/jaegertracing/jaeger/examples/hotrod/pkg/log.logger.Error /home/msn2708/GoCode/src/github.com/jaegertracing/jaeger/examples/hotrod/pkg/log/logger.go:42 github.com/jaegertracing/jaeger/examples/hotrod/pkg/tracing.jaegerLoggerAdapter.Error /home/msn2708/GoCode/src/github.com/jaegertracing/jaeger/examples/hotrod/pkg/tracing/init.go:58 github.com/jaegertracing/jaeger/examples/hotrod/pkg/tracing.(*jaegerLoggerAdapter).Error

:1 github.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go.(*remoteReporter).flush /home/msn2708/GoCode/src/github.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go/reporter.go:258 github.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go.(*remoteReporter).processQueue /home/msn2708/GoCode/src/github.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go/reporter.go:246
samisamantha commented 5 years ago

Is there any update on this? I'm having the same issue. My project(not a Spring Boot project) is running on a different host and port (not localhost:8080) and my jaeger is running in docker. I can see the spans created in the project via logs but the Jaeger UI only shows the service for jaeger-query. I created a separate Spring Boot project to try if the trace will reflect if the project is running on localhost:8080 and it did. The corresponding jaeger service appeared on the Jaeger UI as well as the traces. Could you please help me with this?

yurishkuro commented 5 years ago

Are you using http sender in the Jaeger client? If you're using udp sender then you need to run the agent on the same host as the application.

If you are using http sender and the spans are still not reaching the collectors, I recommend bringing your networking toolbox (eg tcpdump) and looking at where the packets are going.

You can also look at the metrics emitted by various Jaeger components, to see where the data loss occurs.

Finally, or rather the very first thing, make sure you configured the client to sample all requests while you're investigating.

samisamantha commented 5 years ago

Hi @yurishkuro , thank you for responding. I'm running this script in docker "docker run --rm --name trace -p 5775:5775/udp -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 -p 16686:16686 -p 14267:14267 -p 14268:14268 -p 9411:9411 jaegertracing/all-in-one:1.7 " so I'm using udp sender. Is there a documentation on how to use http sender? I checked https://www.jaegertracing.io/docs/1.7/client-libraries/. However, the link for HTTPSender there just displays the HttpSender class on github. Thank you so much and I apologize for the inconvenience. Thank you.

By the way, for additional information, I'm using io.jaegertracing>jaeger-client>0.31.0

yurishkuro commented 5 years ago

@samisamantha see https://github.com/jaegertracing/jaeger-client-java/tree/master/jaeger-core for documentation about JAEGER_ENDPOINT env var (you can also set it programmatically on the Configuration class)

samisamantha commented 5 years ago

Hi @yurishkuro, thanks again for responding. I apologize for responding at a later time - I was assigned to a different ticket for a time.

I tried setting it programmatically: SenderConfiguration sender = new SenderConfiguration(); sender.withEndpoint("http://ipv4address:14268/api/traces"); ReporterConfiguration reporter = new ReporterConfiguration(); reporter.withLogSpans(true); reporter.withSender(sender); Configuration.fromEnv("sample_jaeger").withSampler(sampler).withReporter(reporter).getTracer();

I can see from the logs of the initialized tracer that I am now using HttpSender. However, I can't still see the service and the spans on the Jaeger UI

jpkrohling commented 5 years ago

@samisamantha could you also paste the code related to how you set the sampler?

samisamantha commented 5 years ago

Hi @jpkrohling, here it is: SamplerConfiguration sampler = new SamplerConfiguration(); sampler.withType("const"); sampler.withParam(1);

ronberna commented 5 years ago

Hello, I'm trying the example in Mastering Distributed Tracing Book. I'm not seeing any traces in the UI when I try running the HotROD app from the docker container using the following command:

docker run --rm -it \
  --link jaeger \
  -p8080-8083:8080-8083 \
  -e JAEGER_AGENT_HOST="jaeger" \
  jaegertracing/example-hotrod:latest \
all

I'm starting jaeger with the following command

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14268:14268 \
  -p 9411:9411 \
  jaegertracing/all-in-one:latest

When I go to the Jaeger UI I don't see any traces. It does work if I start the HotROD app from the src directory using:

cd $GOPATH/src/github.com/jaegertracing/jaeger
$ go run main.go all

Any help with getting the docker app to work would be greatly appreciated.

ronberna commented 5 years ago

Got it to work. Had to run the following docker command to start the hotrod app:

docker run \
  --rm \
  --link jaeger \
  --env JAEGER_AGENT_HOST=jaeger \
  --env JAEGER_AGENT_PORT=6831 \
  -p8080-8083:8080-8083 \
  jaegertracing/example-hotrod:latest \
  all