search

openzipkin-attic / docker-zipkin

Docker images for OpenZipkin
Apache License 2.0
688 stars 329 forks source link

Sending traces with Kafka doesn't seem to work #146

Open elyara opened 7 years ago

elyara commented 7 years ago

I tried both docker-compose-kafka.yml and docker-compose-kafka10.yml but nothing. in the logs I see errors like:

zipkin             | 2017-06-04 14:04:23.548  INFO 5 --- [ KafkaCollector] org.apache.zookeeper.ZooKeeper           : Initiating client connection, connectString=kafka-zookeeper sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@1bb9fb75
zipkin             | 2017-06-04 14:04:23.686  INFO 5 --- [n_default:2181)] org.apache.zookeeper.ClientCnxn          : Opening socket connection to server kafka-zookeeper.dockerzipkin_default/172.21.0.3:2181. Will not attempt to authenticate using SASL (unknown error)
kafka-zookeeper    | 2017-06-04 14:04:23,709 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /172.21.0.5:50974
zipkin             | 2017-06-04 14:04:23.715  INFO 5 --- [n_default:2181)] org.apache.zookeeper.ClientCnxn          : Socket connection established to kafka-zookeeper.dockerzipkin_default/172.21.0.3:2181, initiating session
kafka-zookeeper    | 2017-06-04 14:04:23,732 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@928] - Client attempting to establish new session at /172.21.0.5:50974
kafka-zookeeper    | 2017-06-04 14:04:23,749 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@673] - Established session 0x15c736b7e5b0001 with negotiated timeout 6000 for client /172.21.0.5:50974
zipkin             | 2017-06-04 14:04:23.765  INFO 5 --- [n_default:2181)] org.apache.zookeeper.ClientCnxn          : Session establishment complete on server kafka-zookeeper.dockerzipkin_default/172.21.0.3:2181, sessionid = 0x15c736b7e5b0001, negotiated timeout = 6000
zipkin             | 2017-06-04 14:04:23.831  INFO 5 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 9411 (http)
zipkin             | 2017-06-04 14:04:23.851  INFO 5 --- [           main] zipkin.server.ZipkinServer               : Started ZipkinServer in 17.062 seconds (JVM running for 19.819)
kafka-zookeeper    | 2017-06-04 14:04:24,045 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@649] - Got user-level KeeperException when processing sessionid:0x15c736b7e5b0001 type:create cxid:0x1 zxid:0x1c txntype:-1 reqpath:n/a Error Path:/consumers/zipkin/ids Error:KeeperErrorCode = NoNode for /consumers/zipkin/ids
kafka-zookeeper    | 2017-06-04 14:04:24,072 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@649] - Got user-level KeeperException when processing sessionid:0x15c736b7e5b0001 type:create cxid:0x2 zxid:0x1d txntype:-1 reqpath:n/a Error Path:/consumers/zipkin Error:KeeperErrorCode = NoNode for /consumers/zipkin
zipkin             | 2017-06-04 14:05:20.677  INFO 5 --- [nio-9411-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring FrameworkServlet 'dispatcherServlet'
zipkin             | 2017-06-04 14:05:20.677  INFO 5 --- [nio-9411-exec-1] o.s.web.servlet.DispatcherServlet        : FrameworkServlet 'dispatcherServlet': initialization started
zipkin             | 2017-06-04 14:05:20.711  INFO 5 --- [nio-9411-exec-1] o.s.web.servlet.DispatcherServlet        : FrameworkServlet 'dispatcherServlet': initialization completed in 33 ms
kafka-zookeeper    | 2017-06-04 14:06:57,318 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@649] - Got user-level KeeperException when processing sessionid:0x15c736b7e5b0000 type:setData cxid:0x4b zxid:0x21 txntype:-1 reqpath:n/a Error Path:/config/topics/zipkin_kafka Error:KeeperErrorCode = NoNode for /config/topics/zipkin_kafka
kafka-zookeeper    | 2017-06-04 14:06:57,334 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@649] - Got user-level KeeperException when processing sessionid:0x15c736b7e5b0000 type:create cxid:0x4c zxid:0x22 txntype:-1 reqpath:n/a Error Path:/config/topics Error:KeeperErrorCode = NodeExists for /config/topics
kafka-zookeeper    | 2017-06-04 14:06:57,460 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@649] - Got user-level KeeperException when processing sessionid:0x15c736b7e5b0000 type:create cxid:0x54 zxid:0x25 txntype:-1 reqpath:n/a Error Path:/brokers/topics/zipkin_kafka/partitions/0 Error:KeeperErrorCode = NoNode for /brokers/topics/zipkin_kafka/partitions/0
kafka-zookeeper    | 2017-06-04 14:06:57,481 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@649] - Got user-level KeeperException when processing sessionid:0x15c736b7e5b0000 type:create cxid:0x55 zxid:0x26 txntype:-1 reqpath:n/a Error Path:/brokers/topics/zipkin_kafka/partitions Error:KeeperErrorCode = NoNode for /brokers/topics/zipkin_kafka/partitions
codefromthecrypt commented 7 years ago

Sorry you are having trouble. I usually use the brave-webmvc-example to test this. Note that the docker compose files have an advertized ip which is what you use from your apps. Please hop on gitter openzipkin/zipkin if you need more pointers

elyara commented 7 years ago

Not sure how brave-webmvc-example helps me to test Kafka transport. Unless I missed something it uses HTTP: https://github.com/openzipkin/brave-webmvc-example/blob/master/servlet3/src/main/java/brave/webmvc/TracingConfiguration.java#L36 Anyway I'll elaborate on the issue: I use zipkin-ruby and the deafult HTTP tracing works flawlessly. However with our scale I need to use Kafka instead. Now putting aside the fact that currently the zipkin-ruby needs a fix when setting it up with non hermann kafka (I'll open a PR with the fix later), the whole docker setup with Kafka fails. With the proper fixes I know that my app sends and kafka-zookeeper receives the messages, but then it throws the errors I pasted above. for Kafka client I use https://github.com/zendesk/ruby-kafka. to properly use it I set an alias (:push, :deliver_message)

codefromthecrypt commented 7 years ago

Hi, elya. Thanks for the reply back.

When I test the docker image, I use the brave example, but change from the default http transport to kafka. I didn't mention that earlier.

The reason I asked was that usually there's something awry with instrumentation. Knowing how it was tested helps identify how to better help you. My guess is that you'll want to use the topic "zipkin" not "zipkin_kafka"

There are some subtlies in zipkin-ruby that seem slightly amiss.. If you can help correct these, it would be appreciated:

Currently, we don't have a ruby example, which I'd love to have a remedy for, if you could help make a version of what others look like. This is how we usually troubleshoot things, and a great way to have others be able to help easier. For example, if I created a repo like the others for rack or whatever you think is better, and you populated it with working code.. well, it would be easier for me or someone else to help next time. https://github.com/openzipkin?utf8=%E2%9C%93&q=example&type=&language=

I appreciate this is asking a lot, but zipkin is volunteer driven. If you don't have time no big deal, just mentioning in case you can help us.

elyara commented 7 years ago

The issue was with zipkin-ruby after all @adriancole you are welcome to take a look https://github.com/openzipkin/zipkin-ruby/pull/107