search

Kong / kong-plugin-zipkin

A Kong plugin for propogating zipkin spans and reporting spans to a zipkin server - this plugin has been moved into https://github.com/Kong/kong, please open issues and PRs in that repo
Apache License 2.0
60 stars 31 forks source link

Can zipkin collect data through kafka? #46

Closed JoeHooo closed 5 years ago

JoeHooo commented 5 years ago

With high concurrency, zipkin loses data image image

codefromthecrypt commented 5 years ago

similar question asked on the other issue. zipkin is mostly a dumb proxy. lost data is almost always elasticsearch queue overflow errors. unless you can carefully tune elasticsearch, enable storage throttling https://github.com/openzipkin/zipkin/tree/master/zipkin-server#throttled-storage-experimental

codefromthecrypt commented 5 years ago

also make sure you are sampling.. if you are doing 100% sampling with no rate-limit you can overwhelm any storage. Rate limiter is the best kind of sampler

shakuzen commented 5 years ago

To make this issue relevant to this repository, perhaps it could be refocused as "can the kong-plugin-zipkin report spans to Zipkin via Kafka?" From the Zipkin Server side, it can collect spans via Kafka. I'm not sure if this plugin supports reporting spans to Kafka or not, though. If not, this could be an enhancement request.

JoeHooo commented 5 years ago

To make this issue relevant to this repository, perhaps it could be refocused as "can the kong-plugin-zipkin report spans to Zipkin via Kafka?" From the Zipkin Server side, it can collect spans via Kafka. I'm not sure if this plugin supports reporting spans to Kafka or not, though. If not, this could be an enhancement request.

I thought about whether I could get kong's data through kafka. I set the environment variable KAFKA_BOOTSTRAP_SERVERS of zipkin, but felt no improvement.I guess kong's data is still sent over HTTP Now, my question is how to transfer kong's data to zipkin through kafka image This is my expected architecture diagram image I don't know if KAFKA_BOOTSTRAP_SERVERS are configured in the environment variable, so zipkin can collect data through kafka

JoeHooo commented 5 years ago

To make this issue relevant to this repository, perhaps it could be refocused as "can the kong-plugin-zipkin report spans to Zipkin via Kafka?" From the Zipkin Server side, it can collect spans via Kafka. I'm not sure if this plugin supports reporting spans to Kafka or not, though. If not, this could be an enhancement request.

Yes, I have the same doubts.

codefromthecrypt commented 5 years ago

there is no kafka code here. so yes this would be a new feature, and likely not that difficult to write if someone knows lua.