search

adrianco / spigo

Simulate Protocol Interactions in Go
Apache License 2.0
1.12k stars 111 forks source link

Flow to Kafka #67

Open codefromthecrypt opened 8 years ago

codefromthecrypt commented 8 years ago

Recent versions of zipkin accept json encoding in kafka messages. I wonder if we can add a feature to write directly to kafka while flow'ing.

https://github.com/adrianco/spigo/blob/master/tooling/flow/flow.go

One option could be to use go-zipkin as opposed to, or as an alternative to writing json. It would be neat as it could make this a tracing load generator! https://github.com/elodina/go-zipkin/blob/master/kafka.go

codefromthecrypt commented 8 years ago

@elodina are you interested enough to help with this?

hubayirp commented 8 years ago

+1 on both

On Mon, May 2, 2016 at 7:48 PM Adrian Cole notifications@github.com wrote:

Recent versions of zipkin accept json encoding in kafka messages. I wonder if we can add a feature to write directly to kafka while flow'ing.

https://github.com/adrianco/spigo/blob/master/tooling/flow/flow.go

One option could be to use go-zipkin as opposed to, or as an alternative to writing json. It would be neat as it could make this a tracing load generator! https://github.com/elodina/go-zipkin/blob/master/kafka.go

@elondina are you interested enough to help with this?

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/adrianco/spigo/issues/67

adrianco commented 8 years ago

I looked at go-zipkin before and it makes too many assumptions about running on real systems I think. All we need is the basic interface wrappers for Kafka, as I already make the zipkin objects.

codefromthecrypt commented 8 years ago

zipkin's kafka support accepts the same json as POST, so yeah this should be doable

codefromthecrypt commented 8 years ago

@basvanbeek does this look like something you might be able to help with (plumb in the ability to write directly to kafka (or http) in addition to writing to a file?)

basvanbeek commented 8 years ago

I might be. Not sure yet when I have time for this though.

adrianco commented 8 years ago

How hard is it to get Kafka itself in place to test against?

On Wednesday, May 25, 2016, Bas van Beek notifications@github.com wrote:

I might be. Not sure yet when I have time for this though.

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/adrianco/spigo/issues/67#issuecomment-221553339

codefromthecrypt commented 8 years ago

Here's a paste of notes from here: https://github.com/openzipkin/zipkin-java/tree/master/zipkin-server#kafka-collector

$ export KAFKA_ZOOKEEPER=$(docker-machine ip docker-machine active)# Run Kafka in the background $ docker run -d -p 2181:2181 -p 9092:9092 \ --env ADVERTISED_HOST=$KAFKA_ZOOKEEPER \ --env AUTO_CREATE_TOPICS=true \ spotify/kafka# Start the zipkin server, which reads $KAFKA_ZOOKEEPER $ java -jar ./zipkin-server/target/zipkin-server-*exec.jar

On Wed, May 25, 2016 at 10:47 PM, Adrian Cockcroft <notifications@github.com

wrote:

How hard is it to get Kafka itself in place to test against?

On Wednesday, May 25, 2016, Bas van Beek notifications@github.com wrote:

I might be. Not sure yet when I have time for this though.

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/adrianco/spigo/issues/67#issuecomment-221553339

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/adrianco/spigo/issues/67#issuecomment-221776487

basvanbeek commented 8 years ago

I created a docker-compose file with Zipkin + Zookeeper + Kafka + Cassandra which can be found here:

https://github.com/go-kit/kit/blob/master/tracing/zipkin/_docker/docker-compose-zipkin.yml

you can start it like this:

HOSTNAME=localhost docker-compose -f docker-compose-zipkin.yml up
hubayirp commented 8 years ago

Getting Kafka in place is not the hard part though..

On Wed, May 25, 2016 at 9:47 PM Adrian Cockcroft notifications@github.com wrote:

How hard is it to get Kafka itself in place to test against?

On Wednesday, May 25, 2016, Bas van Beek notifications@github.com wrote:

I might be. Not sure yet when I have time for this though.

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/adrianco/spigo/issues/67#issuecomment-221553339

— You are receiving this because you commented.

Reply to this email directly or view it on GitHub https://github.com/adrianco/spigo/issues/67#issuecomment-221776487

codefromthecrypt commented 8 years ago

hard or not.. here's a kafka file that seems to work well enough https://github.com/openzipkin/docker-zipkin/pull/105

hubayirp commented 8 years ago

👍🏽 On Fri, Jul 22, 2016 at 1:48 AM Adrian Cole notifications@github.com wrote:

hard or not.. here's a kafka file that seems to work well enough openzipkin/docker-zipkin#105 https://github.com/openzipkin/docker-zipkin/pull/105

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/adrianco/spigo/issues/67#issuecomment-234489129, or mute the thread https://github.com/notifications/unsubscribe-auth/ACWGWHFVPuiCLwbCQOOfwUU6FQa3S4ecks5qYIPqgaJpZM4IV453 .