RabbitMQ collector support

[shakuzen]

As mentioned by Adrian on Gitter, support for RabbitMQ has been requested multiple times, so I am making this issue to track it. Those who would like to see this implemented can vote for it. (As it were, there seemed to be some interest in it at the Zipkin Meetup tonight)

@byeswanth @shakuzen rabbitmq has been incidentally requested many times, and custom implemented in many zipkin systems. However, it has never been requested as a standard transport. You could be the first to actually make an issue requesting it

Deliverables:

• [x] Defining the transport data format (Span2?)
• [x] Implementing a collector that accepts that format #1742

---

Out of scope for this issue (should be implemented in other repositories), but can be listed here for convenience when searching this topic.

• Implementing reporters that speak defined format
  • openzipkin/zipkin-reporter-java#58

Feel free to comment/edit this if anything is missing.

[shakuzen]

It is also worth noting that Spring Cloud Sleuth is one library that offers an option to use RabbitMQ via Spring Cloud Stream and a custom Zipkin server (Zipkin Stream Server), as per the documentation. That implementation could be a good reference for standardizing this.

[codefromthecrypt]

Thanks for finally raising this! I'd like to see a guage of sites interested and also those who have rabbit experience or otherwise could help maintain this.

On data format, it is best to go with what's supported on the others imho. Serialized json or thrift lists. This allows the plumbing to be swappable.

Anecdotally, I've heard one reason for popularity is cloud foundry deployments which usually or always have rabbit.

Another thing to note here is the version range if any to indicate support of.

Finally, and more of a note, spring-cloud-sleuth stream collector indeed supports rabbit. However, the data format is sleuth-specific which would interfere especially with non java apps who want to share the transport (as there is no thrift or schema for that and also it is bespoke)

[codefromthecrypt]

so.. folks interested, please click thumbs up or otherwise note interest! Pretty sure there is interest, but we'd like to make sure this is easy to see (as we can ping you when any feature comes out)

[cforce]

No idea where I gets thumbs here 👍

[nheitz]

Definitely interested...this is key channel for us that needs to be properly traced. +1

[codefromthecrypt]

I'm interested in this because very often folks pasting problems with classpath are those trying to workaround lack of RabbitMQ support and instead using Spring Cloud Stream. Here's a recent example, but not the only one: https://stackoverflow.com/questions/44997050/sleuth-zipkins-with-elasticsearch

Having RabbitMQ out of the box can help avoid using our time for undifferentiated maven/eclipse support and focus it on things about zipkin instead.

[amitsaha]

hey all, I assume this issue here is about being able to achieve the following workflow:

1. Client code publishes a span to a certain RMQ queue
2. ZipKin directly reads from this queue

If so, if I were to implement the above before this issue is implemented, I would do it the following way:

1. Same as above
2. Write my own consumer which will read off the queue
3. The consumer in (2) will then make HTTP request (or Kafka, Scribe) to ZipKin server with the span

Is my understanding correct?

[codefromthecrypt]

you could do it that way, which is more an adapter model. A more direct way would be to make a java component which reads off the queue and writes to storage (similar to SQS in zipkin-aws project). On the sender side, it depends on the language, but if java for example a part of the zipkin-reporter project. The spans themselves should be encoded the same way as http body. Regardless, I think we'd need to settle any minimum version of RMQ.

[amitsaha]

Thanks for the explanation and pointer to zipkin-aws. In my case the services are non-Java (Python, golang, PHP).

[codefromthecrypt]

Thanks for the explanation and pointer to zipkin-aws. In my case the services are non-Java (Python, golang, PHP).

gotcha, so for now, you can indeed write an adapter and write the py, go, php producers to send in zipkin's http format (json or thrift). At some point, some server will directly implement rabbit obviating your RMQ -> HTTP or otherwise adapter.

The encoding is here (don't assume "legacy encoding" as that isn't necessary for new work) https://github.com/openzipkin/zipkin/tree/master/zipkin-collector/kafka10#encoding-spans-into-kafka-messages

Note we will soon have a simpler encoding, too, which is one reason I've not pushed on this work as yet. https://github.com/openzipkin/zipkin/issues/1644

[codefromthecrypt]

encoding prep work is done. We need to pick a dependency light or free rabbitmq java driver (ex one that doesn't pin to a spring version), and also a version of rabbit MQ to test against.

[shakuzen]

There is an official RabbitMQ Java client: https://github.com/rabbitmq/rabbitmq-java-client (only depends on SLF4J, I believe) Since version 4.0, its releases are decoupled from the RabbitMQ server releases while compatibility is still maintained, which is nice.

Regarding the version of RabbitMQ to test against, I think we are probably fine to use the latest stable (currently 3.6.11). I took a quick look at some SaaS offerings for RabbitMQ (AmqpHosting.com, CloudAMQP, and Compose) and cloud providers' RabbitMQ offerings (Azure, Google Cloud Platform, and Pivotal Cloud Foundry). They all seem to keep up-to-date with the latest stable, and most of the SaaS offerings provide one-click RabbitMQ cluster version upgrades.

That said, companies managing their own RabbitMQ servers are quite likely not on the latest version. Even so, our use case should be straight forward enough. That is, I wouldn't expect our use case to have had backwards incompatibilities introduced in the past few years (famous last words). Since there is a RabbitMQ server Docker image, we could use TestContainers to test against any version of the server we want fairly easily.

---

Also, let me reduce the scope on this issue to just collection since that is all that would be implemented in this repo. We can still list tracers that support reporting to RabbitMQ here after collection is implemented in case people find this issue when searching. I will open and link an issue in zipkin-reporter-java for reporting to RabbitMQ, for now.

[codefromthecrypt]

zipkin-server 2.1 supports rabbitmq

tracers who want to support this will need to add a transport/reporter which produces messages (by default to the queue named "zipkin") which are encoded lists (ex a json list of v1 or v2 span format)