Question: How to configure Sleuth with Zipkin over RabbitMQ

[robachmann]

I would like to send spans through a RabbitMQ to Zipkin. Therefore I use the Zipkin distribution jar (zipkin-server-2.8.4-exec.jar) and deploy it to Cloud Foundry with these manifest arguments:

  services:
    - elasticsearch-zipkin
    - rabbitmq-zipkin
  env:
    RABBIT_URI: ${vcap.services.rabbitmq-zipkin.credentials.uri}

In the producer app with sleuth, I add the following dependencies:

    compile('org.springframework.boot:spring-boot-starter-amqp')
    compile('org.springframework.cloud:spring-cloud-starter-zipkin')
    compile('org.springframework.amqp:spring-rabbit')

Note that the first one is used to send other messages to another application through RMQ and the last one is just to debug as it's included in the first one.

The used Cloud and Boot versions are Finchley.RC2 and 2.0.2.RELEASE.

Finally, I add the following properties to application.yml:

  sleuth:
    sampler:
      probability: 1.0
    messaging:
      enabled: true
      rabbit:
        enabled: true
        remote-service-name: rabbitmq-zipkin
  zipkin:
    sender:
      type: rabbit
    service:
      name: myapp

Note the remote-service-name corresponds to the service name in Cloud Foundry.

When both apps are started, I see that zipkin-server auto-creates the queue named 'zipkin'. Also, log-files from the sleuth-app look like this:

2018-06-07T15:58:11.42+0200 [APP/PROC/WEB/0] OUT 2018-06-07 13:58:11.421  INFO [myapp,f21e0ded48255245,f21e0ded48255245,true] 8 --- [pool-5-thread-1] o.s.a.r.c.CachingConnectionFactory       : Created new connection: SpringAMQP#404913f6:0/SimpleConnection@6beb433f [delegate=amqp://Y342KiwcvGv34ruq@10.10.22.22:5672/9437a9a6-597c-4e94-b4d7-6cb85a2ab647, localPort= 53618]

Unfortunately, I never see the span written to RMQ, the queue named 'zipkin' remains empty.

Is there anything I missed to configure? What are possible ways to debug it?

[marcingrzejszczak]

Here you have a very simple example of 2 apps using rabbitmq as a sender of spans - https://github.com/openzipkin/sleuth-webmvc-example/tree/rabbitmq-sender

[robachmann]

Hi Marcin, thanks for the quick reply. I tried the repo and those examples are really helpful. Unfortunately, branch add-rabbit-tracing is currently not working. However, I see that there might be a misunderstanding: Is it possible to trace rabbitmq-messages between frontend and backend and send those traces through rabbitmq to the zipkin-server? I'm a bit confused because in this commit Adrian enables rabbit-tracing but switches spring.zipkin.sender.type to web.

[marcingrzejszczak]

I care to disagree. It's working - our ci jobs pass. Maybe you refer to issues with python. You have to use python 2 to run the scripts.

There are two things. Instrumenting rabbit and using rabbit to send spans. Can you precisely say what you're asking about? Anyways, both instrumenting rabbit template and sending spans via rabbit works fine

[robachmann]

I was trying the script on a clean host, so my local Maven repo was empty at that time. Now even add-rabbit-tracing works, I guess a snapshot was assumed in the local repo that wasn't initially there.

I would like to have both :-) Both messages and spans shall be sent to the same RabbitMQ service - in their respective queue of course. I wasn't able to combine the two examples to accomplish that but I'll try again later if you say that this use-case should be working fine.

[marcingrzejszczak]

I'll try to update the code to support such a case

[marcingrzejszczak]

I can't make it more explicit than that. https://github.com/openzipkin/sleuth-webmvc-example/commit/289c6bfabdd361ff9d82c2f52823fdd784afa5ef

No matter what I do, it just can't stop working ;)

RabbitMq queues:

A consumer of the test queue

Zipkin - the consumer of the zipkin queue

[robachmann]

Dang, I really appreciate your effort and I am able to reproduce your results locally but as soon as I push the apps to Cloud Foundry I don't see any traffic in the queue 'zipkin', so there's nothing published. For zipkin-server (the deployed jar), I set RABBIT_URI: ${vcap.services.rabbitmq-zipkin.credentials.uri} to pick up credentials from the bound service. Is there anything I need to do for the sleuth-apps? Publishing other messages works fine, so Spring generally seems to be able to make use of the bound service. I'm sorry for the circumstances but I'm out of ideas for this one. :(

[marcingrzejszczak]

For Sleuth apps just bind the services. Sleuth reuses the connection factory you have.

[robachmann]

Alright, good to know :-) I went ahead and saw that the Zipkin Sender is declared in ZipkinRabbitSenderConfiguration.class as follows:

@Value("${spring.zipkin.rabbitmq.queue:zipkin}")
private String queue;

@Bean Sender rabbitSender(CachingConnectionFactory connectionFactory, RabbitProperties config) {
    return RabbitMQSender.newBuilder()
            .connectionFactory(connectionFactory.getRabbitConnectionFactory())
            .queue(this.queue)
            .addresses(config.determineAddresses())
            .build();
}

To check which address is taken into account, I created a custom configuration and print the connection details upon starting the application:

@Value("${spring.zipkin.rabbitmq.queue:zipkin}")
private String queue;

@Autowired RabbitProperties config;
@Autowired CachingConnectionFactory connectionFactory;

@PostConstruct
void printConnectionConfig() {
    log.info("connectionFactory={}, queue={}, addresses={}",
            connectionFactory.getRabbitConnectionFactory(),
            this.queue,
            config.determineAddresses());
}

During deployment to Cloud Foundry, I read these log messages:

2018-06-08 17:34:00 [APP/PROC/WEB/0] OUT 2018-06-08 15:34:00.344 " INFO [producer,,,,]" 10 --- [ main] com.example.producer.BeanConfiguration : connectionFactory=com.rabbitmq.client.ConnectionFactory@2e3967ea, queue=zipkin, addresses=localhost:5672
2018-06-08 17:34:00 [APP/PROC/WEB/0] OUT 2018-06-08 15:34:00.417 " INFO [producer,,,,]" 10 --- [ main] o.s.a.r.c.CachingConnectionFactory : Attempting to connect to: [rabbitmq-node-0.service.consul, rabbitmq-node-1.service.consul, rabbitmq-node-2.service.consul]
2018-06-08 17:34:00 [APP/PROC/WEB/0] OUT 2018-06-08 15:34:00.485 " INFO [producer,,,,]" 10 --- [ main] o.s.a.r.c.CachingConnectionFactory : Created new connection: SpringAMQP#4ed51109:0/SimpleConnection@1a942c18 [delegate=amqp://myuser@10.20.30.40:5672/9437a9a6-597c-4e94-b4d7-6cb85a2ab647, localPort= 45840]

Connecting to the instance via SSH (cf ssh application-name) tells me that port 5672 isn't open:

vcap@instance-id:~$ netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
<..>
tcp        0      0 instance-id:45840 10.20.30.40:amqp     ESTABLISHED

so localhost:5672 might be a wrong address to connect to RabbitMQ? Adding the following dependencies unfortunately doesn't help either:

compile 'org.springframework.boot:spring-boot-starter-cloud-connectors'
compile 'org.springframework.cloud:spring-cloud-starter-cloudfoundry'

Is there anything else I can check? I would be interested in providing you logs of Sleuth writing whether it tries to send Spans to RMQ and if so, whether it succeeds or not.

Apologies for the long write-up I just want to include all details pointing to a possible solution.

[marcingrzejszczak]

@adriancole @shakuzen sth wrong with rabbitmq, cloud foundry and Sleuth connection factories. Have you seen such an exception?

[shakuzen]

Looks like it is successfully creating connections to me. localhost isn't being used. You can see from the log Created new connection the URI being used (because you're setting URI, not ADDRESSES). So unfortunately we still don't have enough information to know why spans aren't being published to the zipkin queue. Are you saying the same application is publishing other messages to RabbitMQ successfully? I'm typing on my phone so sorry for the brievity. I can try to help diagnose more later, but it may be easiest to do that on Gitter.

[robachmann]

Are you saying the same application is publishing other messages to RabbitMQ successfully?

Yes, exactly:

I appreciate your offer and joined your gitter. So if you find some time, I'd be more than happy to discuss it there.

[oiavorskyi]

I also have the same problem and the workaround provided in Gitter has helped me to solve it until the permanent solution is in place. For the benefit of others here is the workaround.

Explicitly configure spring.rabbitmq.addresses property to point to the credentials of the bounded RabbitMQ service. For example, if the bounded service is called rabbitmq the following configuration will work:

spring.rabbitmq.addresses=${vcap.services.rabbitmq.credentials.uri}

[shakuzen]

For other's reference, the conversation on Gitter starts here: https://gitter.im/spring-cloud/spring-cloud-sleuth?at=5b1e13a0ffd8896fe91c44c4

I believe we determined this issue only happens when using Spring Cloud Connectors to configure the RabbitMQ client in your application from the bound service. If you set spring.rabbitmq.addresses then Spring Boot (instead of Spring Cloud Connectors) will setup the necessary RabbitMQ client and everything should work.

@marcingrzejszczak I think we can use this issue to fix the reporter auto-configuration when Spring Cloud Connectors is used, or document that Spring Cloud Connectors doesn't work with the RabbitMQ reporter.

[vagner-nascimento]

Hi fellas.

I readded this issue and explored the example project to guide my adventure to build a POC to demonstrate a trace with spring cloud sleuth integratted with RabbitMQ and Zipkin.

After some days readding and adjusting thousen and thousen of Spring's version, i finnaly found compatibillity to run a fully traced application with 3 simple microservices.

Here's the result, hope be usefull for someone: https://github.com/vagner-nascimento/poc-zipkin-service/tree/adding-rabbitmq

OBS.: Builded with Gradle for the win!! <3 /,,/

[spring-projects-issues]

Closing due to age of the question. If you would like us to look at this issue, please comment and we will look at re-opening the issue.