Prerequisites to get trace details from java application (spring bootstrap) to Zipkins

[JasmithaMeka]

Is this a BUG or FEATURE REQUEST?:

Did you review istio.io/help and existing issues to identify if this is already solved or being worked on?:

Bug: N. Query

What Version of Istio and Kubernetes are you using, where did you get Istio from, Installation details

istioctl version - 0.4 
kubectl version - 1.7.8

Is Istio Auth enabled or not ? installed istio.yaml

What happened:

Hi, I'm trying to understand prerequisites to get trace details from java application (spring bootstrap) to Zipkins. I've deployed two Microservices - one as API provider and other as API consumer in K8s cluster with Istio. Envoy is injected into pods. Zipkins is deployed as pod. I didn't get trace details in Zipkins though I generated traffic. I understand from IBM book info demo that few headers (x-request-id, x-b3-traceid, x-b3-spanid, x-b3-parentspanid, x-b3-sampled, x-b3-flags and x-ot-span-context) need to be propagated across . Who sets these headers? I didn't use any instrumentation library. I would like to understand role of Envoy and Instrumentation libraries. Do I need to use Instrumentation libraries even though Im using Envoy? Kindly provide some insight.

[objectiser]

Did you deploy zipkin in the istio-system namespace? Have you tried tracing the bookinfo example and seen it working?

In Istio, the responsibility for creating tracing data for inbound and outbound requests is handled by the Envoy proxy. So the envoy proxy layer will create the trace (spans) and associated trace context (i.e. the headers you mentioned) that will be passed into the invoked service. The service then needs to propagate those headers to any outbound requests, to ensure that the outbound proxy etc creates further spans as part of the same trace.

You don't need to use any instrumentation libraries within your application - you could manually code the propagation of the headers as in the bookinfo application. However it is possible to use instrumentation (such as OpenTracing based library instrumentation) to automatically handle the propagation for you.

This means that your application could then also participate in the trace, creating its own spans for the internal work it performs.

I have an example that was originally written to run on plain Kubernetes, but has instructions on how to run on Istio. This may help to see how tracing can be obtained from both Istio/Envoy and the service (springboot).

This example uses Jaeger, but I also have a branch that uses the Zipkin server - the main difference is that the springboot services need to explicitly instantiate the tracer.

[JasmithaMeka]

Hi Gary Brown,

Thanks a lot for reply. Yes Zipkins is deployed in istio-system namespace and I could see metrics for book Info which is deployed default namespace.

I've deployed branch code you have refered as two Microservices in K8s with Istio. It is giving metrics.

Please correct me If I misunderstood. Here AccountMgrApplication, AccountMgrController, OrderMgrApplication and OrderMgrController classes form an Application right. When I deploy an application just with these classes. I could not see traces. But I was expecting output something like order (k8s service-name) called account (k8s service-name) like bookinfo. Did I miss anything?

I understood when you said Springboot services need to explicitly instantiate the tracer. Is it mandatory to instantiate the tracer for Java Applications with other frameworks as well? Could you refer an example with no tracing libraries in Java Application code and Tracing happens with Envoy.

[JasmithaMeka]

Is it like envoy sets trace context for each request entering a pod (say A). By propagating those headers to other pod (say B) along the request, We are overriding trace context generated by Envoy for request entering pod B?

[objectiser]

Hi @JasmithaMeka

So just to be clear, you deployed the services in the branch I referenced, and you were able to see traces in the zipkin UI?

I understood when you said Springboot services need to explicitly instantiate the tracer. Is it mandatory to instantiate the tracer for Java Applications with other frameworks as well?

If you look at the master branch of the example I referred to, you will see that it does not have code to instantiate the Tracer bean. This is because it uses a TracerResolver mechanism, that can be used with Jaeger but not Zipkin (at present), which automatically obtains the Tracer without any code changes.

So for tracer implementations that support it, the TracerResolver can be used to obtain a tracer implementation simply by adding it as a dependency (possibly with some environment variables defined).

Is it like envoy sets trace context for each request entering a pod (say A). By propagating those headers to other pod (say B) along the request, We are overriding trace context generated by Envoy for request entering pod B?

The trace context is set (or updated) whenever a new span is created - so the inbound proxy will create a span and pass that context to the service - which may or may not create its own spans (which would update the trace context) until eventually those headers (trace context) are passed out to the outbound proxy - which would also create a span (updating the tracing context) before passing the request to the called service's inbound proxy - and so on. So at each point in the chain where a span is created, it will result in the trace context being updated.

[JasmithaMeka]

Hi @objectiser

So just to be clear, you deployed the services in the branch I referenced, and you were able to see traces in the zipkin UI?

Yes. I have deployed the services in the istio-zipkin branch and I was able to see the metrics. Right now I'm working on only Zipkins not on Jaeger.

The trace context is set (or updated) whenever a new span is created - so the inbound proxy will create a span and pass that context to the service - which may or may not create its own spans (which would update the trace context) until eventually those headers (trace context) are passed out to the outbound proxy - which would also create a span (updating the tracing context) before passing the request to the called service's inbound proxy - and so on. So at each point in the chain where a span is created, it will result in the trace context being updated.

Thanks for the explanation. It is very helpful. I have modified some part of your code. I have removed MetricConfiguration class. I have passed headers from order to account (The way it was in BookInfo) I could not see traces.

Snippet from OrderMgrController class

 private static String accountMgrUrl = System.getProperty("ACCOUNTMGR_URL", "http://api1-tomcat.default.svc.cluster.local:8080");

    @Autowired
    private HttpServletRequest request;

    @RequestMapping("/buy")
    public String buy() throws InterruptedException {
        Thread.sleep(1 + (long)(Math.random()*500));
            HttpHeaders requestHeaders = new HttpHeaders();
            requestHeaders.set("myname","jasmi");
            requestHeaders.set("user-agent", request.getHeader("user-agent"));
            requestHeaders.set("x-request-id", request.getHeader("x-request-id"));
            requestHeaders.set("x-b3-traceid", request.getHeader("x-b3-traceid"));
            requestHeaders.set("x-b3-spanid", request.getHeader("x-b3-spanid"));
            requestHeaders.set("x-b3-parentspanid", request.getHeader("x-b3-parentspanid"));
            requestHeaders.set("x-b3-sampled", request.getHeader("x-b3-sampled"));
            requestHeaders.set("x-b3-flags", request.getHeader("x-b3-flags"));
            requestHeaders.set("x-ot-span-context", request.getHeader("x-ot-span-context"));

            HttpEntity<String> entity = new HttpEntity<String>(requestHeaders);
            System.out.println("-----");
            System.out.println(entity.getHeaders());
            ResponseEntity<String> response = restTemplate.postForEntity(accountMgrUrl + "/account", requestHeaders, String.class);
            return "BUY + " + response.getBody();

    }

Snippet from AccountMgrController class

@RestController
public class AccountMgrController {

    @Autowired
    private HttpServletRequest request;

      @RequestMapping("/account")
       public String getAccount() throws InterruptedException {

        List<String> list=new ArrayList<String>();

        list.add(request.getHeader("myname"));  
        list.add(request.getHeader("user-agent"));
        list.add(request.getHeader("x-request-id"));
        list.add(request.getHeader("x-b3-traceid"));
        list.add(request.getHeader("x-b3-spanid"));
        list.add(request.getHeader("x-b3-traceid"));
        list.add(request.getHeader("x-b3-parentspanid"));
        list.add(request.getHeader("x-b3-sampled"));
        list.add(request.getHeader("x-request-id"));
        list.add(request.getHeader("x-b3-flags"));

        System.out.println(list);

        Thread.sleep(1 + (long)(Math.random()*500));
        if (Math.random() > 0.8) {
            throw new RuntimeException("Failed to find account");
        }
        return "Account details";
    }

}

Pods are deployed with Envoy. Application is up and Zipkins is up. I don't see traces.

These are logs of order pod

---

{myname=[jasmi], user-agent=[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36], x-request-id=[null], x-b3-traceid=[null], x-b3-spanid=[null], x-b3-parentspanid=[null], x-b3-sampled=[null], x-b3-flags=[null], x-ot-span-context=[null]}

These are logs of account pod

[jasmi, Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36, , , , , , , , ]

Could you tell me why headers are not getting set?

[objectiser]

Hi,

From the changes you have described, I can't see why you are not getting any traces (even just from the inbound requests to the proxy) - but if tracing wasn't enabled, that would account for why you are not seeing headers in the app.

Could you submit you changes as a PR on the example repo, and I'll take a look.

[JasmithaMeka]

Hi, I have submitted.

[objectiser]

Thanks - but why have you submitted every small change as a separate PR? Can you create a single PR that has all of the changes?

[JasmithaMeka]

Hi, I have created a new branch in your repository and committed changes there but when I'm trying to push it, I'm getting below error

remote: Permission to objectiser/opentracing-prometheus-example.git denied to JasmithaMeka. fatal: unable to access 'https://github.com/objectiser/opentracing-prometheus-example.git/': The requested URL returned error: 403

Please suggest

[objectiser]

You need to fork the repo and create the branch in your own repo - and then submit the PR from there.