search

jaegertracing / jaeger

CNCF Jaeger, a Distributed Tracing Platform
https://www.jaegertracing.io/
Apache License 2.0
20.58k stars 2.45k forks source link

Services combobox formatting #727

Closed zsojma closed 6 years ago

zsojma commented 6 years ago

Hello, we just started using Jaeger in our infrastructure and it is really great tool! Thanks! ;)

We are using it together with Linkerd and unfortunatelly, the Linkerd logs names of services in really complex format like the following:

%/io.l5d.k8s.daemonset/l5d-system/http-incoming/l5d/#/io.l5d.k8s.http/k2-development/http/rightsmanagement

Which causes an issue in services listing where it is hard to find a concrete service:

image

Can you please take a look at it if there is a better way how to display the services? Thanks a lot.

yurishkuro commented 6 years ago

maybe this should be addressed in linkerd instead? It looks like instead of service name it gives unique instance name.

zsojma commented 6 years ago

Hmm, I am not an expert, but I think, that they are not instance names, but communication routes in a service mesh architecture. If you look at following picture, each logged service name reflects a communication arrow to and from the linkerd:

image

Here are examples of the service names logged:

These two services are logged for each application deployed in the cluster. We have multi node cluster and second route varies based on a node which the request came from. So, for our 6 nodes cluster and cca 20 applications deployed, it makes sense to see about 140 (20 + 6 * 20) services logged.

Therefore, based on my opinion, everything is logged correctly and it gives us an opportunity to analyze every route independently if required. Only issue here is, that we have hard time to localize a concrete route in the GUI... :)

Thanks

yurishkuro commented 6 years ago

why does each box say "application"? Are you running a monolith code in all those pods, or do you have actual microservices? If the latter, when you discuss something with your colleagues, which of the two sentences do you use?

If you need to distinguish different instances of each service in the trace, you can add extra dimensions as tracer/process tags, and we can figure out how to enhance the UI to surface those dimensions better.

zsojma commented 6 years ago

OK, I see your point. You are right, we are running microservices and we say: "to do X we need RightsMgmt service to call HRMgmt service"... (picture is just copy & paste from linkerd web)

But I am not exactly sure what to propose as an update to linkerd? Here is, how a trace looks like now: image (0.0.0.0/4140 etc. are addresses of linkerd proxies)

I understand, that there should be no difference whether a request is accepted by one node or other, but there should still be a difference if it is an incomming or outgoing communication, right?

So, if I remove information about nodes, something like this still remains:

> 0.0.0.0/80
  > /#/io.l5d.k8s/k2-development/http/orchestration
    > 0.0.0.0/4340
      > /#/io.l5d.k8s.grpc/k2-development/grpc/metadatamanagement
        > 0.0.0.0/4341
          > /#/io.l5d.k8s/k2-development/grpc/metadatamanagement
    > 0.0.0.0/4140
      > /#/io.l5d.k8s.http/k2-development/http/rightsmanagement
        > 0.0.0.0/4141
          > /#/io.l5d.k8s/k2-development/http/rightsmanagement

Which is still complex but maybe it can fit into the view :) But my bigger consideration here is, is this architecture correct or do I miss something which can make our further analysis too complicated or even not possible?

Thanks

yurishkuro commented 6 years ago

If it was up to me, I would have the following (assuming I get the names for IP-only nodes right):

> linkerd
  > orchestration
    > linkerd
      > metadatamanagement
        > linkerd
          > metadatamanagement
    > linkerd
      > rightsmanagement
        > linkerd
          > rightsmanagement

All the additional dimensions (inbound/outbound, IP, cluster name, etc) should be recorded as either span tags or Process tags. They will be accessible by expanding each individual span, but we can also enrich the UI to add extra columns between service name and the Gantt chart (see https://github.com/jaegertracing/jaeger-ui/issues/132).

zsojma commented 6 years ago

Thank you for the reply. I created an issue in Linkerd github: https://github.com/linkerd/linkerd/issues/1860

yurishkuro commented 6 years ago

Closing this since there's little we can do if the instrumentation generates such insanely long names like %/io.l5d.k8s.daemonset/l5d-system/http-incoming/l5d/#/io.l5d.k8s.http/k2-development/http/rightsmanagement - the issue is that the names themselves are largely unusable, not how Jaeger displays them.