Incorrect timestamps on Kong phase start/end logs in Jaeger UI

Kong / kong-plugin-zipkin

A Kong plugin for propogating zipkin spans and reporting spans to a zipkin server - this plugin has been moved into https://github.com/Kong/kong, please open issues and PRs in that repo

Apache License 2.0

60 stars 31 forks source link

Incorrect timestamps on Kong phase start/end logs in Jaeger UI #85

Closed OmriHarary closed 4 years ago

OmriHarary commented 4 years ago

I currently have Kong and some backend services shipping spans in Zipkin format to the Zipkin-compatible endpoint on a Jaeger all-in-one instance, all running in Kubernetes and using the Kong Kubernetes Ingress Controller. For the most part everything is working, except the offset timestamps for the phase start/end logs viewed in the UI are all set to huge negative numbers that indicate the beginning of the epoch. See image below:

Zipkin-formatted spans received by the same endpoint from other sources correctly indicate the log time relative to the start of the trace:

Any assistance with this matter would be greatly appreciated.

Environment: Kong Image: kong:2.0 Ingress Controller Image: kong-docker-kubernetes-ingress-controller.bintray.io/kong-ingress-controller:0.8.0 Jaeger Image: jaegertracing/all-in-one:1.18.0

kikito commented 4 years ago

Jaeger is currently not a first-class citizen on this plugin. Some Jaeger features (like baggage headers) are included in order to maintain backward-compatibility with previous versions of the plugin. This might change in the future, but that's the current status.

I've done a cursory search and it seems that Jaeger comes with a couple endpoints specifically designed to import Zipkin data: https://www.jaegertracing.io/docs/1.18/getting-started/#migrating-from-zipkin

Are you using the /api/v2/spans endpoint in order to import the traces in Jaeger? My suspicion is that the default Jaeger endpoint expects milliseconds, while Zipkin uses microseconds on their timestamps. It might convert from microseconds to milliseconds by using that endpoint.

OmriHarary commented 4 years ago

@kikito I am using the Zipkin-compatible endpoints for importing the spans, yes. Other spans from Zipkin-based sources elsewhere in our applications are displaying just fine when sent to the same endpoint, as shown in the second screenshot above.

Edit: Sorry, to more specifically answer your actual question: I am in fact using the /api/v2/spans endpoint on Jaeger's Zipkin-compatibility collector listening on port 9411.

kikito commented 4 years ago

Hi, apologies for the late reply. I just noticed your Kong version:

kong:2.0

This version of kong uses an older version of the zipkin plugin (v0.2.0). The problem you are experiencing was known and solved in #71, but that bugfix is only available in the zipkin plugin v1.0.0. You have several options:

updating to kong 2.1 (which is currently in beta)
updating the zipkin plugin version of your kong 2.0 rockspec file to a newer version of the plugin (1.0.0 or above).

OmriHarary commented 4 years ago

Oh, excellent. Thank you @kikito, I don't know why I didn't see that PR when I was looking before creating this issue. I am not currently in a position to take either of those options, but I will be waiting for the 2.1 release for the resolution. I will close this issue at this time.

OmriHarary commented 3 years ago

This issue has been closed for a while, but for posterity: I have finally been able to test with a newer Kong release and I can confirm that this is no longer an issue.