search

jaegertracing / jaeger

CNCF Jaeger, a Distributed Tracing Platform
https://www.jaegertracing.io/
Apache License 2.0
20.25k stars 2.41k forks source link

Document and update integration test with the versions of Kafka Jaeger supports #3068

Open Ashmita152 opened 3 years ago

Ashmita152 commented 3 years ago

Requirement - what kind of business use case are you trying to solve?

Update jaeger-v2 kafka integration tests to test with the two most recent versions of Kafka

Problem - what in Jaeger blocks you from solving the requirement?

Right now we don't mention anywhere in the doc the version of kafka we support. This has lead to multiple open issues: https://github.com/jaegertracing/jaeger/issues/3059 https://github.com/jaegertracing/jaeger/issues/2950 https://github.com/jaegertracing/jaeger/issues/2068

Although Sarama says "Sarama is a Go library for Apache Kafka 0.8, and up." but their functional tests only test for kafka v2.7.1 and v2.8.0.

The dependabot or other PR bump up the version of Sarama which may break integration with older version of kafka.

Proposal - what do you suggest to solve the problem or improve the existing situation?

Update kafka integration test for each kafka versions we support just like how we do for ES

Any open questions to address

This will increase the integration test time but I think we should do it to prevent unknown breaking changes.

jpkrohling commented 3 years ago

We have a couple of action items here. An initial suggestion could be:

  1. Establish a version policy for Kafka support. For instance, in the Kubernetes world, it's common to support N-2 minor versions. Our promise on the operator is then that people can use any one of those 3 versions and expect Jaeger to work.
  2. Once the policy is established, document it.
  3. We can then use the latest version in the CI tests, with the other versions as part of a nightly build. This way, we keep the CI waiting times short, while still catching issues in a timely fashion. This can be a job that can be triggered on-demand as well so that we run it before releasing as well.

About the nightly jobs: I think we could even consider moving some of the tests we have from the CI to the nightly build as well.

Rinse and repeat for other storage mechanisms :-)

pavolloffay commented 3 years ago

Great topic :)

Note that in the future Jaeger will use Kafka implementation from OTEL collector. It's a newer implementation that does not use deprecated sarama-cluster implementation. Therefore the OTELcollector might be a better place for the itest.

dgoscn commented 3 years ago

Special thanks @Ashmita152.

Excellent topic