Closed tigrannajaryan closed 4 years ago
I will be happy to help with this :)
I will soon be submitting a PR for generating and managing "Golden Data". It will have the following components and process steps:
@kbrockhoff great, this will be a very useful addition. Please make smaller incremental PRs if possible to make reviewing easier.
I plan to write these tests to verify the API in the generalize-testbed branch. If @pmcollins has not started, you can assign the ticket to me. Or else I am happy to advise on how to write the tests using the generalized testbed.
Either way works for me, @kbrockhoff . I was part way through a proof of concept for how to test for correctness: two pipelines, a pipeline under test and a test harness pipeline. The test harness pipeline has a processor that sends metrics to an exporter that is configured to talk to the pipeline under test, from which it is configured to also receive metrics. The same processor in the test harness pipeline compares the received metrics to what it sent. But maybe the generalize-testbed
branch is the way to go instead (I was mostly out last week and wasn't aware of it).
Originally, I think we, including @tigrannajaryan, thought that maybe you (or someone) could work on the traces tests and I could work on the metrics tests (I'm more familiar with metrics). But maybe the way to go is for one of us to hold off until the other has an implementation. I'm happy to be the one to hold off since it looks like you have made significantly more progress than I have.
I think since @kbrockhoff started the trace PICT generator it is best that he continues working on it and @pmcollins you can work on the similar capability and tests for metrics part. Pablo, you are right that it may be best to wait a bit until Kevin is done with testbed refactoring. Kevin, do you need more changes to the testbed after this PR is merged?
I was planning to do all the refactoring in one PR. Still have a few improvements to make yet before it is ready for merging.
Testbed changes have been merged to master. Correctness tests for traces have been completed as part of the PR. Work on metrics correctness tests can now proceed.
Closing this issue, correctness tests now exist.
We currently have E2E tests that benchmark the performance of various formats.
We also need E2E tests that verify the correctness of the Collector operation as it receives and exports the data in various formats. Performance tests don't verify this today. We need separate tests that will send telemetry data to the Collector, covering all possible variety of such data and then verify that the Collector exports this data precisely as it is supposed to be represented in the configured export format.
The preference is to have a matrix test that verifies many receiver/exporter combinations and uses golden data sets for verification.
Possible approach:
ScenarioTraceTranslation
and also call it in Contrib to test receivers and exporters in Contrib. Enable contrib processors (e.g. k8s processor).