search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.7k stars 8.24k forks source link

Ingest PR CI Test Results into kibana-stats cluster to better understand test coverage #77094

Open MindyRS opened 4 years ago

MindyRS commented 4 years ago

Describe the feature: As a Test Lead, I need to understand the tests that get run and their high level results (skipped, passed, failed). I will use this data to identify feature areas that aren't getting tested and provide that information back to Dev Leads. This data will be used in combination with the Test Coverage data build out by Tre and Dima.

Describe a specific use case for the feature:

I would like to get this information (Class, Duration, Skip, Pass, Total) from here: https://kibana-ci.elastic.co/view/Kibana/job/elastic+kibana+master/lastSuccessfulBuild/testReport/X-Pack%20Jest%20Tests/ and put it into Lee Drengenberg's https://kibana-stats.elastic.dev/ instance.

I can then parse out the class path to get at everything that belongs to Security Solutions (the combined SIEM/Endpoint teams). From there I know which teams within that group are working on different features (detections, overview, network, timeline...). I can then give the Dev Lead this information so that they can better understand what their team is producing in the way of automation. Individual devs can benefit from this too by getting an understanding of the duration their tests take to run.

elasticmachine commented 4 years ago

Pinging @elastic/kibana-qa (Team:QA)

elasticmachine commented 4 years ago

Pinging @elastic/kibana-operations (Team:Operations)

LeeDr commented 4 years ago

I just realized we don't run all the tests in the Code Coverage job. We only run the ones that we can collect coverage on. Recent run has 31,945 tests compared to a recent master CI job 42,430 tests.

We had discussed indexing all the test data in the past where the purpose was to track changes in how long different tests took as some measure of performance. We've turned away from that purpose in favor of looking at bundle size and load testing. But here's another use case for indexing test results and durations.

spalger commented 4 years ago

I think https://github.com/elastic/kibana/issues/79480 will address part of this too

tylersmalley commented 2 years ago

@spalger, I believe we are now collecting this - but we need to create a dashboard or something accessible to folks to use?

spalger commented 2 years ago

Yeah, we have this data but it's a lot of data to query and to replicate to a user-facing location like kibana-stats. About one month's worth of test data comes out to about 1.5TB. It's not absurd, but I think it would be nice to have more than one months data to look through. I'm thinking through two options:

  1. summarize the tests per day and branch and store a daily summary of how many times the test ran, average duration, skipped status at the end of the day, success/failure/skipped counts, along with identify information like name/file/type for each test
  2. Replicate the per-execution documents just for the main branch. I'm guessing this would be something like 50-100GB of data per month and we could probably keep a years worth of data with no problem.