search

camunda / camunda

Process Orchestration Framework
https://camunda.com/platform/
3.35k stars 612 forks source link

GHA: Collect test statistics #9132

Open lenaschoenburg opened 2 years ago

lenaschoenburg commented 2 years ago

Jenkins collects statistical information about our tests:

The flaky test statistics are collected in BigTable while the total number of tests and the timings are stored by Jenkins directly. Maybe we can find a solution where we just analyze the test reports once and extract all necessary information from there. It's not yet clear to me where we would display or store the results

@npepinpe brought up that the total number of tests is something that we should make very visible since we don't want to accidentally skip running tests (as has happened in the past).

blocked by https://jira.camunda.com/browse/INFRA-3363

npepinpe commented 2 years ago

Some options:

All of which should be free for open source projects, I think.

npepinpe commented 2 years ago

Seems like DataDog also offers this, but DataDog in my experience is :money_mouth_face: (though it's also very, very, very good): https://www.datadoghq.com/blog/datadog-ci-visibility/

lenaschoenburg commented 2 years ago

That does sound like exactly what we'd need. Apparently it starts at 20$ per committer per month: https://www.datadoghq.com/pricing/?product=ci-visibility#ci-visibility

npepinpe commented 2 years ago

Yeah, for an open source project this may be less than ideal :smile: DataDog is very good, just always very pricey

npepinpe commented 2 years ago

In the end, we can keep doing what we're doing, and simply send the statistics to BigQuery - but instead of only flaky tests, we send more general statistics. Grafana 8.x has official support for BigQuery, so we could import data in there for our dashboards without having to learn a new tool to create dashboards. I guess that would be one option if none of the others work for us. It's more complex than I'd like though :disappointed:

npepinpe commented 2 years ago

Or let's keep it simple, and let's just send our statistics as we do now. Maybe infra can help us adapt how Jenkins connects to BigQuery to GHA. We can add some basic statistics later (# of tests) once that works.

npepinpe commented 2 years ago

Also, supposedly, SonarCloud should be able to at least count the number of tests. So we have our basic stats imo

npepinpe commented 2 years ago

I think SonarCloud could do this. See https://docs.sonarcloud.io/enriching/test-coverage/test-execution-parameters/ - I don't think we need to set a target, but just reporting it might be good enough for now.

Zelldon commented 2 years ago

Out of interest what is the usage benefit of this? Isn't something like this https://github.com/camunda/zeebe/pull/9450#issuecomment-1137447651 enough for now?

npepinpe commented 2 years ago

What's missing now is the build status/flaky test data collection which feeds our flaky test dashboard. I make use of it to track where to focus efforts, so I would like to have this at least.

Zelldon commented 2 years ago

Thanks. Do we want to use the same infrastructure ? Like showing the flaky tests? Do you have a resource/link to where and how it is currently done in jenkins? So we have it easily accessible from this issue?

npepinpe commented 2 years ago

For now I would propose we stick with the dashboard. So it's just about feeding the data to our BigQuery instance in GCP. It's done via a helper function provided by the infra team.

Here's the code in our own pipeline:

// we track each flaky test as a separate result so we can count how "flaky" a test is
if (flakyTestCases) {
  for (flakyTestCase in flakyTestCases) {
    org.camunda.helper.CIAnalytics.trackBuildStatus(this, 'flaky-tests', flakyTestCase)
  }
} else {
  org.camunda.helper.CIAnalytics.trackBuildStatus(this, currentBuild.result)
}

flakyTestCases contains a list of all the flaky tests (class name and test name).

The helper function is in https://github.com/camunda/jenkins-global-shared-library/blob/master/src/org/camunda/helper/CIAnalytics.groovy

:crossed_fingers: infra could help us there, but I assume they will be busy with their own stuff.

cmur2 commented 2 years ago

The function is a frontend to https://github.com/camunda/infra-ci-analytics-proxy (to avoid writing all the logic and authentication in Jenkins where no suitable GoogleCloud library existed).

More details should be on https://confluence.camunda.com/display/HAN/CI+Analytics

Conceptually, the same approach should be possible on Github Actions. On GHA you can also use existing frameworks more easily in e.g. Docker images.

menski commented 2 years ago

Moved to blocked as we need support by infra

menski commented 2 years ago

Infra ticket: https://jira.camunda.com/browse/INFRA-3363

cmur2 commented 2 years ago

FYI: Your input is needd on INFRA-3363, thanks!

npepinpe commented 2 years ago

See: