Our unit and integration tests run automatically using GitLab-CI on self-hosted runners on Gitlab.com via pull/push mirror or a dedicated bot. With this issue, we want to evaluate this setup and propose future updates.
Pros
gitlab-ci allows dynamic generation of job scrips, which enable convenient implementations of the integration test generation
easy inclusion of GPU tests (because we have runners with the needed hardware)
one test pipeline, which includes unit and integration tests, which gives a nice overview of the status of all tests in one place.
Cons
switching to another platform (here Gitlab.com) and limited hardware creates an overhead for all test triggers. This is especially critical for unit tests because they should provide the test information rapidly.
Convenient secondary services (e.g. Codecov, CoverageAll) are not possible or cumbersome to use. Especially the post of Codecov-reports directly into the pull-request increases developer ergonomics a lot.
Proposed solution
We run the unit and integration tests for every PR using GitHub actions and use GitLab-CI only for GPU tests (which by themselves have unit and integration tests). This would most probably decrease the time for the first test report and allow the native usage of Codecov and CoverageAll (both have native support for GitHub actions and GitHub itself).
Since the services in mind only use the unit tests, it is also conceivable to just move the unit tests to GitHub actions and leave the integration tests in the GitLab-CI pipeline.
Problem statement
Our unit and integration tests run automatically using GitLab-CI on self-hosted runners on Gitlab.com via pull/push mirror or a dedicated bot. With this issue, we want to evaluate this setup and propose future updates.
Pros
Cons
Proposed solution
We run the unit and integration tests for every PR using GitHub actions and use GitLab-CI only for GPU tests (which by themselves have unit and integration tests). This would most probably decrease the time for the first test report and allow the native usage of Codecov and CoverageAll (both have native support for GitHub actions and GitHub itself).
Since the services in mind only use the unit tests, it is also conceivable to just move the unit tests to GitHub actions and leave the integration tests in the GitLab-CI pipeline.