mirage / alcotest

A lightweight and colourful test framework
ISC License
454 stars 80 forks source link

Basic CI integration #364

Closed MisterDA closed 2 years ago

MisterDA commented 2 years ago

This PR attempts to do simple integration from Alcotest to CI systems. For now, Alcotest detects the CI system using the CI, GITHUB_ACTIONS, or OCAMLCI environment variables. As a first showcase of what we can do, when detecting GitHub Actions it will output annotations allowing to fold/unfold a test suite in the web interface. When detecting it's running in CI, Alcotest will also enable the show_errors modes (this can be overridden via the API) so that all failing tests are displayed in the CI (otherwise users would have to fix the failing test, push, and so on, to see the other errors). I plan to expand the features if this PR is well received.

samoht commented 2 years ago

I think you forgot to promote the expect test outputs - might be worth being explicit if these environment variables are set in the tests (so we can also check these are working as expected).

MisterDA commented 2 years ago

I think you forgot to promote the expect test outputs - might be worth being explicit if these environment variables are set in the tests (so we can also check these are working as expected).

I've set CI=false in the environment of the tests. Testing the CI will likely need to duplicate the tests. As the features are quite simple for now, maybe that can be done later?

MisterDA commented 2 years ago

For reference, this is the language GitHub Actions uses for annotations: Workflow commands for GitHub Actions. Here's an open feature request of mine: Grouping log lines: set the folded/unfolded state with ::endgroup:: at the end of the logs.

samoht commented 2 years ago

Many thanks!