kbsali / php-redmine-api

A simple PHP Redmine API client, Object Oriented
MIT License
420 stars 183 forks source link

Improve BDD with new testing command #420

Closed Art4 closed 4 months ago

Art4 commented 4 months ago

Using Behat for BDD and BDT is really impressive and useful. For new features I create a temporary tag @wip and run docker compose exec php composer behat -- --tags=wip. To test the feature on a specific redmine version I run docker compose exec php composer behat -- --tags=issue,group --suite=redmine=50103. If the feature is ready I run all behat tests on all redmine versions using docker compose exec php composer behat.

However the tests are becoming bigger and that leads to 1.) higher usage of time and memory to execute all tests at once. And 2.) it is not possible to change the format of the output using CLI (because the --output=progress is already declared in composer.json).

This PR solves these problems by creating a new bdt composer command for tests grouped by Redmine versions (aka behat suites). The output there is always formatted as progress. If one test fails it is now easier to find the Redmine version, where the test has failed thanks to the separation by behat suite.

Test-run: docker compose exec php composer bdt ```bash $ docker compose exec php composer bdt > behat --config tests/Behat/behat.yml '--format=progress' '--suite=redmine_50103' ...................................................................... 70 ...................................................................... 140 ...................................................................... 210 ...................................................................... 280 ...................................................................... 350 ...................................................................... 420 ...................................................................... 490 ...................................................................... 560 ...................................................................... 630 ...................................................................... 700 ...................................................................... 770 ...................................................................... 840 .............................................. 89 scenarios (89 passed) 886 steps (886 passed) 1m56.30s (15.69Mb) > behat --config tests/Behat/behat.yml '--format=progress' '--suite=redmine_50009' ...................................................................... 70 ...................................................................... 140 ...................................................................... 210 ...................................................................... 280 ...................................................................... 350 ...................................................................... 420 ...................................................................... 490 ...................................................................... 560 ...................................................................... 630 ...................................................................... 700 ...................................................................... 770 ...................................................................... 840 .............................................. 89 scenarios (89 passed) 886 steps (886 passed) 2m9.03s (15.66Mb) > behat --config tests/Behat/behat.yml '--format=progress' '--suite=redmine_40210' ...................................................................... 70 ...................................................................... 140 ...................................................................... 210 ...................................................................... 280 ...................................................................... 350 ...................................................................... 420 ...................................................................... 490 ...................................................................... 560 ...................................................................... 630 ...................................................................... 700 ...................................................................... 770 ...................................................................... 840 .. 84 scenarios (84 passed) 842 steps (842 passed) 1m59.39s (15.57Mb) ```

The behat command will be changed to have no output format (defaults to --format=pretty). So if I run docker compose exec php composer behat -- --tags=wip I will get the pretty format that helps me with debugging. If I want to have the progress output I can declare it with docker compose exec php composer behat -- --tags=wip --format=progress.

This PR also improves the test docs explaining the different commands. I've tested theses new commands while working on #419.

codecov[bot] commented 4 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 98.54%. Comparing base (dfa6b49) to head (e502824).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## v2.x #420 +/- ## ========================================= Coverage 98.54% 98.54% Complexity 639 639 ========================================= Files 29 29 Lines 1925 1925 ========================================= Hits 1897 1897 Misses 28 28 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.