Open pli888 opened 5 years ago
Hi @pli888,
yes, one of the usage for tags is to run an arbitrary subset of the suite. I do that using a syntax like this:
docker-compose run --rm test bin/behat -v --stop-on-failure --tags "@issue-49&&@ok"
which run scenarios that are tagged with @issue-49 AND @ok
But my most used command is:
docker-compose run --rm test bin/behat -vv --stop-on-failure --tags @wip
where @wip is ever-changing work-in-progress tag I put on the only one specific scenario I want to test at any given moment.
I replace @wip with @ok when I got the test to pass. It helps work on feature files with many scenarios.
If you look at the existing tags on scenarios header in the repository, you'll see I also use area or functionality based tags like you wish to do (e.g: @dataset-view, @login, @author-names-display).
If the command is too long to run repeatedly, you can first shell into test container and then run the command:
$ docker-compose run --rm test bash
# bin/behat --tags @author-names-display
The behat command offers many ways to help the developer write, run and debug tests that a not thoughfully designed wrapper script would hide and we don't want to make the wrapper script too complex either.
The wrapper script was created mainly to cover a limited number of canned scenarios like CI pipeline, javascript-dependent tests and running everything that's implemented and seen passing once (that's what my @ok means) before going for lunch.
However, if we can identify some common often-used non trivial scenarios or running mode, we could consider refactoring the wrapper to run them easily, while using the behat command for day to day development need.
I'm already seeing the need for refactoring as for the File Upload Wizard, I've switched to Codeception (industry standard, modern and maintained) for running tests, so the wrapper script will need to be changed to take that into account.
The only snag at the moment for directly using the behat command as suggested above, is that currently in the develop branch, the wrapper script is doing too much: it does environment selection, back up database beforehand and restore database after test run.
This means, you will need to do those database management manually as the situations requiring it arise (not needed if scenario doesn't alter the database or if you don't care much about the state of your database).
This has been fixed in my development branches and the latest PR I have submitted in which the DB backup/restore is done in before/after test code hooks and the environment selection is an argument passed to behat (as the behat configs now contain environment configuration).
In summary:
--profile local
to the command)
@rija When developing new acceptance tests, there is a need to check that they run successfully in Behat. To run acceptance tests, you need to execute
docker-compose run --rm test ./tests/acceptance
but this runs all tests tagged with@ok
which takes a long time when all I want to do is run the test(s) under developmentWe could enable the
acceptance
bash script to accept tags provided as command line arguments which are then passed to thebin/behat
call so that specific tests are executed as follows: