search

gigascience / gigadb-website

Source code for running GigaDB
http://gigadb.org
GNU General Public License v3.0
9 stars 15 forks source link

Running specific Behat acceptance tests #356

Open pli888 opened 5 years ago

pli888 commented 5 years ago

@rija When developing new acceptance tests, there is a need to check that they run successfully in Behat. To run acceptance tests, you need to execute docker-compose run --rm test ./tests/acceptance but this runs all tests tagged with @ok which takes a long time when all I want to do is run the test(s) under development

We could enable the acceptance bash script to accept tags provided as command line arguments which are then passed to the bin/behat call so that specific tests are executed as follows:

$ docker-compose run --rm test ./tests/acceptance @admin_page
rija commented 4 years ago

Hi @pli888,

yes, one of the usage for tags is to run an arbitrary subset of the suite. I do that using a syntax like this:

docker-compose run --rm test bin/behat -v --stop-on-failure --tags "@issue-49&&@ok"

which run scenarios that are tagged with @issue-49 AND @ok

But my most used command is:

docker-compose run --rm test bin/behat -vv --stop-on-failure --tags @wip

where @wip is ever-changing work-in-progress tag I put on the only one specific scenario I want to test at any given moment.

I replace @wip with @ok when I got the test to pass. It helps work on feature files with many scenarios.

If you look at the existing tags on scenarios header in the repository, you'll see I also use area or functionality based tags like you wish to do (e.g: @dataset-view, @login, @author-names-display).

If the command is too long to run repeatedly, you can first shell into test container and then run the command:

$ docker-compose run --rm test bash
# bin/behat --tags @author-names-display

The behat command offers many ways to help the developer write, run and debug tests that a not thoughfully designed wrapper script would hide and we don't want to make the wrapper script too complex either.

The wrapper script was created mainly to cover a limited number of canned scenarios like CI pipeline, javascript-dependent tests and running everything that's implemented and seen passing once (that's what my @ok means) before going for lunch.

However, if we can identify some common often-used non trivial scenarios or running mode, we could consider refactoring the wrapper to run them easily, while using the behat command for day to day development need.

I'm already seeing the need for refactoring as for the File Upload Wizard, I've switched to Codeception (industry standard, modern and maintained) for running tests, so the wrapper script will need to be changed to take that into account.

The only snag at the moment for directly using the behat command as suggested above, is that currently in the develop branch, the wrapper script is doing too much: it does environment selection, back up database beforehand and restore database after test run.

This means, you will need to do those database management manually as the situations requiring it arise (not needed if scenario doesn't alter the database or if you don't care much about the state of your database).

This has been fixed in my development branches and the latest PR I have submitted in which the DB backup/restore is done in before/after test code hooks and the environment selection is an argument passed to behat (as the behat configs now contain environment configuration).

In summary: