This PR introduces a new testing scheme for the repo. The idea is to use pytest as the main testing framework and express all existing tests in its terms.
:information_source: allure-generate and allure-open require you to have allure installed. You may download it here.
Linting
Use the linter task to run static checks against the entire codebase:
poe linter
Or against a single package:
cd allure-<framework>
poe linter
E.g., for allure-pytest this would be:
cd allure-pytest
poe linter
Motivation
We have some long-running issues with tests in this repository. To better understand them, lets first describe the current testing scheme.
Current state
Before the change, each framework integration package has had its own end-to-end tests with the following characteristics:
these tests are written using the target framework itself (i.e., tests on allure-behave are written using behave, tests on allure-robotframework - using robotframework, etc.)
the tests are contained within the package, in the tests folder (features in case of allure-behave).
Constraints
While this makes packages self-contained, the cons are quite notable:
testing code is hard to reuse
tests are more diverse and harder to maintain compared to if they were written using a single framework
some examples are hidden inside tests and thus are hard to follow
it's hard to run all tests before, e.g., committing to the repo.
Tests diversity and reusability issues
To test an allure integration with a framework you need a test runner to invoke the target framework and some code to assert the results.
Currently, we have integrations with five testing frameworks: behave, nose2, pytest, pytest-bdd and robot framework and, accordingly, behave tests, nose2 tests, pytest tests, pytest-bdd tests and robot framework tests. This results in five different test runner implementations, hence, duplicated and more complex test logic. Also, it's impossible to reuse, say, the code of behave tests in a test that uses robot framework and vice versa.
Additionally, tests folders are not included in the package distributions and thus cannot be imported from other modules. Unlike, say, in java, in python we can't conditionally make package subfolders resolvable from other places. The only option here is to move tests to a separate package.
There is no strong reason to have this diversity and distribution of tests across packages in the first place. It's much easier from maintainer's perspective to have all tests written using the same framework and contained in a single separate package to share as much code with each other as possible.
Poor examples
Actually, we don't have allure-behave examples at all. What we have in the allure-behave/features folder is a description on how to test allure-behave. The actual example of usage is just one part of it:
Feature: Label
Scenario: Scenario label
Given feature definition
"""
Feature: Step status
@allure.label.owner:me
Scenario: Scenario with passed step
Given simple passed step
"""
When I run behave with allure formatter
Then allure report has a scenario with name "Scenario with passed step"
And scenario has "owner" label with value "me"
Here the example is provided as a step description and the rest of the file is test info on how to check if this example is correct.
Some feature files doesn't include an example on how to use allure with behave at all. They only describe how exactly allure should support the framework and contain no allure API usage inside.
The same applies to allure-pytest-bdd's examples. Examples on allure-robotframework look better but lack of textual descriptions (mostly contain code blocks only).
Examples should be written as human readable textual documents with code blocks inside. Test descriptions should go separately.
Running tests
To test all packages, one had to do the following:
Install allure-python-commons and allure-python-commons-test packages.
Install other packages.
Install dependencies for package tests.
Change the current directory to the package's directory.
Run tests. Optionally, run the linter.
Repeat steps 4 and 5 for all other packages.
Ideally, this should be doable in two steps:
Install all dependencies
Run all tests
Incompatibility between allure-pytest and allure-pytest-bdd
There is a known issue (#109) that prevents allure-pytest and allure-pytest-bdd from running simultaneously in the same pytest session. That means, we cannot test either of plugins if we have both of them installed at the same time even if we use the -p pytest option to disable one of the plugins. That's because the -p option doesn't propagate to a nested pytest session.
Another option would be using PYTEST_DISABLE_PLUGIN_AUTOLOAD environment variable, but we can't use that because some tests check allure-pytest's support of 3rd party pytest plugins that needs to be loaded.
Ideally, this should be fixed in allure-pytest-bdd itself, and I have plans to do that. But for now, there should be a way to explicitly list the pytest plugins required by a test.
New testing scheme
All tests were rewritten using pytest. They were moved from allure-<framework> folders to tests/allure_<framework>. Therefore, the tests folder now contains all tests.
On the first level we have framework-specific test folders, a high-level conftest.py and the e2e module. Potentially, there could be tests on cross-package functionality on that level as well (i.e., compatibility tests).
tests/e2e.py
This module contains some functions and classes that come handy for end-to-end tests. Historically, those are almost the only tests we have (e.g., the repo originally contained no unit tests) and there are some common patterns used here allowing us to reuse lots of code. See docstrings on functions and classes themselves to better understand what they could be used for.
tests/conftest.py
This is a high level conftest.py of out pytest setup. Its job is to enable pytester and to declare commonly used fixtures: rst_examples and docstring.
Framework-specific test folders
A Framework-specific test folder mainly consists of tests on allure integration with the framework. The tests are grouped into nested folders depending on area they test, their granularity, purpose, etc. I used the following categories on the highest level of separation:
acceptance - for end-to-end tests, i.e., tests to ensure the allure fully and correctly supports a corresponding framework. Usually, those tests have doc examples or other form of human readable description associated with them. I further divided acceptance tests into framework support tests (on whether allure correctly translates framework constructs into allure ones) and allure API tests (whether allure API is accessible to a framework user and produces the expected results).
defects - for bug reproduction tests.
externals - for tests on allure compatibility with 3rd party packages (e.g., pytest plugins).
Framework-specific test runners
Each allure integration has its own test runner (except allure-pytest-bdd; it uses allure-pytest's one) and a fixture associated with it. Each runner inherits from a common base class and implements how exactly the framework should be executed. It also provides a simple way to specify test description (depending on the framework it could be a path to a file, a string or an ID of a code block in an example file or any combination of all these options).
Refer to test runner's docstrings on what capabilities does it provides.
Framework-specific conftest.py
These conftest.py files declare additional fixtures to use by tests. Typically, they are test runner fixtures.
Examples
Examples were rewritten as reStructuredTest (.rst) documents with code blocks inside. Each code block has an ID assigned to it. A test may refer to that ID to access the code block content and use it as an input for a framework run.
Here is a fragment of a document, describing how to assign an allure label to a behave test:
It's possible to add a custom label to a behave scenario. Simply apply :code:`@allure.label.<name>:<value>` tag to your scenario, e.g.:
.. code:: gherkin
:name: label-feature
Feature: Allure label for behave tests
@allure.label.author:John-Doe
Scenario: Scenario marked with an author label
Given noop
The document is human readable. Also, there is a test that checks whether the example is correct:
The test takes the feature file from the example using the code block ID and provides the step definition as an inline string.
Other changes
The PR also contains lots of minor changes.
Linting
Previously, the linting was performed on a per-package basis. It's not necessary, because the linting is fast (compared to building and testing). It performs only static checks, i.e., requires no deps to be installed hence no conflicts.
You can still lint on a per-package basis if you wish, but now there is more general way to lint the entire code base with single poe linter command.
The build allure python workflow was rewritten to use this way of linting instead of creating a full-blown package matrix and linting each package separately. Dependencies installation was removed (linting doesn't require dependencies).
Dependency management
Dependency management wasn't the main concern during this work. We definitely have more to do to ease project management but that will be later. The PR contains some improvements though.
Dependencies now could be installed using the following requirement files:
requirements/core.txt: dependencies required to manage the repository (currently, poethepoet only).
requirements/commont.txt: allure_commons and allure_commons_test in editable mode.
This was originally in the PR #728. It was moved into this PR to accommodate layout changes.
Closes #727
Obsoletes #728
Allure test result flushing
Allure test result closing was moved from the pytest_runtest_logfinish hook to pytest_runtest_protocol. This makes more sense because this is the last hook in the lifecycle (see the hookspec for more info on that). The only practical implication of this I can think of though is when a test fixture calls pytest.exit. The pytest_runtest_logfinish is not executed in that case, previously leaving the test result unreported.
User experience improvements
The PR contains the following changes that affect user experience:
[core]: The allure_commons.mapping.parse_tag function is now less strict: any characters except : and = are now allowed between the last period of a tag and its value separator. Previously, only word characters were allowed. This affects robotframework and behave tags.
[allure-pytest]: A test now is reported as skipped if one of its fixtures calls pytest.exit.
[allure-robotframework]: General allure tag syntax in form allure.<link-type>[.<link-name>]:<value> is now supported for link tags in a Robot Framework test case file (see examples).
[allure-robotframework]: An allure testplan entry now may omit either id or selector. Previously, both properties were mandatory.
[allure-pytest]: If --clear-alluredir is specified, all content is now removed from the allure directory. Previously, directories were not deleted. Typically, those were history directories copied from the previous allure report (closes #470).
Bug fixes
The following bugs or errors were fixed:
[core]: Invalid allure_commons.typing.LinkType.TEST_CASE enum value "testcase". It was changed to "tms" as expected by allure reporter (closes #448).
[allure-robotframework]: Allure links specified using robot framework tags in form <link type>:[<link text>]<URL> (e.g., link:[homepage]https://qameta.io) were parsed incorrectly if the link text contains characters from outer parts of the URL due to incorrect usage of str.strip.
Test fixes and optimizations
The PR contains the following fixes and improvements:
[allure-nose2, allure-pytest]: Unnecessary parametrization was removed from some tests.
[allure-behave, allure-pytest]: Unicode tests are obsolete and cover no logic now since python 2 support was dropped. These tests were removed.
[allure-pytest]: The following tests were fixed and got their skip marks removed:
[allure-pytest]: The tests/allure_pytest/acceptance/step/test_step_with_several_step_inside_thread.py::test_step_with_reused_threads test (original) used the time.sleep function to mix allure steps it creates. This led to a massive time overhead. The steps are now shuffled using threading.Event. The test execution time reduced by 30-40 times.
[allure-pytest]: Allure-results from doctest examples are now cached and could be reused by another test in a module. It reduces the test execution time by approximately 10%. Caching should be enabled explicitly on a per-test basis to prevent sophisticated test errors.
[allure-pytest]: The --log-cli-level pytest option was replaced with more common --log-level in log capture tests.
[allure-pytest]: The executed_docstring_source and executed_docstring_path fixtures were removed. The "act" test phase was moved inside test functions (a fixture should ideally takes care of the "arrange" phase only).
[allure-pytest]: Less then assertions were added to duration tests to make stop time testing more expressive.
[allure-pytest]: The tests/allure_pytest/acceptance.fixture.fixture_test.py::test_one_fixture_on_two_tests (originally) test was fixed: the second assertion is now properly executing.
[allure-pytest]: The test files step_parameters.py and custom_label.py were renamed to step_parameters_test.py and custom_label_test.py and now could be collected by pytest as intended.
[allure-pytest]: The duplicated parameter set was removed from tests/allure_pytest/acceptance/step/step_parameters_test.py::test_step_parameters (originally).
[allure-pytest]: The parameter set with the None testplan was removed from the tests/allure_pytest/acceptance/testplan/select_test_from_testplan_test.py::test_select_by_testcase_id_test (originally) test as redundant. Almost any other test is a None-testplan test.
[allure-pytest]: The tests/allure_pytest/acceptance/testplan/select_test_from_testplan_test.py::test_select_by_testcase_id_test (originally) test was rewritten to not depend upon pytest's basetemp folder. Previously, the basetemp folder had to be nested inside the allure-pytest folder.
[allure-pytest, allure-pytest-bdd]: Plugins are now not loaded automatically when a nested pytest session is created. All required plugins must be explicitly specified by a test in one of its fixtures.
[allure-robotframework]: Test results now contain no duplicated tags. Previously, all allure tags were added twice.
Other minor fixes
The PR contains the following small changes:
[allure-pytest]: The typo in duration_time_test.py was fixed.
[allure-pytest]: The allure_pytest.utils.escape_name function was removed as it does nothing after parameters were removed from a test full name (closes #280).
[allure-pytest]: Parameter set ids were assigned to some parametrized tests.
[allure-pytest]: The select_test_from_testplan_test.py and pytest_get_allure_plugin_test.py tests were moved to acceptance as they check the pytest support by allure.
[allure-pytest]: Replace exception-based logic with precondition checking logic in the allure_pytest.utils.allure_title function and in the allure_commons.logger.AllureFileLogger constructor.
Documentation fixes
The following changes were made in the documentation:
Invalid badge links in package's README files were fixed.
The separator between the listener and its argument was changed from the semicolon (;) to colon (:) in allure-robotframework's README as noted here (closes #427).
Links to alluure-behave, allure-pytest and allure-robotframework examples were added to the main README and packages' README files.
This PR introduces a new testing scheme for the repo. The idea is to use pytest as the main testing framework and express all existing tests in its terms.
Table of contents
Brief usage examples Package installation Testing Collect allure results Linting Motivation Current state Constraints Tests diversity and reusability issues Poor examples Running tests Incompatibility between allure-pytest and allure-pytest-bdd New testing scheme The
tests
folder structure Examples Other changes Linting Dependency management Dynamic parameter examples Allure test result flushing User experience improvements Bug fixes Test fixes and optimizations Other minor fixes Documentation fixesBrief usage examples
All commands are executed from the root of the repo.
Package installation
You may install all packages in editable mode or only one/some of them depending on your needs.
Install all packages
Install all allure-python packages in editable mode and their dependencies (including testing and linting dependencies) with this single command:
Install single package
The sequence remains the same as before:
Example: installing allure_pytest
Testing
Below are some examples on how to run tests.
Run all tests
With poethepoet:
With pytest directly:
Test single package
With poethepoet:
With pytest directly:
Example: test allure-pytest
or
Collect allure results
If you want to run tests with allure enabled, use the
allure-collect
task:Generate and view the report
Use the
allure-generate
task to create an allure report from the previously collected allure results:Use the
allure-open
task to serve the generated report:You can navigate to the report using a browser (if it didn't run automatically).
You can run all tests with allure, generate and open the report using this one-liner:
Linting
Use the
linter
task to run static checks against the entire codebase:Or against a single package:
E.g., for allure-pytest this would be:
Motivation
We have some long-running issues with tests in this repository. To better understand them, lets first describe the current testing scheme.
Current state
Before the change, each framework integration package has had its own end-to-end tests with the following characteristics:
tests
folder (features
in case of allure-behave).Constraints
While this makes packages self-contained, the cons are quite notable:
Tests diversity and reusability issues
To test an allure integration with a framework you need a test runner to invoke the target framework and some code to assert the results.
Currently, we have integrations with five testing frameworks: behave, nose2, pytest, pytest-bdd and robot framework and, accordingly, behave tests, nose2 tests, pytest tests, pytest-bdd tests and robot framework tests. This results in five different test runner implementations, hence, duplicated and more complex test logic. Also, it's impossible to reuse, say, the code of behave tests in a test that uses robot framework and vice versa.
Additionally,
tests
folders are not included in the package distributions and thus cannot be imported from other modules. Unlike, say, in java, in python we can't conditionally make package subfolders resolvable from other places. The only option here is to move tests to a separate package.There is no strong reason to have this diversity and distribution of tests across packages in the first place. It's much easier from maintainer's perspective to have all tests written using the same framework and contained in a single separate package to share as much code with each other as possible.
Poor examples
Actually, we don't have allure-behave examples at all. What we have in the
allure-behave/features
folder is a description on how to test allure-behave. The actual example of usage is just one part of it:Here the example is provided as a step description and the rest of the file is test info on how to check if this example is correct.
Some feature files doesn't include an example on how to use allure with behave at all. They only describe how exactly allure should support the framework and contain no allure API usage inside.
The same applies to allure-pytest-bdd's examples. Examples on
allure-robotframework
look better but lack of textual descriptions (mostly contain code blocks only).Examples should be written as human readable textual documents with code blocks inside. Test descriptions should go separately.
Running tests
To test all packages, one had to do the following:
allure-python-commons
andallure-python-commons-test
packages.Ideally, this should be doable in two steps:
Incompatibility between allure-pytest and allure-pytest-bdd
There is a known issue (#109) that prevents allure-pytest and allure-pytest-bdd from running simultaneously in the same pytest session. That means, we cannot test either of plugins if we have both of them installed at the same time even if we use the
-p
pytest option to disable one of the plugins. That's because the-p
option doesn't propagate to a nested pytest session. Another option would be usingPYTEST_DISABLE_PLUGIN_AUTOLOAD
environment variable, but we can't use that because some tests check allure-pytest's support of 3rd party pytest plugins that needs to be loaded.Ideally, this should be fixed in allure-pytest-bdd itself, and I have plans to do that. But for now, there should be a way to explicitly list the pytest plugins required by a test.
New testing scheme
All tests were rewritten using pytest. They were moved from
allure-<framework>
folders totests/allure_<framework>
. Therefore, thetests
folder now contains all tests.The
tests
folder structureThe
tests
folder structure now looks like this:The first level
On the first level we have framework-specific test folders, a high-level
conftest.py
and thee2e
module. Potentially, there could be tests on cross-package functionality on that level as well (i.e., compatibility tests).tests/e2e.py
This module contains some functions and classes that come handy for end-to-end tests. Historically, those are almost the only tests we have (e.g., the repo originally contained no unit tests) and there are some common patterns used here allowing us to reuse lots of code. See docstrings on functions and classes themselves to better understand what they could be used for.
tests/conftest.py
This is a high level
conftest.py
of out pytest setup. Its job is to enablepytester
and to declare commonly used fixtures:rst_examples
anddocstring
.Framework-specific test folders
A Framework-specific test folder mainly consists of tests on allure integration with the framework. The tests are grouped into nested folders depending on area they test, their granularity, purpose, etc. I used the following categories on the highest level of separation:
acceptance
- for end-to-end tests, i.e., tests to ensure the allure fully and correctly supports a corresponding framework. Usually, those tests have doc examples or other form of human readable description associated with them. I further divided acceptance tests into framework support tests (on whether allure correctly translates framework constructs into allure ones) and allure API tests (whether allure API is accessible to a framework user and produces the expected results).defects
- for bug reproduction tests.externals
- for tests on allure compatibility with 3rd party packages (e.g., pytest plugins).Framework-specific test runners
Each allure integration has its own test runner (except
allure-pytest-bdd
; it usesallure-pytest
's one) and a fixture associated with it. Each runner inherits from a common base class and implements how exactly the framework should be executed. It also provides a simple way to specify test description (depending on the framework it could be a path to a file, a string or an ID of a code block in an example file or any combination of all these options).Refer to test runner's docstrings on what capabilities does it provides.
Framework-specific conftest.py
These
conftest.py
files declare additional fixtures to use by tests. Typically, they are test runner fixtures.Examples
Examples were rewritten as reStructuredTest (.rst) documents with code blocks inside. Each code block has an ID assigned to it. A test may refer to that ID to access the code block content and use it as an input for a framework run.
Here is a fragment of a document, describing how to assign an allure label to a behave test:
The document is human readable. Also, there is a test that checks whether the example is correct:
The test takes the feature file from the example using the code block ID and provides the step definition as an inline string.
Other changes
The PR also contains lots of minor changes.
Linting
Previously, the linting was performed on a per-package basis. It's not necessary, because the linting is fast (compared to building and testing). It performs only static checks, i.e., requires no deps to be installed hence no conflicts.
You can still lint on a per-package basis if you wish, but now there is more general way to lint the entire code base with single
poe linter
command.The
build allure python
workflow was rewritten to use this way of linting instead of creating a full-blown package matrix and linting each package separately. Dependencies installation was removed (linting doesn't require dependencies).Dependency management
Dependency management wasn't the main concern during this work. We definitely have more to do to ease project management but that will be later. The PR contains some improvements though.
Dependencies now could be installed using the following requirement files:
allure_commons
andallure_commons_test
in editable mode.See examples above.
Dynamic parameter examples
This was originally in the PR #728. It was moved into this PR to accommodate layout changes. Closes #727 Obsoletes #728
Allure test result flushing
Allure test result closing was moved from the
pytest_runtest_logfinish
hook topytest_runtest_protocol
. This makes more sense because this is the last hook in the lifecycle (see the hookspec for more info on that). The only practical implication of this I can think of though is when a test fixture callspytest.exit
. Thepytest_runtest_logfinish
is not executed in that case, previously leaving the test result unreported.User experience improvements
The PR contains the following changes that affect user experience:
:
and=
are now allowed between the last period of a tag and its value separator. Previously, only word characters were allowed. This affects robotframework and behave tags.skipped
if one of its fixtures callspytest.exit
.allure.<link-type>[.<link-name>]:<value>
is now supported for link tags in a Robot Framework test case file (see examples).id
orselector
. Previously, both properties were mandatory.--clear-alluredir
is specified, all content is now removed from the allure directory. Previously, directories were not deleted. Typically, those were history directories copied from the previous allure report (closes #470).Bug fixes
The following bugs or errors were fixed:
allure_commons.typing.LinkType.TEST_CASE
enum value"testcase"
. It was changed to"tms"
as expected by allure reporter (closes #448).<link type>:[<link text>]<URL>
(e.g.,link:[homepage]https://qameta.io
) were parsed incorrectly if the link text contains characters from outer parts of the URL due to incorrect usage ofstr.strip
.Test fixes and optimizations
The PR contains the following fixes and improvements:
tests/allure_python/acceptance/duration/duration_time_test.py::test_duration[exit]
(original)tests/allure_python/acceptance/duration/duration_time_test.py::test_with_fixture_duration[exit]
(original)tests/allure_python/acceptance/duration/duration_time_test.py::test_with_fixture_finalizer_duration[exit]
(original)tests/allure_python/acceptance/labels/suite/default_suite_test.py::test_default_suite
(original)tests/allure_python/acceptance/labels/suite/default_suite_test.py::test_default_class_suite
(original)tests/allure_python/acceptance/step/step_placeholder_test.py::test_args_less_than_placeholders
(original)tests/allure_python/acceptance/parametrization/parametrization_test.py::test_parametrization_with_ids
(original)tests/allure_python/acceptance/parametrization/parametrization_test.py::test_parametrization_decorators_with_partial_ids
(original)tests/allure_pytest/acceptance/step/test_step_with_several_step_inside_thread.py::test_step_with_reused_threads
test (original) used thetime.sleep
function to mix allure steps it creates. This led to a massive time overhead. The steps are now shuffled usingthreading.Event
. The test execution time reduced by 30-40 times.--log-cli-level
pytest option was replaced with more common--log-level
in log capture tests.executed_docstring_source
andexecuted_docstring_path
fixtures were removed. The "act" test phase was moved inside test functions (a fixture should ideally takes care of the "arrange" phase only).tests/allure_pytest/acceptance.fixture.fixture_test.py::test_one_fixture_on_two_tests
(originally) test was fixed: the second assertion is now properly executing.step_parameters.py
andcustom_label.py
were renamed tostep_parameters_test.py
andcustom_label_test.py
and now could be collected by pytest as intended.tests/allure_pytest/acceptance/step/step_parameters_test.py::test_step_parameters
(originally).None
testplan was removed from thetests/allure_pytest/acceptance/testplan/select_test_from_testplan_test.py::test_select_by_testcase_id_test
(originally) test as redundant. Almost any other test is aNone
-testplan test.tests/allure_pytest/acceptance/testplan/select_test_from_testplan_test.py::test_select_by_testcase_id_test
(originally) test was rewritten to not depend upon pytest'sbasetemp
folder. Previously, thebasetemp
folder had to be nested inside the allure-pytest folder.Other minor fixes
The PR contains the following small changes:
duration_time_test.py
was fixed.allure_pytest.utils.escape_name
function was removed as it does nothing after parameters were removed from a test full name (closes #280).select_test_from_testplan_test.py
andpytest_get_allure_plugin_test.py
tests were moved to acceptance as they check the pytest support by allure.allure_pytest.utils.allure_title
function and in theallure_commons.logger.AllureFileLogger
constructor.Documentation fixes
The following changes were made in the documentation:
;
) to colon (:
) in allure-robotframework's README as noted here (closes #427).