Test suites support for pytest

[kanstantsin]

Hi!

When lots of thousands tests are run on teamcity, build log becomes huge, viewing build log with flat tests structure either freezes browser or is shown in parts. One option that solves the issue is splitting tests into test suites by test file names automatically. Currently teamcity-messages doesn't support test suites for pytest. I would like to add an option to support that but I see that it would be a problem for parallel test execution. Right now flowId parameter is passed to all started/finished/etc. messages to support parallel testing feature. As I can see in my test build configurations, test suites reporting doesn't work well with flowIds even when correct parent is passed to test messages. So, here is a suggestion for test suites support:

1. Add an option to pytest ini file or command line arguments to turn test suites generation on. If this flag is on, then test suites are generated automatically based on test file names. Also flowId in not passed to teamcity messages. If this flag is off or not set, everything works as before. I've prepared a PR for this one.
2. Document this option (and all other options) and write that currently this flag doesn't work with parallel tests execution.
3. Add buildProblem reporting if this flag is on and parallel tests execution is detected. Also available in upcoming PR.
4. Add flowId support to test suites reporting in build logs on teamcity side. When it works, there is an option to add flowId support in teamcity-messages code and support both parallel tests execution and test suites reporting.

Regarding test suites flow id support - here's what I tried and what I got. NoflowId:

@echo off

echo ##teamcity[blockOpened name='Info']
echo Second step message one
echo Second step message two
echo ##teamcity[blockClosed name='Info']

echo ##teamcity[blockOpened name='Tests']
for /l %%%%x in (1, 1, 100) do (
    echo ##teamcity[testStarted name='my.long.test.name.example.parameter1_parameter2_parameter3_sequnce_number_%%%%x' flowId='my.long.test.name.example.parameter1_parameter2_parameter3_sequnce_number_%%%%x']
    echo ##teamcity[testFinished name='my.long.test.name.example.parameter1_parameter2_parameter3_sequnce_number_%%%%x' flowId='my.long.test.name.example.parameter1_parameter2_parameter3_sequnce_number_%%%%x']
)
echo ##teamcity[blockClosed name='Tests']

Produces expected structure.

Flow ids with parent:

@echo off

echo ##teamcity[blockOpened name='Info']
echo Second step message one
echo Second step message two
echo ##teamcity[blockClosed name='Info']

echo ##teamcity[blockOpened name='Tests' flowId='Tests']
for /l %%%%x in (1, 1, 10) do (
    echo ##teamcity[testSuiteStarted name='my.long.test_suite.name.example.sequnce_number_%%%%x' flowId='my.long.test_suite.name.example.sequnce_number_%%%%x' parent='Tests']
    for /l %%%%m in (1, 1, 10) do (
        echo ##teamcity[testStarted name='my.long.test.name.example.parameter1_parameter2_parameter3_sequnce_number_%%%%x_%%%%m' flowId='my.long.test.name.example.parameter1_parameter2_parameter3_sequnce_number_%%%%x_%%%%m' parent='my.long.test_suite.name.example.sequnce_number_%%%%x']
        echo ##teamcity[testFinished name='my.long.test.name.example.parameter1_parameter2_parameter3_sequnce_number_%%%%x_%%%%m' flowId='my.long.test.name.example.parameter1_parameter2_parameter3_sequnce_number_%%%%x_%%%%m' parent='my.long.test_suite.name.example.sequnce_number_%%%%x']
    )
    echo ##teamcity[testSuiteFinished name='my.long.test_suite.name.example.sequnce_number_%%%%x' flowId='my.long.test_suite.name.example.sequnce_number_%%%%x' parent='Tests']
)
echo ##teamcity[blockClosed name='Tests' flowId='Tests']

Produces following structure:

Test suites are not nested in parent block, tests are not nested in suite blocks, all subtrees are empty.

Environment:

• teamcity-messages 1.29
• TeamCity Enterprise 2021.1 (build 92597)
• pytest 3.2.5

[kanstantsin]

Found a mistake in generating messages with parent flows: forgot to report flowStarted and flowFinished. Now structure seems ok:

@echo off

echo ##teamcity[blockOpened name='Info']
echo Second step message one
echo Second step message two
echo ##teamcity[blockClosed name='Info']

echo ##teamcity[blockOpened name='Tests' flowId='Tests']
for /l %%%%x in (1, 1, 10) do (
    echo ##teamcity[flowStarted flowId='my.long.test_suite.name.example.sequence_number_%%%%x' parent='Tests']
    echo ##teamcity[testSuiteStarted name='my.long.test_suite.name.example.sequence_number_%%%%x' flowId='my.long.test_suite.name.example.sequence_number_%%%%x' parent='Tests']
    for /l %%%%m in (1, 1, 10) do (
        echo ##teamcity[flowStarted flowId='my.long.test.name.example.parameter1_parameter2_parameter3_sequence_number_%%%%x_%%%%m' parent='my.long.test_suite.name.example.sequence_number_%%%%x']
        echo ##teamcity[testStarted name='my.long.test.name.example.parameter1_parameter2_parameter3_sequence_number_%%%%x_%%%%m' flowId='my.long.test.name.example.parameter1_parameter2_parameter3_sequence_number_%%%%x_%%%%m' parent='my.long.test_suite.name.example.sequence_number_%%%%x']
        echo ##teamcity[testFinished name='my.long.test.name.example.parameter1_parameter2_parameter3_sequence_number_%%%%x_%%%%m' flowId='my.long.test.name.example.parameter1_parameter2_parameter3_sequence_number_%%%%x_%%%%m' parent='my.long.test_suite.name.example.sequence_number_%%%%x']
        echo ##teamcity[flowFinished flowId='my.long.test.name.example.parameter1_parameter2_parameter3_sequence_number_%%%%x_%%%%m' parent='my.long.test_suite.name.example.sequence_number_%%%%x']
    )
    echo ##teamcity[testSuiteFinished name='my.long.test_suite.name.example.sequence_number_%%%%x' flowId='my.long.test_suite.name.example.sequence_number_%%%%x' parent='Tests']
    echo ##teamcity[flowFinished flowId='my.long.test_suite.name.example.sequence_number_%%%%x' parent='Tests']
)
echo ##teamcity[blockClosed name='Tests' flowId='Tests']

But even so, I would better start with simple test suites support and then try to move to flows and parallel tests support in future PRs. Parallel tests support in test suites reporting would require some more work in suite finish detection.