Add --tr-testrun-refs option

[mickyJNST]

Is your feature request related to a problem? Please describe. TestRail allows for integration with Jira via the refs parameter in test runs. pytest-testrail doesn't currently support setting the refs param (which is just a comma delimited string).

Describe the solution you'd like Add --tr-testrun-refs option to existing list of pytest-testrail options

Describe alternatives you've considered [Third edit:-)] To work around this limitation, we tried creating the test run (including the refs parameter) in TestRail directly via the TestRail API (before invoking our pytest-based tests), and then writing the run_id to the testrail.cfg file using the test run id we get from the TestRail API. Our testrail.cfg file looks like this:

[API]
url = xxx
email = xxx
password = xxx

[TESTRUN]
run_id = 999

However, this workaround didn't work. We get the following error at the end of running the pytest suite:

============================= test session starts ==============================
collecting ... [testrail] Failed to create testrun: "Field :project is not a valid ID."

Also, adding the project_id resolves the error, but then results in a new run being created instead of the specific run being used:

============================= test session starts ==============================
collecting ... [testrail] New testrun created with name "Automated Run 06-02-2022 21:57:53" and ID=449

Furthermore, even if the above approach did work, I think we would run into another issue. I think pytest-testrail assumes the test run already has test cases set (with status "Untested") when publishing results. However, using above approach means we are creating an empty test run (because we are trying to avoid creating a mechanism to determine what all test case ids are in our test suite). If this assumption is true then publishing the results would fail as the tests wouldn't yet exist in the test run in TestRail. To handle this use case pytest-testrail could update the test run with known test cases at the start of the publishing process if the test run is empty (via update_run endpoint).

[voloxastik]

Hi @mickyJNST Could you explain more about your request? Do you mean to create a new test run with test cases that contain specified refs from option --tr-testrun-refs?

[voloxastik]

As a workaround, you can create necessary test run directly and then run pytest with option --tr-run-id XXXXX

[mickyJNST]

As a workaround, you can create necessary test run directly and then run pytest with option --tr-run-id XXXXX

Yeah - tried this but ran into other issues (as described in the Describe alternatives you've considered section of original post)

[voloxastik]

As a workaround, you can create necessary test run directly and then run pytest with option --tr-run-id XXXXX

Yeah - tried this but ran into other issues (as described in the Describe alternatives you've considered section of original post)

Your testrail.cfg should look like this:

[API]
url = xxx
email = xxx
password = xxx
[TESTRUN]
assignedto_id = 111
project_id = 83

Then you can run your tests using following command: pytest /tests/my_tests.py --testrail --tr-run-id=XXXXXX where XXXXXX is the ID of your test run created earlier

[mickyJNST]

Thanks for the explanation @voloxastik. Unfortunately your proposed solution doesn't fit with how we have our workflow set up as we're trying to do everything via the testrail.cfg file...and we don't want to have to do a combination of file and pytest command line options. In any case, we have a very roundabout workaround in place that will do for now, but would like to see support for the test run refs option added (for both the config file and command line).

[voloxastik]

@mickyJNST Could you explain more about your request? Do you mean to create a new test run with test cases that contain specified refs from option --tr-testrun-refs?

[mickyJNST]

@voloxastik I want to associate the test run with Jira tickets. This is supported by the TestRail API via the refs parameter. Hopefully this clarifies what I am getting at:

[voloxastik]

@mickyJNST FYI. The plugin does not use information about testrun_id from the config file run_id=config.getoption('--tr-run-id') only from run command but it's easy to fix I will do a PR for it. But this is a bad way as it is a dynamic variable