search

allankp / pytest-testrail

pytest plugin for integration with TestRail, for creating testruns and updating results
MIT License
96 stars 124 forks source link

Support xdist plugin to run test in parallel #89

Closed YakovLittle closed 5 years ago

YakovLittle commented 5 years ago

Currently pytest-testrail plugin creates Test-Run for each xdist workers and then publishes test cases like Untested

We could modify pytest_collection_modifyitems() to postpone call of self.create_test_run() and run it in pytest_sessionfinish() if self.results

astroVale commented 5 years ago

Hi, I'm having the same issue. pytest-testrail plugin creates Test-Run for each xdist workers. I see the corresponding pull request is closed but this issue is still open.

Could you please tell me when this will be implemented, or if you have a workaround I could use? Thank you!

YakovLittle commented 5 years ago

Hi @astroVale pls see https://github.com/allankp/pytest-testrail/pull/90#issuecomment-469741084

astroVale commented 5 years ago

Thanks for your reply! I was trying to understand that comment before writing here. From your comment: Step 1. My base conftest.py is modified in order not to add the testcase id to each single test function, but to take it from a list:

with open("all_tests.json") as json_file:
    data = json.load(json_file)
    tests = [[test["testcase_id"], test["testcase_name"]] for test in data]

def pytest_collection_modifyitems(session, config, items):
    for item in items:
        for testid, testname in tests:
            if item.name == testname:
                print(testid)
                item.add_marker(pytestrail.case(testid))

So the suggested code is not working in my case.

Step 2. Writing that class to a new file, from where is that class called in the end? Step 3, 4 and 5. Isn't the pytest_testrail plugin supposed to run those steps automatically already?

YakovLittle commented 5 years ago

pytest_testrail plugin works within xdist worker that means you have to run "testrail import" after pytest

astroVale commented 5 years ago

@YakovLittle, I see the fix you did in #90 was not merged to master branch, that's probably why for me pytest-testrail plugin still creates a test run for each xdist worker.

kobenguyent commented 5 years ago

Hi, any updates on this? I'm getting the same issue.

kobenguyent commented 5 years ago

@YakovLittle could you help update this issue?

YakovLittle commented 5 years ago

@PeterNgTr xdist-plugin is using separate processes (workers) and also merge test reports by rsync locally that means we couldn't use pytest-testrail plugin for import test report inside each process. I suggest to run the import into TestRail by separate step in your CI.

kobenguyent commented 5 years ago

@YakovLittle thanks for your reply. Could you maybe expose more details step how to do this? I'm sorry, I'm just getting to use this.

YakovLittle commented 5 years ago

@PeterNgTr pls see https://github.com/allankp/pytest-testrail/pull/90#issuecomment-469741084

kobenguyent commented 5 years ago

https://github.com/pytest-dev/pytest-html/blob/master/pytest_html/plugin.py pytest_html can make it work with xdist. Could we do that? :)

YakovLittle commented 5 years ago

yep, sure - we're using pytest-html too

kobenguyent commented 5 years ago

@YakovLittle thanks! can anyone help providing me an example code base how to make the testrail publishment with xdist? I'm in the middle of nowhere. Not sure what to do. Many thanks

YakovLittle commented 5 years ago

@PeterNgTr I've pushed source code example into https://github.com/allankp/pytest-testrail/pull/90#issuecomment-469741084

kobenguyent commented 5 years ago

@YakovLittle thanks, I tried to read it but sorry I'm noobie. Could you help format it to make it easier to read. Step 2. Writing that class to a new file, from where is that class called in the end? Step 3, 4 and 5. Isn't the pytest_testrail plugin supposed to run those steps automatically already?

kobenguyent commented 5 years ago

@allankilpatrick would you mind considering fixing this in the core?

kobenguyent commented 5 years ago

@astroVale I would manage to publish the test results to testrail when running tests with xdist without creating multiple test runs. Thanks @YakovLittle! I followed your path to parse the test results from junitxml report. I'm happy to help if you need me!

allankp commented 5 years ago

Appears this issue has been resolved in another comment. https://github.com/allankp/pytest-testrail/pull/90#issuecomment-469741084