search

gurock / trcli

TR CLI (trcli) is a command line tool for interacting with TestRail.
Mozilla Public License 2.0
48 stars 39 forks source link

Unable to update test results generated in junit format from pytest in CircleCI #231

Open gjambaisivanandham opened 1 month ago

gjambaisivanandham commented 1 month ago

TestRail CLI Version

v1.9.5

CLI Environment

python:3.10.5

TestRail Version

8.0.3 Default (3070)

TestRail Instance Type

Enterprise Cloud

Current behavior

Unable to update testrail with the test result from pytest that is run in parallel in CircleCI.

Run command in CircleCI is circleci tests glob "e2e/tests/**/*.py" | circleci tests split --split-by=timings | xargs pytest --log-cli-level=INFO --junitxml "reports/junit-report-${CIRCLE_NODE_INDEX}.xml". This generates multiple junit reports(junit-report-0.xml, junit-report-1.xml etc) depending on the number of parallel runs but processing of the results fails.

Step used to process the test results in CircleCI is below 

    docker:
      - image: cimg/python:3.11
    steps:
      - attach_workspace:
          at: <<workspace-path>>
      - run:
          name: Install TR CLI and upload the Python test results to TestRail
          command: |
            pip install trcli
            trcli -y \
            -h $TESTRAIL_INSTANCE_URL \
            --project "Project name" \
            -u $TESTRAIL_USERNAME \
            -p $TESTRAIL_PASSWORD \
            parse_junit \
            --title "Automated Python Tests from CircleCI Workflow" \
            --run-description $CIRCLE_BUILD_URL \
            -f "./reports/junit*.xml"

Below is the log when the above job is run in a workflow:


Copyright 2024 Gurock Software GmbH - www.gurock.com
Parser Results Execution Parameters
> Report file: /home/circleci/project/reports/*.xml
> Config file: None
> TestRail instance: **************************** (user: *******************************)
> Project: API E2E Tests
> Run title: Automated Python Tests from CircleCI Workflow
> Update run: No
> Add to milestone: No
> Auto-create entities: True
Parsing JUnit report.
Processed 48 test cases in section pytest.
Processed 29 test cases in section pytest.
Processed 90 test cases in section pytest.
Processed 61 test cases in section pytest.
Checking project. Done.
Found 112 test cases not matching any TestRail case.
Adding missing test cases to the suite.
  0% 0/124 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/circleci/.pyenv/versions/3.11.9/bin/trcli", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/circleci/.local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/trcli/cli.py", line 242, in main
    return super().main(windows_expand_args=False, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/circleci/.local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/trcli/commands/results_parser_helpers.py", line 101, in wrapper_common_options
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.local/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.local/lib/python3.11/site-packages/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/trcli/commands/cmd_parse_junit.py", line 36, in cli
    result_uploader.upload_results()
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/trcli/api/results_uploader.py", line 90, in upload_results
    added_test_cases, result_code = self.add_missing_test_cases()
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/trcli/api/results_uploader.py", line 318, in add_missing_test_cases
    added_cases, result_code = self.prompt_user_and_add_items(
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/trcli/api/results_uploader.py", line 342, in prompt_user_and_add_items
    added_items, error_message = add_function()
                                 ^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/trcli/api/api_request_handler.py", line 353, in add_cases
    responses, error_message = self.handle_futures(
                               ^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/trcli/api/api_request_handler.py", line 517, in handle_futures
    response = future.result()
               ^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/trcli/api/api_request_handler.py", line 619, in _add_case_and_update_data
    response = self.client.send_post(f"add_case/{case_body.pop('section_id')}", case_body)
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'section_id'

Exited with code exit status 1```

### Desired behavior

Process the results and update the tests

### More Details

_No response_
tyatposit commented 1 month ago

We hit this same error when uploading our junit xml results.

At least for us, the issue appears to be how the xml we want to upload is nesting several child testsuite sections under a parent testsuites section. It looks like trcli only accepts uploads where all the tests are nested under a single testsuite section and blows up with the same error outlined in the ticket if there are multiple.

I confirmed the same test suite uploads/reports with no issues after modifying the structure of our xml so all the tests are under a single testsuite section.

Our temporary work around for this issue is to use a XSLT stylesheet and reformat the merged xml so all tests are in a single testsuite section with xsltproc before uploading it with trcli.