Upload JUnit reports generated by Codeception - Cases not found

piotroxp commented 11 months ago

What would you like the TestRail CLI to be able to do?

Utilize the fact our test cases have TestRail Case ID's in the name (testcase level). Allow trcli to create a run based solely on the test case ids from the automation report

Our requirement is to create a Test Run that will: 0) Process a JUnit file like this (report from automation run):

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="Tests.Api" tests="2" assertions="2" errors="0" failures="2" skipped="0" useless="0" time="177.707585">
    <testcase name="testcase_C101" class="testCase" file="/app/tests/testCase.php" feature="test case C101" time="2.380936" assertions="2"/>
    <testcase name="testcase_C102" class="testCase" file="/app/tests/testCase.php" feature="test case C102" time="0.787607" assertions="0">
      <error>Useless Test</error>
    </testcase>
  </testsuite>
</testsuites>

1) Go through ALL cases available in the used TR Project (when passing the "--project [proj_name]" option) 2) Upload the JUnit automation report of the above structure with creating a run that will contain valid cases referenced

Ideal scenario: 1) We have a Project 2) The project has suites 3) Case IDs not in the project are ignored 4) Case IDs in the project become part of the uploaded run 5) The TR run result contains the existing cases from the automation report

Inb4: our test structure is not reflected in the code and we cannot revamp thousands of test cases to use the "automation_id" field. The whole reason we need this feature is to bind the case id from testrail to an automation test case and get the suites in order.

Why is this feature necessary on the TestRail CLI?

Making its possible to easily bind automation cases with testrail cases (based on id in test case name in the code)

More details

Currently, when using the following cmd:

trcli -n \
  -h "https://testrail.instance.com" \
  -u "NAME" \
  -p "PASSWD" \
  --project "Automation" \
  -v parse_junit \
  --suite-id 1 \
  --title "Automation Test Run" \
  --case-matcher name_skip_unknown \
  -f "./report.xml"

only an empty Automation Test Run is created

I am also using a trcli fork that contains the following PR: https://github.com/gurock/trcli/issues/148

Example result from running the cmd:

****
Found 2 unknown test cases to skip.
Creating test run. 
**** API Call
method: POST
url: https://testrail.instance.com/index.php?/api/v2/add_run/12
payload: {'name': 'Automation Test Run', 'suite_id': 582, 'description': '', 'milestone_id': None, 'include_all': False, 'case_ids': []}
response status code: 200
response body: {'id': 1310, 'suite_id': 2, 'name': 'Automation Test Run', 'description': None, 'milestone_id': None, 'assignedto_id': None, 'include_all': False, 'is_completed': False, 'completed_on': None, 'config': None, 'config_ids': [], 'passed_count': 0, 'blocked_count': 0, 'untested_count': 0, 'retest_count': 0, 'failed_count': 0, 'custom_status1_count': 0, 'custom_status2_count': 0, 'custom_status3_count': 0, 'custom_status4_count': 0, 'custom_status5_count': 0, 'custom_status6_count': 0, 'custom_status7_count': 0, 'project_id': 12, 'plan_id': None, 'created_on': 1691663613, 'updated_on': 1691663613, 'refs': None, 'created_by': 12, 'url': 'https://testrail.instance.com/index.php?/runs/view/1310'}
****
Run created: https://testrail.instance.com/index.php?/runs/view/1310
Adding results: 0/0, Done.
No attachments found to upload.

When using the verbose mode of trcli, it shows the list of cases from the automation report, but then the automation ids from the particular run is ignored - as trcli grabs a list of IDs from a specific suite.

Using the cmd without --suite-id creates a new suite - instead, what we expect is a test run with the cases combined from many suites while passing a project when we have -n as argument.

Interested in implementing it yourself?

Yes

d-rede commented 10 months ago

Hello @piotroxp

As you may understand, I can not provide support or speak to any issue that relates to usage of a fork of this repository. Nevertheless, what you are suggesting works perfectly fine with trcli v1.5. I tested your scenario using the report you provided and the specification-first approach using the --case-matcher name option works perfectly.

The only difference to the fork (as far as I know), and note that this is by design, is that you must have valid case IDs in your report. We do not intend to change this because it simply does not make sense to have IDs in the report if they do not exist in TestRail.

Let me know if this helps.

bitcoder commented 8 months ago

Waiting for your feeback @piotroxp to decide on how to handle this issue

bitcoder commented 4 months ago

If no further feedback is received, I'll close this issue

gurock / trcli