Automation ID field does not work if test case names are different with test case names in automation project

[tdnguyen84]

Problem statement

My issue is:

1. Let's say I set up automation_id custom field for my project

2. My junit xml like below

   <testsuites name="test suites root">
   <testsuite failures="0" errors="0" skipped="1" tests="1" time="0.05" name="LoginTests">
   <properties><property name="setting1" value="True"/></properties>
   <testcase classname="LoginTests" name="test_case_1" time="159">
     <skipped type="pytest.skip" message="Please skip">skipped by user</skipped>
   </testcase>
   <testcase classname="LoginTests" name="test_case_2" time="650">
   </testcase>
   </testsuite>
   </testsuites>

3. In TesRail, my project will have section & test cases are structured as below

LoginTests
|
|____Test case 1 (has Automation ID = `LoginTests.test_case_1`)
|
|____Test case 2 (has Automation ID = `LoginTests.test_case_2`)

4. Now when I run the Testrail CLI with right configure and I'm seeing
   • It can create a new Test Run, but there neither test case "Test case 1" nor "Test case 2" are added to the test run

My expectation:

• It should add the existing test cases "Test case 1" and "Test case 2" to the test run because they have the "Automation ID" matched with combination "classname.name" of node in xml file

What I played around

• If I rename my test cases in TestRail like "test_case_1" and "test_case_2" with the Automation ID are still like above
• Re-run TestRail CLI, those existing test cases will be added to test run successfully

So my concern and question Are we requiring both test case name and automation id in TestRail to be able to map the existing test cases to add to test run when running TestRail CLI?

My use case here is, I have a lot of test cases defined in TestRail that were set the names are not the same in automation test scripts. So we just tried to update the Automation IDs with the format we defined in TestRail CLI document and hopefully the tool can base on the automation ids to add to test result.

Originally posted by @tdnguyen84 in https://github.com/gurock/trcli/issues/70#issuecomment-1187594346

[ddanielyan]

We are also experiencing the same issue, it seems the automation ID does not work if Test Name on Test Rails and Test Name in the automation project do not match, was hoping to hear back if this is a bug or it is expected? Please update.

Thank you.

[molibdenmatt]

Hi, Any updates on this? I've got a similar case, even weirder one:

1. There are pretty complex BDD scenarios in my automation. One scenario is visible as a one in the Junit.xml report but this one scenario covers multiple TestCases in the TestRail
2. TestCases in TestRail are not changable. I can't edit their Titles
3. When I set the automation_id custom field to eg. xxx.yyy but the TC Title in TestRail is "zzz" the trcli parser uploads 0/0 results

I was hoping to add xxx.yyy automation_id to multiple TC's in TR and with one trcli run pass them all.

Could this be even configurable so the trcli shoud or should not care about the Test Title in TestRail? The automation_id is enough to map the TC's

[amuthan-sakthivel]

Hi Team, I am facing the same issue. It would make sense to have automation_id as the only thing to map the Test Cases

[d-rede]

Hello everyone!

Thanks for reporting this issue @tdnguyen84, it definitely wasn't an expected behavior. Version 1.3.1 was just released and should have this fixed by matching cases exclusively by automation_id. If this solves your problem, please close the issue :)