TRCLI Failing to Upload Test Cases in Code-First Approach

TestRail CLI Version

v1.6.2

CLI Environment

MacOS Monterey 12.7, Python 3.11.4

TestRail Version

TestRail v8.0.2 Default (3138)

TestRail Instance Type

Professional Cloud

Current behavior

I have a project that I would like to move from a specification-first approach to a code-first approach. I do not need to keep any of the old test cases, so I wanted to see if I can simply take the junit file I was originally including to use for the specification-first approach (and was working with Test Rail, since I had added all the case IDs for this other project into test rail), create a new "Single Repository" Project, and upload this junit file to the new project to see if it would generate the test-cases. However, when trying to upload the file, the trcli would seem to parse the file and create a test run, but not create any associated test cases. Note: I did create an automation_id text field on my project that applies to all projects for the code-first approach as well.

I used the following trcli command with a brand new single repository project, and would recieve the following output:

➜  demeter git:(main) ✗ trcli -v -h "https://ritualco.testrail.io" --project "Code-First Ritual Relay Test Suites" --username "CHANGE_ME" --password "CHANGE_ME" parse_junit --title "Automated Tests Run" -f "test-output.xml"
TestRail CLI v1.6.2
Copyright 2023 Gurock Software GmbH - www.gurock.com
Parser Results Execution Parameters
> Report file: test-output.xml
> Config file: None
> TestRail instance: https://ritualco.testrail.io (user: manorfreeman@ritual.co)
> Project: Code-First Ritual Relay Test Suites
> Run title: Automated Tests Run
> Update run: No
> Add to milestone: No
> Auto-create entities: None
Parsing JUnit report.
Processing JUnit suite - Ritual Demeter Plugin Tests
Processed 180 test cases in 20 sections.
Checking project.
**** API Call
method: GET
url: https://ritualco.testrail.io/index.php?/api/v2/get_projects
response status code: 200
response body: {'offset': 0, 'limit': 250, 'size': 8, '_links': {'next': None, 'prev': None}, 'projects': [{'id': 1, 'name': 'Consumer', 'announcement': None, 'show_announcement': False, 'is_completed': False, 'completed_on': None, 'suite_mode': 1, 'default_role_id': None, 'url': 'https://ritualco.testrail.io/index.php?/projects/overview/1', 'users': [], 'groups': []}, {'id': 2, 'name': 'Merchant', 'announcement': None, 'show_announcement': False, 'is_completed': False, 'completed_on': None, 'suite_mode': 1, 'default_role_id': None, 'url': 'https://ritualco.testrail.io/index.php?/projects/overview/2', 'users': [], 'groups': []}, {'id': 3, 'name': 'OPS/Internal Tools', 'announcement': 'Repo for tests on OCC, Thor and Concierge', 'show_announcement': False, 'is_completed': False, 'completed_on': None, 'suite_mode': 1, 'default_role_id': None, 'url': 'https://ritualco.testrail.io/index.php?/projects/overview/3', 'users': [], 'groups': []}, {'id': 4, 'name': 'Cross Pod Tools/Modules', 'announcement': None, 'show_announcement': False, 'is_completed': False, 'completed_on': None, 'suite_mode': 1, 'default_role_id': None, 'url': 'https://ritualco.testrail.io/index.php?/projects/overview/4', 'users': [], 'groups': []}, {'id': 5, 'name': 'Ritual for Teams Portal', 'announcement': None, 'show_announcement': False, 'is_completed': False, 'completed_on': None, 'suite_mode': 1, 'default_role_id': None, 'url': 'https://ritualco.testrail.io/index.php?/projects/overview/5', 'users': [], 'groups': []}, {'id': 6, 'name': 'Ritual Relay Plugin QA', 'announcement': None, 'show_announcement': False, 'is_completed': False, 'completed_on': None, 'suite_mode': 1, 'default_role_id': None, 'url': 'https://ritualco.testrail.io/index.php?/projects/overview/6', 'users': [], 'groups': []}, {'id': 10, 'name': 'Ritual Relay Merchant Test Suites', 'announcement': None, 'show_announcement': False, 'is_completed': False, 'completed_on': None, 'suite_mode': 1, 'default_role_id': None, 'url': 'https://ritualco.testrail.io/index.php?/projects/overview/10', 'users': [], 'groups': []}, {'id': 11, 'name': 'Code-First Ritual Relay Test Suites', 'announcement': None, 'show_announcement': False, 'is_completed': False, 'completed_on': None, 'suite_mode': 1, 'default_role_id': None, 'url': 'https://ritualco.testrail.io/index.php?/projects/overview/11', 'users': [], 'groups': []}]}
****
Done.

**** API Call
method: GET
url: https://ritualco.testrail.io/index.php?/api/v2/get_suites/11
response status code: 200
response body: [{'id': 231, 'name': 'Master', 'description': None, 'project_id': 11, 'is_master': True, 'is_baseline': False, 'is_completed': False, 'completed_on': None, 'url': 'https://ritualco.testrail.io/index.php?/suites/view/231'}]
****

**** API Call
method: GET
url: https://ritualco.testrail.io/index.php?/api/v2/get_cases/11&suite_id=231
response status code: 200
response body: {'offset': 0, 'limit': 250, 'size': 0, '_links': {'next': None, 'prev': None}, 'cases': []}
****
Found 180 test cases not matching any TestRail case.
Creating test run.
**** API Call
method: POST
url: https://ritualco.testrail.io/index.php?/api/v2/add_run/11
payload: {'name': 'Automated Tests Run', 'suite_id': 231, 'description': '', 'milestone_id': None, 'include_all': False, 'case_ids': []}
response status code: 200
response body: {'id': 889, 'suite_id': 231, 'name': 'Automated Tests Run', 'description': None, 'milestone_id': None, 'assignedto_id': None, 'include_all': False, 'is_completed': False, 'completed_on': None, 'config': None, 'config_ids': [], 'passed_count': 0, 'blocked_count': 0, 'untested_count': 0, 'retest_count': 0, 'failed_count': 0, 'custom_status1_count': 0, 'custom_status2_count': 0, 'custom_status3_count': 0, 'custom_status4_count': 0, 'custom_status5_count': 0, 'custom_status6_count': 0, 'custom_status7_count': 0, 'project_id': 11, 'plan_id': None, 'created_on': 1700508018, 'updated_on': 1700508018, 'refs': None, 'created_by': 53, 'url': 'https://ritualco.testrail.io/index.php?/runs/view/889'}
****
Test run: https://ritualco.testrail.io/index.php?/runs/view/889
Adding results: 0/0, Done.
No attachments found to upload.

When navigating to the uploaded test-run in test-rail, we can see that the test run was created without any test cases and results, despite the TRCLI correctly noting that it parsed 180 test cases in its output.

I attached the example junit "test-output.xml" as well, in case it is helpful to see what the contents of the junit file are. I had to convert it to a text file to be compatible for upload with github: test-output.xml.txt

Desired behavior

Uploading the above test run to a single repository project will automatically:

Create test cases/sections for any unrecognized test cases
Upload any included test results

More Details

I originally opened this as a support issue in Testrail, Request #297154, but was asked to open a Github issue instead.

gurock / trcli