Uploading malformed results to an already created test suite can casue the suite to be delted

TestRail CLI Version

1.6.1

CLI Environment

Windows 10, Python 3.10.10

TestRail Version

TestRail v8.0.1 Default (1033)

TestRail Instance Type

Professional Cloud

Current behavior

Was working on formating VStest automation results file (TRX) into the correct XML that matched JUNIT format and contained the right information to match our already created tests in the cloud.

I had a sample script that worked on a single result to manage uploads, so I tried with out full suite. When I ran the parse_junit it ended up deleting the existing test suite and all our data. I had selected -n for autocreate as I did not want to have to clean up any badly formatted tests that would create new cases, sections, or even suites.

I ran the following command: >trcli -n -h [URL] --project "Smart Package Room and iPickup Projects" --username [USER] --password [PASS] parse_junit --case-matcher "property" --title "Testing automated upload" -f FinalTestResult.xml

This was the output

Report file: FinalTestResult.xml Config file: None TestRail instance: https://positionimagingqa.testrail.io/ (user: eringo@position-imaging.com) Project: Smart Package Room and iPickup Projects Run title: Testing automated upload Update run: No Add to milestone: No Auto-create entities: False Parsing JUnit report. Processing suite - iPickup UPS - Front End Processed 131 test cases in 13 sections. Checking project. Done. Found 3 test cases without case ID in the report file. Nonexistent case IDs found in the report file: [1448, 799, 774, 782, 797, 1545, 1542, 1541, 886, 888] Error occurred while checking for 'missing test cases': 'Case IDs not in TestRail project or suite were detected in the report file.' Creating test run. Field :case_ids contains unrecognized case IDs. Deleted created suite

I believe the problem is that when the results_uploader tries to get the suite id on line 60, though I may not be reading the code right.

def get_suite_id(self, project_id: int, suite_mode: int) -> Tuple[int, int]: says that if it a new suite is created it will return the ID otherwise it will return -1. So in all other cases we should expect a -1 regardless of whether a suite id was specified in the command or not. However if I read the code right that would mean in the case where a user does not specify a suite id no suite ID is set.

Also Line 173 calls ApiRequestHandler.resolve_suite_id_using_name(self, project_id: int) -> Tuple[int, str] and it appears that that will return a valid test suite ID if one matches. That value is then returned instead of -1. The same appears to be true on line 196 and line 211 which both call ApiRequestHandler.get_suite_ids(self, project_id: int) -> Tuple[List[int], str]: which then return valid ids, which are in turn returned to the upload_results(self) method.

The problem with this is that upload_results assigns that value to added_suite_id, regardless of whether it was an added suite or not. So when the upload chokes and calls def rollback_changes(self, added_suite_id=0, added_sections=None, added_test_cases=None, run_id=0) it puts the added_suite_id as the added_suite_id and then the rollback will delete it, assuming it meets other criteria, namely that the SuiteMode was not single_suite. Since my project has, or well had, 5 projects the suite mode should probably have been set to multiple_suites.

Desired behavior

I would expect suites that already exists not to be delete when rolling back changes.

More Details

Screenshot 2023-09-19 130652

I've created a suppor request to get my data back here https://support.testrail.com/hc/en-us/requests/289862

gurock / trcli