search

gurock / trcli

TR CLI (trcli) is a command line tool for interacting with TestRail.
Mozilla Public License 2.0
47 stars 40 forks source link

trcli fails to update existing suites, defaults to case duplication #246

Open ablack-jpl opened 1 month ago

ablack-jpl commented 1 month ago

TestRail CLI Version

1.9.5

CLI Environment

Mac (apple silicon), python3.10.12

TestRail Version

8.0.4

TestRail Instance Type

Enterprise Server

Current behavior

Given the following setup:

I executed the variations of the following trcli command from the directory where the test file is downloaded, with the indicated variations and received the described results.

trcli \         
  -y \
  -h ${server} \
  --project ablack_test \
  --username ${email} \
  --password ${api_key} \
  parse_junit \
  --suite-name "test_suite" \
  --title "test1" \
  -f "TEST-gov.nasa.jpl.mbee.mdk.util.UtilsTest.xml"

When I run the above command but omit the --title "test" argument, I receive an error that the tool cannot run without generating a test run as well.

When I run the above command as is, the upload fails and I receive the following error:

Error occurred while checking for 'missing test cases': 'Field :limit is too small (minimum 1).'
Error occurred while checking for 'missing sections': 'Field :limit is too small (minimum 1).'

I receive similar errors when I substitute --suite-id {real number}, --section-name "gov.nasa.jpl.mbee.mdk.util.UtilsTest", and --section-id {real number}. In all cases, the tool fails to update the existing suite with the cases.

I also receive similar errors when I run with made up names for suite and section.

When I run the above command but omit the -- suite-name "test_suite" argument, the upload works, but not in a way that is repeatable. Specifically:

  1. A new suite is generated for the upload, and suite contains the timestamp from the upload.
  2. Multiple runs of the command generate multiple new suites, each with duplicate cases and timestamped suite names.

Desired behavior

I expect five(ish) things:

  1. When I provide a --suite-name or --suite-id, the trcli tool will attempt to add the new items to container with the same name / id. a. If I provide a name that does not exist in the project, I expect tool to create a container with that name (inferring section name as necessary if not provided) and to add the cases to that new container. b. If I provide a name or id that does exist, it will succeed at adding those and not fail with an error that "Error occurred while checking for 'missing test cases': 'Field :limit is too small (minimum 1).'"
  2. I expect similar behavior for --section-name and --section-id, though I also expect at least -name to need a named suite to accommodate. This includes sub-point behavior as appropriate.
  3. I expect that when I am uploading new cases for a project, that all cases in the project (or at least all in the named / inferred suite) are evaluated so that duplicates are not created if they have a matching automation_id field value. a. Significantly, when I re-run the same command without a specified suite or section name, I expect that it will NOT generate duplicate test cases but will correctly identify previously uploaded test cases and not re-create them.
  4. I expect that can add test cases without having to add test runs so that the suite cases look like I expect testing to look and I can identify untested cases (for those times when things are skipped). To support this, the test run based "--title ${test_run}" flag should be optional, and when missing it should just update the test suite / sections (inferred or specified) to add the new cases.
  5. Created test suites should not have test run dates associated with them. Test runs can (and probably should) have that, but I expect the same tests to run over and over and over and over again with minor updates or additions (see above points) so test suites should be date agnostic and should instead pull a test suite name from the ingested file (if not otherwise provided at ingestion time via the appropriate *-name argument).

More Details

I've vented in other fields already, "more details" will thus be empty-ish.

ablack-jpl commented 1 month ago

There's probably a feature request (make title optional so I can just upload test cases without results) mixed in with this bug report (--suite-name field isn't working, suites are duplicated if the name fields are missing) that I should separate out...

acuanico-tr-galt commented 1 month ago
  1. When they provide a --suite-name or --suite-id, the trcli tool will attempt...

Workaround 1: There is an option to have the test cases created using "Specification First", in this approach you have to create the test cases first in Testrail, map the IDs into your code or test report and when you are ready to upload the results to Testrail, indicate the “--case-matcher” and point to the property where you put the IDs together with the prompt mechanism set to “-n” argument to ensure test cases are not duplicated/automatically created. For more a comprehensive guide see: Specification-first workflow Workaround 2: Automate the IDs generation and test cases creation by uploading the test report initially (this would mean that the test cases will be uploaded to Testrail at first with prompt mechanism set to “-y” so you dont need to manually create them and IDs are generated and assigned, on the next test execution, you need to have the IDs mapped into your code/test reports, put a "--case-matcher” and prompt mechanism to “-n" argument to be able to ensure that they won’t be duplicated. To do this:

  1. They expect similar behavior for --section-name and --section-id...
  1. They expect that when they are uploading new cases for a project...
  1. They expect that can add test cases without having to add test runs ….
  1. Created test suites should not have test run dates associated with them...
yashpunyani25 commented 3 weeks ago

Hi,

Until couple of weeks back I was able to push my code to testrail after every execution simply using command :

trcli -y -c "trcli-config.yml" parse_junit -f "./target/TEST-junit-jupiter.xml"

Suddenly I started getting error : Error occurred while checking for 'missing sections': 'Field :limit is too small (minimum 1).'

I am using below dependency

com.testrail 
        <artifactId>testrail-junit-extensions</artifactId>
        <version>0.2.1</version>
    </dependency>

Can you suggest what changed suddenly? how should I resolve this. I cannot ask my team to manually create test and map thats whyI was using above dependency

ablack-jpl commented 3 weeks ago

@yashpunyani25 - You are missing required arguments in your invocation and I can't tell if they're in your config file or not. I'd guess --project or --suite-id are missing or your config file moved or something, but there's not enough information to pick it apart. My now successful runs are invoked like this:

trcli \
  -h "${HOST}" \
  --project "${PROJECT_NAME}" \
  --username "${EMAIL}" \
  --password "${API_KEY}" \
  parse_junit \
  --case-matcher auto \
  --suite-id "${SUITE_ID}" \
  --title "${NEW_TEST_RUN_TITLE}" \
  -f path/to/results/file

Hopefully there's enough in there for you to determine the missing bits of yours.

@acuanico-tr-galt - From your response I worked out that I needed to add the --case-matcher auto --suite-id NUMBER options to mostly achieve expectaitons 1-3. But

Case matcher is interesting for a couple of reasons. The first is that it requires a specified option and I could not significant documentation on what the options for it are. I ran across the "auto" option on the Code-first specification documentation page; without that I would have had no way of telling how to get this behavior. But that entry is problematic to me as well. It indicates that

This is the default approach supported by the TestRail CLI. It uses the --case-matcher auto option, which automatically creates test cases and adds their results in TestRail.

The wording here might be a bit unclear, but calling it a default suggests to me that it is a behavior run when a user doesn't specify otherwise. The option didn't appear in any of the documentation's examples either, which I would expect if you didn't want to default to case duplication. And I genuinely don't know why you wouldn't default to not duplicating test cases. Duplicating cases needlessly bloats your test project, means you can't associate references or requirements with test cases manually, destroys case and result history, etc. It is counter to all of the testrail recommendations and best practice I've seen in your videos and documentation.

That option wasn't enough to prevent duplication on its own however. Because trcli appends runtime to the created suite names (which is NOT discussed in the mapping documentation), each trcli run generates a new suite and populates new cases there as well. The use of runtime instead of testtime (which is present in the testsuite.timestamp field of most junit xml) is particularly frustrating since it means that I get a duplicate suite and cases for each invocation instead of just each testfile.

Ultimately specifying the id of an empty suite (or a suite that already had cases with the appropriate automation_id values) resolved the duplication issue, but I again am at a loss for why the tool would make it hard to achieve what looks like should be the baseline behavior.

All of this is to say that expectations 1-3 should probably be re-contextualized a bit. I expect trcli to only create new suites and cases as a last resort. I expect trcli to default to finding existing suites and cases, and updating them with new values from the xml if present (but otherwise retaining values present on testrail). I expect the command invocation to deviate from that behavior should be harder than the base case. I also expect that the mappings documentation includes correct names for the testrail suites, which it does not currently given the lack of timestamp discussion.

For expectation 4, fair enough.

I think expectation 5 might have been misunderstood a bit. The current arrangement makes test runs with the same name (via the title argument) and test suites with all different names (because each has a unique runtime stamp), and that seems exactly backwards.

I agree that I'm ingesting xml results, and that results should be the primary expectation of ingesting this file. But timestamping the suite is unrelated to results. It makes the name unique in ways that aren't helpful, and encourages suite (and thus case) duplication. It means that even if I wanted to match on --suite-name I would need an already created suite OR to know the timestamp of the suite that was first created by trcli. And it's not reflected in the mappings documentation at all.

Removing the timestamp from the suite makes suite-name matching easier, makes it easier to find existing cases when suite name isn't provided (because you have specified the mapping from file to suite already), makes the mapping documentation correct again, and doesn't violate best practices around over-specifying project suites so they can be reused for multiple runs. So again, I expect test suites to not have timestamps on them. I would be quite happy with a timestamp on the test run though, even appended to the test run title argument (where I currently manually generate a timestamp).

yashpunyani25 commented 3 weeks ago

@ablack-jpl Thanks bro. But the problem is happening for me with -y option. When i run below command i get error "Error occurred while checking for 'missing sections': 'Field :limit is too small (minimum 1).'"

trcli -y \ -h "HOSTNAME" \ --project "PROJECTNAME" \ --username "USERNAME" \ --password "PASSWORD" \ parse_junit \ --case-matcher auto \ --suite-id "SUITEID" \ --title "Automated Tests Run" \ -f "./target/TEST-junit-jupiter.xml"

But when I remove the -y flag its working fine but not inserting the new test case. I am literally stuck from a mont. A prompt reply will help

acuanico-tr-galt commented 1 week ago

Hi @ablack-jpl Thank you so much for the response. Apologies for replying a bit late. Rest assured that we have your suggestions noted and these will be used to improve the future releases of TRCLI. Thank you!

ablack-jpl commented 1 week ago

@acuanico-tr-galt are any of these suggestions worth my time to implement and open a PR for? I recognize that some of these are significant changes from current behavior, and I'm not interested in spending time on a feature that doesn't align with maintainer goals and is going to sit in PR purgatory forever.

@yashpunyani25 I'd try adding a --project-id "PROJID" argument in there as well. Best guess is that the missing sections error pops up when it can't uniquely identify a suite to put stuff in, and project names aren't guaranteed unique. If that doesn't work, I'll recommend you read the source to figure it out or just create your own ingester against the REST API.

acuanico-tr-galt commented 5 days ago

Hi @ablack-jpl Thank you for your interest in contributing to our codebase! We truly value your input and are excited to see the enhancements you have suggested. It’s crucial that any new features or changes you propose maintain backward compatibility. This means that the default behavior of the code should remain unaffected unless a user explicitly opts into the new functionality. This approach helps us ensure that existing users won’t encounter any unexpected changes or disruptions in their workflows.

While we are open to discussing and eventually implementing major changes, these require more thorough consideration. We prioritize maintaining stability and continuity in the project, so major changes will be reviewed carefully and may not be approved as quickly as minor, backward-compatible enhancements. I encourage you to first propose them in a discussion so we can collectively weigh the benefits and potential impacts.

Speaking of PRs, we have recently closed quite a few community PRs on the latest release. So yes, if you have the time, you are very welcome to do so :)