search

gurock / trcli

TR CLI (trcli) is a command line tool for interacting with TestRail.
Mozilla Public License 2.0
48 stars 39 forks source link

Add an option to create only TestCases, but skip adding or updating Runs or Results. #149

Open Testinator-X opened 1 year ago

Testinator-X commented 1 year ago

What would you like the TestRail CLI to be able to do?

Add an option to only create TestCases, but skip adding them to Runs and skip adding Results.

Why is this feature necessary on the TestRail CLI?

You cannot automatically update an existing run with newly created TestCases. On the one hand, this is currently not supported by the CLI tool, on the other hand, test cases created via the API are not added to runs even though they match the dynamic filter criteria of the run. (If you think the dynamic filtering vs API problem should be fixed, vote for my feature request by creating a support request here and reference my Request #265113) . When the CLI tool is currently running, the cases are created, cannot be added to the existing run, the CLI tool throws an error and deletes all cases that were created in the first step. (A workaround would be to create a new run each time, but this would unnecessarily spam our instance with runs.) We would like to be able to just create new TestCases via the CLI in TestRail without also having to create a new Run each time, into which the results then have to be fully reported. If needed, we could then manually update our existing run after creating the TestCases via the CLI.

More details

See also:

https://github.com/gurock/trcli/issues/99 https://github.com/gurock/trcli/issues/148 https://github.com/gurock/trcli/issues/136

Interested in implementing it yourself?

No

bitcoder commented 8 months ago

If others are interested on this feature, please leave your feedback here and tell a bit more about your specific use case.

Testinator-X commented 8 months ago

I have described my use case in detail here: https://github.com/gurock/trcli/issues/136#issuecomment-1551552385

In general, the point is that our automated regression suite is gradually being extended with new TestCases and we would like to add new TestCases to a Run in an automated way. Currently we have to do the following relatively complex workaround:

  1. we create the new cases with the CLI tool and add them to a new temporary run, so that the import does not fail and the new cases are not deleted again.
  2. we compare the tests of the temporary run with the tests of the existing run without using the CLI tool via the API and update the existing run using the API, i.e. we add missing new tests to the existing run.
  3. we delete the temporary run using the API so as not to unnecessarily spam TestRail with runs.
bitcoder commented 8 months ago

@Testinator-X ,

  1. are your test methods annotated with the test case ids?
  2. which "matcher" are you using whenever importing the test results with the CLI?
  3. your initial need, from my understanding, is that you aim to import test results, continuously, always to the same run.. while your list of automated tests keeps growing, right?
Testinator-X commented 8 months ago

@bitcoder

  1. I‘m not sure what you are meaning. We write our automated tests using „Katalon Studio“ and use the report.xml as it is created by Katalon Studio.
  2. We use the „code first approach“. We create new Tests in Katalon and use the CLI-Tool with the report.xml to create new TestCases in TestRail. And we use the default settings to update existing Cases.
  3. Exactly.

From our point of view, the smartest thing to do would be to fix the following bug in the TestRail application itself:

"TestCases created via the API are not automatically added to a TestRun that uses dynamic filtering."

If this were possible, newly programmed tests would be created in TestRail using the CLI tool and automatically added to the run at the same time. The cases would then be in the run and the CLI tool would no longer complain when adding results that the new tests could not be found in the run. In this error situation, no rollback would be performed and the newly created TestCases would remain in TestRail.

If this worked, we could save ourselves the whole workaround with the temporary run and the manual synchronization of the TestCases. The only thing missing would be the ability to create runs with dynamic filtering via the CLI tool (or the API itself)