search

gurock / trcli

TR CLI (trcli) is a command line tool for interacting with TestRail.
Mozilla Public License 2.0
48 stars 39 forks source link

Add a flag -ReportSkippedAs Retest|AcceptedFailure|etc. #159

Open JonathansDCI opened 10 months ago

JonathansDCI commented 10 months ago

What would you like the TestRail CLI to be able to do?

Tell Testrail how to handle skipped tests.

Why is this feature necessary on the TestRail CLI?

A great way to handle an Accepted Failure in automation is to set some variation of IgnoredReason = "link to your bug causing this test to be skipped".

However, your default behavior is to take skipped automation (that look great in Visual Studio and you can see the bug link) and set that to "ReTest".

Well I am never going to manually change daily test results of skipped... for eternity or I quit my current job from retest to Accepted Failure. So I will just check my daily results and close them.

So lets be accurate. I know why I skipped the test, so let me tell TestRail what the result of a skipped test vs. letting TestRail determine it.

More details

No response

Interested in implementing it yourself?

No

bitcoder commented 10 months ago

I think this issue is similar to #107

JonathansDCI commented 10 months ago

Yes but might expand the issue a little bit since Skipped might not be a true result state. That said you could solve my problem and theirs by allowing a mapping in the cli.yml file. ResultMapping: { Passed = Automation Passed, Failed = Automation Failed, Skipped = Automation Accepted Failure }

I know the old school managers that like to dot i's and cross t's would also kinda freak about the concept of closing a test run with an Accepted Failure that doesn't link to a bug in the repro... so to massage their pedantic brains a Custom Result of Automation Accepted Failure which includes the concept of not attaching the bug would calm them down.

Now my ask is avoiding the obvious that the solution to my issue is to also change the Nunit report to JUnit report transform such that the reason string is available for attaching to the test result comments in test rail.

If I really need this, I could call the API directly and stop using the test cli but it feels so good stripping code from the project and move reporting to a line in Jenkins using the cli.