Handle TestRail Sections more intelligently

[xirsoi]

Currently the plugin handles sections in an idiotic fashion. It creates a new section with the name of the test results file if one by that name doesn't exist, even if the test case exists in a different section.

My current thought is to ignore sections unless there are multiple test cases with the same name.

I am open to other ideas though. Altering this behavior could have a large impact on those using the plugin, and I don't want to make assumptions about how it is used.

[sraschitor]

Hello,

Probably the ideal solution would be to somehow link a certain automated test with an existent test case from Testrail using it's id. I have found a similar concept in a different plug-in that was developed for Python. (https://github.com/eventbrite/nose-annotator). But to be honest I do not know how much work and development this would imply.

But I believe a more feasible solution could come right from the project's setting. Currently a Jenkins job will add all tests from the automated test suite in the Testrail's designated test suite under a single Section, even though maybe some sections are already there with each one of them having a few test cases under them.

• Starting from the general idea that most automated projects have different classes with multiple test methods that have a similar objective I would suggest for this behavior to be migrated in Testrail when posting results. For reach Class create a Section with the same name and under the section add the automated tests as test cases. After the job is done, the user would be able to edit the test cases by adding steps, issues id's etc. and finally have a project with a test suite that contains sections focused on different objectives that contain all test cases related to it.

• Example: LogIn Class in the automated project will contain logIn_1, logIn_2 tests; MainMenu Class will contain mainMenu_1, mainMenu_2 tests. After Jenkins job completes, create LogIn and MainMenu sections with each one of them having assigned their respective test cases. (This is just a basic example of the plugin's post result logic).

I believe that this will add some order in the test suite run compared to having 100 tests bulked and mixed, Sorry for the long post :)

[xirsoi]

I think that is a good behavior in the case that a test case does not already exist, but I think it would be best to ignore Sections when it does exist.

Is this acceptable?

[xirsoi]

I think having a per job toggle for old and new behavior would be good too. Basically: "Test Suites as Section Names" vs "Namespaces and Class names as Section names.

For example, if you have a Test method ProductName.CMS.Login_AsPublisher_WithGoodCredentials() it would be in section ProductName, subsection CMS, and the test case would be named Login_AsPublisher_WithGoodCredentials.

How does that sound?

[sraschitor]

If the Sections would be ignored altogether, after the first job run a Testrail user would be able move, add, edit tests in different Sections of the Test Suite and for the next Jenkins job runs, the same tests would receive the result?

If yes then this approach would be great, right now the tests are always duplicated after each job run if they are moved in different section, or the current section name is edited.

[xirsoi]

Yeah, that's how it would work. At my work, the placement of the test cases in sections has almost nothing to do with how they are organized in the code, so this is doubly important for our situation.

[alshari86]

@sraschitor Any update on this enhancement? I tried the plugin and it always create section as well as test cases after running. After each run my test suites got duplicate cases created. Another thing is in my automation code, the case name is the function name in that class, but the case name in testrail is different, how can I map it? Seems like this doesn't support using id in testrail, right? So everytime it will also create test cases because of different name in automation code and testrail

[sraschitor]

Hi @alshari86 ,

I think that Adam is currently working on the changes, maybe he can update us on the status.

[mikec-bullhorn]

I think allowing us to map the test suite (from the jJenkins side) to the suite + section would be nice, especially if we could pass in the section programatically.

The use case I have been dealing with is the same set of tests run on different environments. Right now we are using open ended milestones to separate the runs, but we have to set up separate jobs for each environment. if we could pass the environment to the notifier in such a way that we could use one job with different parameters/inputs, that could reduce maintenance of the test jobs.

[fimash]

Would be great to exclude addition of 'untested' test in Run. For example, if we have 2 sections and only 1 section of tests executed, run populated with 2 sections - tested and untested http://i.prntscr.com/nUpYyZR7Th_i2XRefO_fcA.png

[xirsoi]

So, don't report untested results?

[fimash]

Yes, exactly

[fimash]

Any luck?

[xirsoi]

I haven't had time to work on this project in a while. I'll see if I can make time soon.

[aziziazizan9]

Can this plugin support using id in testrail ??? or how to edit it manually?

[xirsoi]

This isn't the place for that question but no, It's not currently possible to link by testrail id.

[aziziazizan9]

okayy.sorry and thanks

[xirsoi]

Np, next time you have a question please feel free to make a new issue :)

[yangquan1982]

Can this plugin create new subsections with test cases under the section that I created manually in TestRail? That would be better to manage all test cases generated after the first run of Jenkins job.

[systemsincode]

Currently it seems you Test rail section is defined by root testsuite pproperty value in you Junit report xml, which is in turn defined by the tested class's package/namespace. I could work around that by parametising this value in my mvn project so I can edit a property on each Jenkins job.

However what I really want is each cucumber feature file to be a sub-section with its test nested under it, so I have clear tree of feature file under each Jenkins job section. Currently I believe the feature files map to classname sections in Junit report xml. Is there a way to get these to generate subsections.?

[systemsincode]

I've posted on SO about how to generate nested testsuites that seem to work with the testrail plugin. https://stackoverflow.com/questions/57577858/how-to-create-nested-testsuite-tags-in-junit-xml-reports Can anyone help on this?

[Vadim2305]

Yeah, that's how it would work. At my work, the placement of the test cases in sections has almost nothing to do with how they are organized in the code, so this is doubly important for our situation.

It's good solution. Do you have any predictions when it will work?