Adding screenshot and other info to test case name means it complicates statistics

[rasmus-unity]

Example of produced xml file from reporter:

<testsuite name="TestCafe Tests: HeadlessChrome 0.0.0 / Linux 0.0.0" tests="4" failures="0" skipped="0" errors="0" time="71.776" timestamp="Wed, 24 Jan 2018 03:52:04 GMT">
  <testcase classname="Authentication" name="logs in and ensures terms of service has been accepted (screenshots: /testcafe/testresults/screenshots/2018-01-24_03-50-51/test-1)" time="16.596"></testcase>
  <testcase classname="Authentication" name="redirects to projects page after log in (screenshots: /testcafe/testresults/screenshots/2018-01-24_03-50-51/test-2)" time="5.737"></testcase>
  <testcase classname="Project creation and platform data editing" name="creates project with all platforms enabled (screenshots: /testcafe/testresults/screenshots/2018-01-24_03-50-51/test-3)" time="24.841"></testcase>
  <testcase classname="Project creation and platform data editing" name="can add/edit platform details (screenshots: /testcafe/testresults/screenshots/2018-01-24_03-50-51/test-4)" time="24.577"></testcase>
</testsuite>

The added (screenshot ...) part of test case name is added by https://github.com/DevExpress/testcafe-reporter-xunit/blob/master/src/index.js#L42:

if (testRunInfo.unstable)
  name += ' (unstable)';

if (testRunInfo.screenshotPath)
  name += ` (screenshots: ${testRunInfo.screenshotPath})`;

However this means that we cannot use tools like https://wiki.jenkins.io/display/JENKINS/Test+Results+Analyzer+Plugin (or in our case, own tool), as same test case will have different name.

What is the reason for adding the extra info to the test case name in produced xml report? Can it be removed, or should I make my own custom plugin in this case?

Thanks, Rasmus

[AlexanderMoskovkin]

Hi @rasmus-unity,

Thanks for pointing this out. It sounds reasonable. We need some time to investigate this. We'll notify you about our progress

[LavrovArtem]

Hi @rasmus-unity,

I've investigated this issue and found out that the nunit have the attachments and attachment fields which can be used for a screenshot info. But in the xunit I did not find the fields intended for this.

Can anyone to suggest where can we place info about screenshot path in the xunit report? It possible someone knows solutions for this?

[rasmus-unity]

For now we strip parts of test name enclosed when we analyze the test results, so problem as such solved for us. And we have Jenkins pick up the screenshots, so we have those accessible from Jenkins artifacts. However the naming, e.g. testresults/e2e-testcafe/screenshots/2018-02-06_02-48-14/test-1/run-1/HeadlessChrome_61.0.3163_Linux_0.0.0/1.png is not clear, which I guess is the reason why you want to add it to test case title?

We were earlier using https://www.npmjs.com/package/protractor-jasmine2-html-reporter for our Protractor tests, and that would generate more meaningful screenshot filenames, e.g. testresults/e2e/screenshots/should-allow-to-switch-between-multiple-organizations.png

I know I'm going slightly off-topic now, but would it be possible to have TestCafe use the test case name when generating screenshot files? Then I guess there wouldn't be a need to include the filename in test case title of xUnit report file?

[AlexanderMoskovkin]

@rasmus-unity We have proposals and the pull-request for such feature (Screenshot file patterns). It looks like it is what you are asking for. I guess it can take some time to figure out the final version of the API and implement this but I think we'll implement this

[rasmus-unity]

Thanks for considering. As said, we have a working solution by stripping out the screenshot filename from test case result. Ideally don't think there is a reason for adding additional information to test case name in result XML file, but might just be my use case is different from others. You can close this issue if you like.

[borisivan]

We started using test cafe and noticed how the test names themselves are changing due to whether or not there is a screenshot, etc. This really should be fixed.

I saw how there was a PR and it was rejected because some people might like how the test names change -- I really wouldn't think that's the case, but if it is, can a solution exist in which the default behavior is to not change test names on the fly, and there is an optional parameter given to those who want their test names to change?

All Jenkins plugins and other processing tools are quite surprised to note that when calculating trends for how long certain tests have been passing/failing, trend of execution time in seconds, etc -- that the test names are changing and therefore they are all "new tests introduced in this last execution", this breaks most every downstream reporting associated with a pipeline of automated test activity.

[AndreyBelym]

Thanks for the feedback, we will improve the reporter to don't store volatile metadata in test names. We plan to release testcafe@1.0.0 in January and ship all potential breaking changes with it. It means we will publish a new version of the reporter together with the new TestCafe version.

[akdor1154]

@AndreyBelym did this get missed with 1.0.0?

[alexschwantes]

FYI, until they fix this issue, I forked this repo to accomplish this over at https://github.com/alexschwantes/testcafe-reporter-junit

[AndreyBelym]

@alexschwantes thank you for using TestCafe and creating your reporter. If you don't mind, can I add your plugin to the list of recommended reporters in the Plugins section in the TestCafe documentation?

[alexschwantes]

@AndreyBelym yes absolutely

[AndreyBelym]

Thank you so much 👍 🦄

[Nisden]

Would it be possible to get an option that would disable this behavior, if you don't want to change the existing functionality?

[AlexKamaev]

@Nisden We do not have plans to introduce an option for these purposes. However, we may reconsider our decision in the future.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for a long period. It will be closed and archived if no further activity occurs. However, we may return to this issue in the future. If it still affects you or you have any additional information regarding it, please leave a comment and we will keep it open.

[github-actions[bot]]

We're closing this issue after a prolonged period of inactivity. If it still affects you, please add a comment to this issue with up-to-date information. Thank you.