Add support for configurable screenshot filename

[gvnabf]

Agent pool: Private pool Agent specification: macOS Context: Android - codeceptjs-appium Test result format: jUnit (generated by using mocha-junit-reporter, mocha-multi)

Describe the bug Publish test result screenshot task completes but with a warning that says

1 tests failed. Will proceed with screenshot upload.
##[warning]All screenshots were missing. 
Task completed. Published 0/1 screenshots

To Reproduce Steps to reproduce the behavior:

1. Add task right after 'Publish Test Results' task
2. Provide organization
3. Provide screenshot folder
4. Run build pipeline **Part of yaml being used by the pipeline related to publishing of test results

   ...
   - task: PublishTestResults@2
     inputs:
       testResultsFormat: 'JUnit'
       testResultsFiles: '**/TEST-*.xml'
       searchFolder: '$(System.DefaultWorkingDirectory)/output'
       publishRunAttachments: true
       # failTaskOnFailedTests: true
     condition: always()
   - task: Genetec.publish-test-result-screenshot.pipeline-extension.PublishTestResultScreenshot@0
     displayName: 'Publish test result screenshot'
     inputs:
       organization: retailshelf
       screenshotFolder: $(System.DefaultWorkingDirectory)/output/

   Sample artifact for test results:

   

Expected behavior It should be able to locate and publish appropriate screenshot for each failed test.

[eric-labelle]

Can you rerun the pipeline with the debug diagnostic enabled? It should print more information on the paths missing like this:

Failure - No screenshot found for " + className + "/" + testName

---

That being said, your problem is that the path for your screenshot does not seem to match the requirements described on https://marketplace.visualstudio.com/items?itemName=Genetec.publish-test-result-screenshot

You can fetch the json from the testReport and see what these 2 fields are but usually it's the following (you can use Azure DevOps restAPI to see the exact values from these 2 fields)

• automatedTestStorage -> className
• automatedTestName -> methodName

So if you have the following

class MyFirstUnitTest {
  @Test
  fun testOnHappyPath() {
      // your test logic
  }
}

the file would be in {screenshotFolder}/MyFirstUnitTest/testOnHappyPath.png

[gvnabf]

I have rerun the pipeline with debug diagnostic enabled and here's a snippet of the logs

##[debug]screenshotFolder=/Users/****/****/****/_work/2/s/output/
##[debug]Searching for image at path: /Users/****/****/****/_work/2/s/output/User can login with Valid Credentials @loginDebug @regressionCriticalDebug/Login: User can login with Valid Credentials @loginDebug @regressionCriticalDebug.png

Based from the generated test artifact, the screenshot was named something like User_can_l_xx00x0xx-x0xx-00xx-xxx0-xxx0x0x0x0x0.failed.png and was saved under output directory and not {screenshotFolder}/{automatedTestStorage}/{automatedTestName}.png. Please see config used below:

...
helpers: {
    Mochawesome: {
      uniqueScreenshotNames: "true",
    },
},
...
mocha: {
    reporterOptions: {
      "codeceptjs-cli-reporter": {
        stdout: "-",
        options: {
          verbose: true,
          steps: true,
        },
      },
      mochawesome: {
        stdout: "output/console.log",
        options: {
          reportDir: "output",
          reportFilename: someDir,
        },
      },
      "mocha-junit-reporter": {
        stdout: "output/console.log",
        options: {
          mochaFile: "output/TEST-" + someDir + ".xml",
        },
        attachments: true, //add screenshot for a failed test
      },
    },
  },
...

codeceptJS test looks something like

Feature('TestSuite Sample');

Scenario('TestCase Sample', => {
  // steps here
}).tag('testThis')

Extension seems to be expecting screenshot file: {screenshot folder}/{TestCase Sample} @testThis/TestSuite Title: TestCase Sample @testThis.png

[eric-labelle]

The task must match a failed test with a screenshot and for that I needed to use variables available in the JSON response of the testcase. I am not familiar with codeceptJS, but clearly in your case it mess up the class and testname with the framework you are using. It was one feature I had consider to add as options to allow something else than className / testName even though it was what made more sense in pure jUnit. That being said, the only way it can be matched is through fields available in the Azure API response so, even if I add support for different fields, your current setup would not work since your file name seems to be custom. Try their API and get the json response to try and match it with fields in there and let me know which fields would make sense to you. https://docs.microsoft.com/en-us/rest/api/azure/devops/test/results/get?view=azure-devops-rest-5.1#examples

[gvnabf]

Thanks for your assistance. I have adjusted the screenshot naming to align with the plugin requirements. It's now attaching to my tests properly. However, since I'm doing parallel testing on different devices, I encountered a new issue related to parallel stage setup. Here's the link: https://github.com/Genetec/azure-devops-extension-publishtestresultscreenshot/issues/6

Thanks again.

[javierfuentesm]

@gvnabf can you help me please? how did you adjust the screenshoot naming ? Im getting the same error

[kjgen]

Closing issue because the extension is being deprecated and will no longer be maintained. An alternative solution is to use the Publish Test Result task which now supports file attachments.