Support for VSTS & VSTS Test Attachments

jessehouwing commented 6 years ago

Current behavior:

I've installed Cypress into a project built on VSTS. To get cypress to run on the Linux Hosted agents I had to perform the following additional steps:

apt-get -qq -y update
apt-get -qq -y install xvfb libgtk2.0-0 libnotify-dev libgconf-2-4 libnss3 libxss1 libasound2

To get reporting to work (to get status information in the VSTS release dashboard) I configured the junit reporter:

cypress run --reporter junit

This works up to a certain point. Cypress runs, a junit report is generated, but all of the recorded videos are ignored after the run is completed.

VSTS supports uploading test attachments when they are referenced in the test report. I tried configuring the xunit-file reporter, but no attachment is added. I tried configuring the mocha-trx-reporter (native visual studio format), but it crashes when running with cypress.

TypeError: test.isPending is not a function
    at C:\temp\node_modules\mocha-trx-reporter\lib\trx.js:63:22
    at Array.forEach (<anonymous>:null:null)
    at Runner.<anonymous> (C:\temp\node_modules\mocha-trx-reporter\lib\trx.js:62:34)
    at emitOne (events.js:120:20)
    at Runner.emit (events.js:210:7)
    at Reporter.emit (C:\temp\node_modules\cypress\dist\Cypress\resources\app\packages\server\lib\reporter.js:191:55)
    at Object.server.startWebsockets.onMocha (C:\temp\node_modules\cypress\dist\Cypress\resources\app\packages\server\lib\project.js:288:22)
    at Socket.<anonymous> (C:\temp\node_modules\cypress\dist\Cypress\resources\app\packages\server\lib\socket.js:237:36)
    at emitOne (events.js:115:13)
    at Socket.emit (events.js:210:7)
    at C:\temp\node_modules\cypress\dist\Cypress\resources\app\packages\socket\node_modules\socket.io\lib\socket.js:503:12
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickCallback (internal/process/next_tick.js:180:9)

Desired behavior:

Cypress logs the recorded video as test attachment in the generated test results so that Visual Studio Team Services can associate them to the test results and show them in the portal.

Steps to reproduce:

Run cypress in a Visual Studio Team Services build.
Run the "Upload test results" step and publish the results file

Check the VSTS portal for the presence of test attachments.

Versions

Windows 10 April 2018 or Ubuntu (https://github.com/Microsoft/vsts-agent-docker/blob/master/ubuntu/16.04/standard/Dockerfile)
Cypress: 2.1.0
VSTS Agent version: 2.133.3

SeriousM commented 5 years ago

any updates on this?

erik-peterson-fox commented 5 years ago

Please add support for this if it's not already there...This would be very useful.

jennifer-shehane commented 5 years ago

Some related work here: https://github.com/cypress-io/cypress/issues/2433

SebastianSchoeberl commented 5 years ago

Any updates here? As far as I know cypres uses the Mocha-Junit-Reporter, which should support attachment files already. Anybody got that working with cypress and CI/CD?

YOU54F commented 5 years ago

I came across this error today, on a totally unrelated issue, see my CI build here

$(npm bin)/cypress run --spec cypress/integration/sample1.spec.js --reporter-options "reportFilename=test1"

For whatever reason, it was trying to use the xunit reporter, despite the fact that my reporterOptions are set to use cypress-multi-reporters, mochawesome and mocha-junit-reporters.

If i comment out the following function at node_modules/mocha/lib/reporters/xunit.js

https://github.com/mochajs/mocha/blob/f3ed729b6908f3893e6f87722c6e996cca5f7733/lib/reporters/xunit.js#L182

I get a xunit xml file at the root of my project called xunit.xml

<testsuite name="Mocha Tests" tests="1" failures="0" errors="0" skipped="0" timestamp="Tue, 04 Jun 2019 23:30:02 GMT" time="2.259">
<testcase classname="Cypress parallel run example - 1" name="should display the title" time="0.645"/>
</testsuite>

This is the equivalent junit report

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Mocha Tests" time="0.527" tests="1" failures="0">
  <testsuite name="Root Suite" timestamp="2019-06-04T23:39:48" tests="0" failures="0" time="0">
  </testsuite>
  <testsuite name="Cypress parallel run example - 1" timestamp="2019-06-04T23:39:48" tests="1" failures="0" time="0.527">
    <testcase name="Cypress parallel run example - 1 should display the title" time="0.527" classname="should display the title">
    </testcase>
  </testsuite>
</testsuites>

I'm not entirely sure why an xunit file is being emitted! as it isn't specified in any of my setup!

Here is my cypress.json

{
  "reporter": "cypress-multi-reporters",
  "reporterOptions": {
    "configFile": "reporterOpts.json"
  }
}

and reporterOpts.json

{
  "reporterEnabled": "mocha-junit-reporters, mochawesome",
  "mochaJunitReportersReporterOptions": {
    "mochaFile": "cypress/reports/junit/test_results[hash].xml",
    "toConsole": false
  },
  "mochawesomeReporterOptions": {
    "reportDir": "cypress/reports/mocha",
    "quiet": true,
    "overwrite": false,
    "html": false,
    "json": true
  }
}

It may or may not be relevant to this issue, just thought I would help add some additional info!

marceloavf commented 5 years ago

Still nothing @SebastianSchoeberl

I saw that they support Selenium, publishing images and test results, but for now I'm only testing e2e and failing the build if they don't pass, not recording anything in portal.

SebastianSchoeberl commented 5 years ago

Thanks for your update @marceloavf , well I suppose we have to hold on a little more. But would be great if they made it possible!

kesiyaabraham commented 4 years ago

Hi ,

Do we have any other option available in cypress to attach screenshots and videos to azure test results other than directly using DevOps API?

hmilas commented 4 years ago

Any update on this one?

hmilas commented 4 years ago

You surely have given up on this one

mschoneman commented 4 years ago

I was really hoping this was going to be fixed in the latest 4.2 upgrade

krileo commented 4 years ago

I wrote a PowerShell script that can be run as a task in the Azure Pipeline. It will locate and attach the screenshots from Cypress to the tests results for failing tests.

You can find the code here and a blog post on how to use it here.

cypress-io / cypress