reportportal / agent-js-cypress

Agent to integrate Cypress with ReportPortal.
https://www.npmjs.com/package/@reportportal/agent-js-cypress
Apache License 2.0
39 stars 39 forks source link

Uploading tests gets stuck when executed from Git Pipeline for failed test cases #37

Open vishallanke opened 4 years ago

vishallanke commented 4 years ago

image

vishallanke commented 4 years ago

It seems agent is not working from Git if test cases gets failed. If test case gets failed, then those launches and attachment wont get uploaded.

I found a launch, when it goes to completed state after 1 day. It has 2 test cases. 1 Pass and 1 failed. Only pass test case gets visible and that after 1 day.

image

vishallanke commented 4 years ago

This issue still occurs on latest agent. Any solution ? I am using https://github.com/reportportal/agent-js-cypress [npm install @reportportal/agent-js-cypress] Locally it works fine but not from Git Pipeline. I also tried "@reportportal/agent-js-cypress": "https://github.com/reportportal/agent-js-cypress.git#master"

vishallanke commented 4 years ago

This issue still exists on latest release. From Git, test cases wont get uploaded to RP.

In previous version (1 month ago), I have observed that if all test cases gets passed, then test script and test cases gets successfully uploaded. If any of the test gets failed, then those test script as well as test cases(passed+failed from test script) wont get uploaded.

However, in this release, I found that irrespective of state of test , it wont get updated.

This issue wont occur if I execute test locally. It occurs only when you execute tests from Gitlab Pipeline.

vishallanke commented 4 years ago

image

Tested today on latest release. Report portal hosted on AWS EC2, latest version - STEPS followed are - https://reportportal.io/docs/Deploy-ReportPortal-on

CYPRESS GIT PIPELINE code that triggers test cases

  script:
    - $(npm bin)/print-env CI
    - npm run start:ci &
    - npx cypress run
    - $(npm bin)/print-env CI
    - npx cypress -v
    - echo Thanks

Package.json

 "dependencies": {
    "@bahmutov/print-env": "1.2.0",
    "@reportportal/agent-js-cypress": "^5.0.0",
    "glob": "7.1.3",
    "mocha": "7.0.1",
    "node-ipc": "9.1.1"
  },
  "license": "Apache-2.0",
  "devDependencies": {
    "cypress": "^4.0.2",
    "eslint": "6.5.1",
    "eslint-config-airbnb-base": "14.0.0",
    "eslint-config-prettier": "6.9.0",
    "eslint-plugin-cypress": "2.8.1",
    "eslint-plugin-import": "2.20.0",
    "eslint-plugin-jest": "^23.8.1",
    "eslint-plugin-prettier": "3.1.2",
    "jest": "^25.1.0",
    "mock-fs": "^4.11.0",
    "prettier": "1.19.1"
  },

Cypress.json

{
  "projectId" : "crossVertical",
  "reporter": "@reportportal/agent-js-cypress",
  "video" : false,
  "reporterOptions": {
      "endpoint": "http://dummy.compute.amazonaws.com:8080/api/v1",
      "token": "12345678dummy-b074-4a1b-8d72-8a5e55c55f8d",
      "launch": "cypress",
      "project": "cypress_project",
      "description": "END_TO_END_CROSS VERTICAL",
      "isLaunchMergeRequired": false,
      "autoMerge": false,
      "integrationFolder": "cypress/integration",
      "screenshotsFolder": "cypress/screenshots",
      "fixturesFolder": "cypress/fixtures",
      "supportFile": "cypress/support/index.js",
      "pluginsFile": "cypress/plugins/index.js"
  }
}

LOGS

 $ npm run start:ci &
 $ npx cypress run
 npm ERR! missing script: start:ci
 npm ERR! A complete log of this run can be found in:
 npm ERR!     /builds/Vishal.Lanke/cypress-report-portal-integration/.npm/_logs/2020-06-30T05_02_12_819Z-debug.log
 /builds/Vishal.Lanke/cypress-report-portal-integration/cache/Cypress/4.0.2/Cypress/Cypress: /lib/x86_64-linux-gnu/libdbus-1.so.3: no version information available (required by /builds/Vishal.Lanke/cypress-report-portal-integration/cache/Cypress/4.0.2/Cypress/Cypress)
 /builds/Vishal.Lanke/cypress-report-portal-integration/cache/Cypress/4.0.2/Cypress/Cypress: /lib/x86_64-linux-gnu/libdbus-1.so.3: no version information available (required by /builds/Vishal.Lanke/cypress-report-portal-integration/cache/Cypress/4.0.2/Cypress/Cypress)
 Fontconfig warning: "/etc/fonts/fonts.conf", line 86: unknown element "blank"
 /proc/self/exe: /lib/x86_64-linux-gnu/libdbus-1.so.3: no version information available (required by /proc/self/exe)
 /proc/self/exe: /lib/x86_64-linux-gnu/libdbus-1.so.3: no version information available (required by /proc/self/exe)
 /builds/Vishal.Lanke/cypress-report-portal-integration/cache/Cypress/4.0.2/Cypress/Cypress: /lib/x86_64-linux-gnu/libdbus-1.so.3: no version information available (required by /builds/Vishal.Lanke/cypress-report-portal-integration/cache/Cypress/4.0.2/Cypress/Cypress)
 ====================================================================================================
   (Run Starting)
   ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
   │ Cypress:    4.0.2                                                                              │
   │ Browser:    Electron 78 (headless)                                                             │
   │ Specs:      4 found (actions.spec.js, aliasing.spec.js, assertions.spec.js, example.spec.js)   │
   └────────────────────────────────────────────────────────────────────────────────────────────────┘
 ────────────────────────────────────────────────────────────────────────────────────────────────────

   Running:  actions.spec.js                                                                 (1 of 4)
 /proc/self/exe: /lib/x86_64-linux-gnu/libdbus-1.so.3: no version information available (required by /proc/self/exe)
 /builds/Vishal.Lanke/cypress-report-portal-integration/cache/Cypress/4.0.2/Cypress/Cypress: /lib/x86_64-linux-gnu/libdbus-1.so.3: no version information available (required by /builds/Vishal.Lanke/cypress-report-portal-integration/cache/Cypress/4.0.2/Cypress/Cypress)
 /proc/self/exe: /lib/x86_64-linux-gnu/libdbus-1.so.3: no version information available (required by /proc/self/exe)
 (node:305) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
 (node:305) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
   (Results)
   ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
   │ Tests:        14                                                                               │
   │ Passing:      14                                                                               │
   │ Failing:      0                                                                                │
   │ Pending:      0                                                                                │
   │ Skipped:      0                                                                                │
   │ Screenshots:  0                                                                                │
   │ Video:        false                                                                            │
   │ Duration:     17 seconds                                                                       │
   │ Spec Ran:     actions.spec.js                                                                  │
   └────────────────────────────────────────────────────────────────────────────────────────────────┘
 ────────────────────────────────────────────────────────────────────────────────────────────────────

   Running:  aliasing.spec.js                                                                (2 of 4)
 /proc/self/exe: /lib/x86_64-linux-gnu/libdbus-1.so.3: no version information available (required by /proc/self/exe)
 /builds/Vishal.Lanke/cypress-report-portal-integration/cache/Cypress/4.0.2/Cypress/Cypress: /lib/x86_64-linux-gnu/libdbus-1.so.3: no version information available (required by /builds/Vishal.Lanke/cypress-report-portal-integration/cache/Cypress/4.0.2/Cypress/Cypress)
 (node:480) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
 (node:480) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
 /proc/self/exe: /lib/x86_64-linux-gnu/libdbus-1.so.3: no version information available (required by /proc/self/exe)
   (Results)
   ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
   │ Tests:        2                                                                                │
   │ Passing:      2                                                                                │
   │ Failing:      0                                                                                │
   │ Pending:      0                                                                                │
   │ Skipped:      0                                                                                │
   │ Screenshots:  0                                                                                │
   │ Video:        false                                                                            │
   │ Duration:     3 seconds                                                                        │
   │ Spec Ran:     aliasing.spec.js                                                                 │
   └────────────────────────────────────────────────────────────────────────────────────────────────┘
 ────────────────────────────────────────────────────────────────────────────────────────────────────

   Running:  assertions.spec.js                                                              (3 of 4)
 /proc/self/exe: /lib/x86_64-linux-gnu/libdbus-1.so.3: no version information available (required by /proc/self/exe)
 /builds/Vishal.Lanke/cypress-report-portal-integration/cache/Cypress/4.0.2/Cypress/Cypress: /lib/x86_64-linux-gnu/libdbus-1.so.3: no version information available (required by /builds/Vishal.Lanke/cypress-report-portal-integration/cache/Cypress/4.0.2/Cypress/Cypress)
 (node:656) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
 (node:656) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
 /proc/self/exe: /lib/x86_64-linux-gnu/libdbus-1.so.3: no version information available (required by /proc/self/exe)
   (Results)
   ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
   │ Tests:        9                                                                                │
   │ Passing:      9                                                                                │
   │ Failing:      0                                                                                │
   │ Pending:      0                                                                                │
   │ Skipped:      0                                                                                │
   │ Screenshots:  0                                                                                │
   │ Video:        false                                                                            │
   │ Duration:     5 seconds                                                                        │
   │ Spec Ran:     assertions.spec.js                                                               │
   └────────────────────────────────────────────────────────────────────────────────────────────────┘
 ────────────────────────────────────────────────────────────────────────────────────────────────────

   Running:  example.spec.js                                                                 (4 of 4)
 /proc/self/exe: /lib/x86_64-linux-gnu/libdbus-1.so.3: no version information available (required by /proc/self/exe)
 /builds/Vishal.Lanke/cypress-report-portal-integration/cache/Cypress/4.0.2/Cypress/Cypress: /lib/x86_64-linux-gnu/libdbus-1.so.3: no version information available (required by /builds/Vishal.Lanke/cypress-report-portal-integration/cache/Cypress/4.0.2/Cypress/Cypress)
 (node:830) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
 (node:830) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
 /proc/self/exe: /lib/x86_64-linux-gnu/libdbus-1.so.3: no version information available (required by /proc/self/exe)
 /proc/self/exe: /lib/x86_64-linux-gnu/libdbus-1.so.3: no version information available (required by /proc/self/exe)
   (Results)
   ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
   │ Tests:        9                                                                                │
   │ Passing:      6                                                                                │
   │ Failing:      2                                                                                │
   │ Pending:      1                                                                                │
   │ Skipped:      0                                                                                │
   │ Screenshots:  4                                                                                │
   │ Video:        false                                                                            │
   │ Duration:     16 seconds                                                                       │
   │ Spec Ran:     example.spec.js                                                                  │
   └────────────────────────────────────────────────────────────────────────────────────────────────┘
   (Screenshots)
   -  /builds/Vishal.Lanke/cypress-report-portal-integration/cypress/screenshots/examp       (115x50)
      le.spec.js/Feature  Cypress Report Portal Integration Test -- SubFeature  Verify               
       Cypress Examples page -- SubFeature  Verify content Cypress example page -- Tes               
      t  Verify page should contain Commands.png                                                     
   -  /builds/Vishal.Lanke/cypress-report-portal-integration/cypress/screenshots/examp     (1280x720)
      le.spec.js/Feature  Cypress Report Portal Integration Test -- SubFeature  Verify               
       Cypress Examples page -- SubFeature  Verify content Cypress example page -- Tes               
      t  Verify page should contain gfkjdgkjdfgl (failed) (failed).png                               
   -  /builds/Vishal.Lanke/cypress-report-portal-integration/cypress/screenshots/examp      (500x200)
      le.spec.js/Feature  Cypress Report Portal Integration Test -- SubFeature  Verify               
       content of Google Dot Com -- Test  Verify contains of google page (failed).png                
   -  /builds/Vishal.Lanke/cypress-report-portal-integration/cypress/screenshots/examp     (1280x720)
      le.spec.js/Feature  Cypress Report Portal Integration Test -- SubFeature  Verify               
       content of Google Dot Com -- Test  Verify contains of google page (failed) (fai               
      led).png                                                                                       
 ====================================================================================================
   (Run Finished)
        Spec                                              Tests  Passing  Failing  Pending  Skipped  
   ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
   │ ✔  actions.spec.js                          00:17       14       14        -        -        - │
   ├────────────────────────────────────────────────────────────────────────────────────────────────┤
   │ ✔  aliasing.spec.js                         00:03        2        2        -        -        - │
   ├────────────────────────────────────────────────────────────────────────────────────────────────┤
   │ ✔  assertions.spec.js                       00:05        9        9        -        -        - │
   ├────────────────────────────────────────────────────────────────────────────────────────────────┤
   │ ✖  example.spec.js                          00:16        9        6        2        1        - │
   └────────────────────────────────────────────────────────────────────────────────────────────────┘
     ✖  1 of 4 failed (25%)                      00:41       34       31        2        1        -  
Uploading artifacts...
00:02
 cypress/screenshots/**/*.png: found 4 matching files 
 Uploading artifacts to coordinator... ok            id=27773739 responseStatus=201 Created token=4bkmyR8U
 ERROR: Job failed: exit code 1
ElenaRomanchuk commented 4 years ago

Hi @vishallanke , We will reproduce and investigate this issue in nearest future. Thanks for updates!

vishallanke commented 4 years ago

Hi @ElenaRomanchuk Is there any update on this defect ?

vishallanke commented 4 years ago

Our team is facing plenty of reliability issues

  1. Test cases stuck in between, never gets uploaded
  2. Test Status not getting updated
  3. No common pattern for any issues, sometimes its working, sometimes not working
vishallanke commented 4 years ago

Due to various reliability and stability issues in current Cypress Agent as specified above, we decide to use JUnit and Cypress together. https://github.com/vishallanke/ReportPortalCypressJUnitAgent

AmsterGet commented 3 years ago

Hello @vishallanke ! I started on this issue and am unable to reproduce it. Tested with GitHub Actions CI in this repo. All launches and their children have finished successfully and do not get stuck. The test run results are fully displayed in the Report Portal Demo instance. Some of the test cases failed, but everything looks fine. Could you provide any other details or conditions on this matter? Thank you in advance.