search

cypress-io / github-action

GitHub Action for running Cypress end-to-end & component tests
https://on.cypress.io/guides/continuous-integration/github-actions
MIT License
1.37k stars 347 forks source link

clicking 'Re-run all jobs' in GHA kicks off workflow, but marks tests as passing with 'Run Finished' but tests are not triggered #510

Closed mmsjd closed 2 years ago

mmsjd commented 2 years ago

Hello!

I am seeing behavior similar to Tests not ran: Getting "Run finished" immediately when re-running tests #147 where clicking 'Re-run all jobs' in Github Actions kicks off a workflow, but marks all Cypress tests as passing with 'Run Finished' but tests are not triggered.

Initial run details:

Re-run details:

Additional notes: I'm not finding jobId in the logs and see that ciBuildId is either set to "" or '-'.

GHA re-run debug log:

====================================================================================================

  (Run Starting)

  cypress:server:run formatting Node version. { version: '16.14.0', path: '/__t/node/16.14.0/x64/bin/node' } +392ms
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        9.5.0                                                                          │
  │ Browser:        Chrome 98 (headless)                                                           │
  │ Node Version:   v16.14.0 (/__t/node/16.14.0/x64/bin/node)                                      │
  │ Specs:          17 found ...                                                   │
  │ Searched:       cypress/specs/regression/**/*                                                  │
  │ Params:         Tag: regression, Group: specs/regression, Parallel: true                       │
  │ Run URL:        https://dashboard.cypress.io/projects/hxd1hz/runs/2983                         │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

  cypress:server:plugins plugin event registered? { event: 'before:run', isRegistered: false } +817ms
  cypress:server:api request to url: POST https://api.cypress.io/runs/dbef3e1c-8870-4338-9f95-d2d5708dd66e/instances with params: {"body":{"spec":null,"groupId":"specs/regression-linux-Chrome-98-b481d0ac4e","machineId":"8158b68f-7d58-4058-88c6-43deebd81200","platform":{"osCpus":[{"model":"Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz","speed":2095,"times":{"user":60520,"nice":130,"sys":22730,"idle":186370,"irq":0}},{"model":"Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz","speed":2095,"times":{"user":64970,"nice":100,"sys":25400,"idle":185190,"irq":0}}],"osName":"linux","osMemory":{"free":145342464,"total":7284850688},"osVersion":"Debian - 10.11","browserName":"Chrome","browserVersion":"98.0.4758.102"}},"headers":{"x-route-version":"5","x-cypress-run-id":"dbef3e1c-8870-4338-9f95-d2d5708dd66e","x-cypress-request-attempt":0,"x-os-name":"linux","x-cypress-version":"9.5.0"}} and token: undefined +54ms
  cypress:network:agent addRequest called { isHttps: true, href: 'https://api.cypress.io/runs/dbef3e1c-8870-4338-9f95-d2d5708dd66e/instances' } +208ms
  cypress:network:agent got family { family: 4, href: 'https://api.cypress.io/runs/dbef3e1c-8870-4338-9f95-d2d5708dd66e/instances' } +0ms
  cypress:server:api response { spec: null, instanceId: null, totalInstances: 17, claimedInstances: 17, estimatedWallClockDuration: null } +6s
  cypress:server:run final results of all runs: { startedTestsAt: undefined, endedTestsAt: undefined, totalDuration: 0, totalSuites: 0, totalTests: 0, totalFailed: 0, totalPassed: 0, totalPending: 0, totalSkipped: 0, runs: [], browserPath: 'google-chrome', browserName: 'chrome', browserVersion: '98.0.4758.102', osName: 'linux', osVersion: 'Debian - 10.11', cypressVersion: '9.5.0', runUrl: 'https://dashboard.cypress.io/projects/hxd1hz/runs/2983', config: { configFile: '***/packages/apps/vue/cypress.json', env: { codeCoverageTasksRegistered: true, CACHE_FOLDER: '/root/.cache/Cypress', INSTALL_BINARY: 0, RECORD_KEY: '1287a...03971' }, testingType: 'e2e', version: '9.5.0', baseUrl: 'https://localhost:8080/', projectId: 'hxd1hz', chromeWebSecurity: false, integrationFolder: '***/packages/apps/vue/cypress/specs', defaultCommandTimeout: 10000, reporter: 'junit', viewportWidth: 1920, viewportHeight: 1080, retries: { runMode: 3, openMode: 0 }, numTestsKeptInMemory: 0, projectRoot: '***/packages/apps/vue', projectName: 'vue', rawJson: { baseUrl: 'https://localhost:8080/', projectId: 'hxd1hz', chromeWebSecurity: false, integrationFolder: 'cypress/specs', defaultCommandTimeout: 10000, reporter: 'junit', viewportWidth: 1920, viewportHeight: 1080, retries: [Object], numTestsKeptInMemory: 15, envFile: {}, projectRoot: '***/packages/apps/vue', projectName: 'vue' }, morgan: false, isTextTerminal: true, socketId: '6pxnb', report: true, browsers: [ [Object], [Object] ], animationDistanceThreshold: 5, blockHosts: null, clientCertificates: [], component: {}, componentFolder: '***/packages/apps/vue/cypress/component', downloadsFolder: '***/packages/apps/vue/cypress/downloads', e2e: {}, execTimeout: 60000, exit: true, experimentalFetchPolyfill: false, experimentalInteractiveRunEvents: false, experimentalSessionSupport: false, experimentalSourceRewriting: false, experimentalStudio: false, fileServerFolder: '***/packages/apps/vue', fixturesFolder: '***/packages/apps/vue/cypress/fixtures', ignoreTestFiles: '*.hot-update.js', includeShadowDom: false, keystrokeDelay: 0, modifyObstructiveCode: true, pageLoadTimeout: 60000, pluginsFile: '***/packages/apps/vue/cypress/plugins/index.js', port: 39899, redirectionLimit: 20, reporterOptions: null, requestTimeout: 5000, resolvedNodePath: '/__t/node/16.14.0/x64/bin/node', resolvedNodeVersion: '16.14.0', responseTimeout: 30000, screenshotOnRunFailure: true, screenshotsFolder: '***/packages/apps/vue/cypress/screenshots', slowTestThreshold: 10000, scrollBehavior: 'top', supportFile: '***/packages/apps/vue/cypress/support/index.js', supportFolder: '***/packages/apps/vue/cypress/support', taskTimeout: 60000, testFiles: '**/*.*', trashAssetsBeforeRuns: true, userAgent: null, video: true, videoCompression: 32, videosFolder: '***/packages/apps/vue/cypress/videos', videoUploadOnPasses: true, waitForAnimations: true, watchForFileChanges: false, autoOpen: false, clientRoute: '/__/', devServerPublicPathRoute: '/__cypress/src', hosts: null, isInteractive: true, namespace: '__cypress', reporterRoute: '/__cypress/reporter', socketIoCookie: '__socket.io', socketIoRoute: '/__socket.io', xhrRoute: '/xhrs/', cypressEnv: 'production', resolved: { animationDistanceThreshold: [Object], baseUrl: [Object], blockHosts: [Object], chromeWebSecurity: [Object], clientCertificates: [Object], component: [Object], componentFolder: [Object], defaultCommandTimeout: [Object], downloadsFolder: [Object], e2e: [Object], env: [Object], execTimeout: [Object], exit: [Object], experimentalFetchPolyfill: [Object], experimentalInteractiveRunEvents: [Object], experimentalSessionSupport: [Object], experimentalSourceRewriting: [Object], experimentalStudio: [Object], fileServerFolder: [Object], fixturesFolder: [Object], ignoreTestFiles: [Object], includeShadowDom: [Object], integrationFolder: [Object], keystrokeDelay: [Object], modifyObstructiveCode: [Object], numTestsKeptInMemory: [Object], pageLoadTimeout: [Object], pluginsFile: [Object], port: [Object], projectId: [Object], redirectionLimit: [Object], reporter: [Object], reporterOptions: [Object], requestTimeout: [Object], resolvedNodePath: [Object], resolvedNodeVersion: [Object], responseTimeout: [Object], retries: [Object], screenshotOnRunFailure: [Object], screenshotsFolder: [Object], slowTestThreshold: [Object], scrollBehavior: [Object], supportFile: [Object], supportFolder: [Object], taskTimeout: [Object], testFiles: [Object], trashAssetsBeforeRuns: [Object], userAgent: [Object], video: [Object], videoCompression: [Object], videosFolder: [Object], videoUploadOnPasses: [Object], viewportHeight: [Object], viewportWidth: [Object], waitForAnimations: [Object], watchForFileChanges: [Object], browsers: [Object], hosts: [Object], isInteractive: [Object], configFile: [Object], testingType: [Object], version: [Object] }, parentTestsFolder: '***/packages/apps/vue/cypress', parentTestsFolderDisplay: 'vue/cypress', scaffoldedFiles: [ [Object] ], proxyUrl: 'http://localhost:39899', browserUrl: 'https://localhost:8080/__/', reporterUrl: 'https://localhost:8080/__cypress/reporter', xhrUrl: '__cypress/xhrs/', proxyServer: 'http://localhost:39899', state: {} }, status: 'finished' } +6s
  cypress:server:plugins plugin event registered? { event: 'after:run', isRegistered: false } +6s

====================================================================================================

  (Run Finished)

We are using Cypress 9.5.0 with the following workflow file:

name: Cypress regression tests
    runs-on: ubuntu-20.04
    container: cypress/included:9.5.0
    needs: [init]
    if: ${{ github.ref == 'refs/heads/main' || github.event.pull_request.draft == false }}
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v2
      - uses: ./.github/actions/dependencies
      - name: cypress
        uses: cypress-io/github-action@v2
        with:
          install: false
          parallel: true
          record: true
          start: yarn start:cypress
          wait-on: 'https://localhost:8080'
          command: yarn test:regression
          browser: chrome
          project: packages/apps/vue
        env:
          DEBUG: 'cypress:*'
          NODE_TLS_REJECT_UNAUTHORIZED: 0
          CYPRESS_INSTALL_BINARY: 0
          CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
          PERCY_PARALLEL_TOTAL: 4
          PERCY_PARALLEL_NONCE: '${{ github.event_name }}-${{ github.sha }}'
          PERCY_TARGET_BRANCH: 'develop'
          PERCY_BRANCH: ${{ github.head_ref }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

yarn test:regression script details: cypress run --group specs/regression --spec 'cypress/specs/regression/**/*' --ci-build-id $GIT_COMMIT-$BUILD_NUMBER --config-file cypress.json --tag 'regression' --browser chrome --parallel --record"

We noticed 'Re-run all jobs' failing to trigger tests roughly about a month ago. I've been sporadically trying to debug the issue since, but am at a loss. 'Re-run all jobs' worked prior to this. Any help or insight on how to resolve this would be greatly appreciated!

robertguss commented 2 years ago

@mmsjd you mention that "'Re-run all jobs' worked prior to this" what do you mean by prior to this? Has this config file changed since then? What did things look like when you had everything working? I am trying to figure out what has changed to dry and debug your issue.

Also, you are using v2 of the GitHub Action, but the latest version is 3.0.2. Can you try and use the latest version and see if that helps?

uses: cypress-io/github-action@v3

Also, I don't suppose this repo is publicly available, is it? It would be great to take a closer look at things if possible.

mmsjd commented 2 years ago

Unfortunately, the repo is not public, but I have temporarily enabled access to Cypress dashboard runs. I have also tried a million different things trying to get 'Re-run all jobs' to work. Most recent change was switching our Cypress run command from using --config integrationFolder=path to using --group specs/regression --spec 'cypress/specs/regression/**/*' and completely gutting the GHA regression workflow file (removed 'mkdir nyc_output' step, 'publish test results step' and 'generate cypress coverage report' step) to try to isolate the issue. Trying cypress-io/github-action@v3 now and will report back.

mmsjd commented 2 years ago

No luck. cypress-io/github-action@v3 is also reporting back totalFailed: 0:

cypress:network:agent addRequest called { isHttps: true, href: 'https://api.cypress.io/runs' } +0ms
cypress:network:connect beginning getAddress { hostname: 'api.cypress.io', port: 443 } +177ms
cypress:network:connect got addresses { hostname: 'api.cypress.io', port: 443, addresses: [ { address: '172.67.69.12', family: 4 }, { address: '104.26.6.176', family: 4 }, { address: '104.26.7.176', family: 4 }, { address: '2606:4700:20::681a:7b0', family: 6 }, { address: '2606:4700:20::681a:6b0', family: 6 }, { address: '2606:4700:20::ac43:450c', family: 6 } ] } +12ms
cypress:network:agent got family { family: 4, href: 'https://api.cypress.io/runs' } +16ms
cypress:server:api response { runId: '31539fc4-a27e-4fef-932e-8825a2aa48dc', groupId: 'specs/regression-linux-Chrome-98-b481d0ac4e', machineId: '62a0aa87-8c6b-44b0-b925-7c3142498552', runUrl: 'https://dashboard.cypress.io/projects/hxd1hz/runs/2985', warnings: [], tags: [ 'regression' ] } +102ms

====================================================================================================

(Run Starting)

cypress:server:run formatting Node version. { version: '16.14.0', path: '/__t/node/16.14.0/x64/bin/node' } +255ms
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress:        9.5.0                                                                          │
│ Browser:        Chrome 98 (headless)                                                           │
│ Node Version:   v16.14.0 (/__t/node/16.14.0/x64/bin/node)                                      │
│ Specs:          17 found...                                                    │
│ Searched:       cypress/specs/regression/**/*                                                  │
│ Params:         Tag: regression, Group: specs/regression, Parallel: true                       │
│ Run URL:        https://dashboard.cypress.io/projects/hxd1hz/runs/2985                         │
└────────────────────────────────────────────────────────────────────────────────────────────────┘

cypress:server:plugins plugin event registered? { event: 'before:run', isRegistered: false } +721ms
cypress:server:api request to url: POST https://api.cypress.io/runs/31539fc4-a27e-4fef-932e-8825a2aa48dc/instances with params: {"body":{"spec":null,"groupId":"specs/regression-linux-Chrome-98-b481d0ac4e","machineId":"62a0aa87-8c6b-44b0-b925-7c3142498552","platform":{"osCpus":[{"model":"Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz","speed":2593,"times":{"user":53870,"nice":0,"sys":21110,"idle":326180,"irq":0}},{"model":"Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz","speed":2593,"times":{"user":55200,"nice":0,"sys":21090,"idle":331390,"irq":0}}],"osName":"linux","osMemory":{"free":138764288,"total":7284846592},"osVersion":"Debian - 10.11","browserName":"Chrome","browserVersion":"98.0.4758.102"}},"headers":{"x-route-version":"5","x-cypress-run-id":"31539fc4-a27e-4fef-932e-8825a2aa48dc","x-cypress-request-attempt":0,"x-os-name":"linux","x-cypress-version":"9.5.0"}} and token: undefined +47ms
cypress:network:agent addRequest called { isHttps: true, href: 'https://api.cypress.io/runs/31539fc4-a27e-4fef-932e-8825a2aa48dc/instances' } +130ms
cypress:network:agent got family { family: 4, href: 'https://api.cypress.io/runs/31539fc4-a27e-4fef-932e-8825a2aa48dc/instances' } +0ms
cypress:server:api response { spec: null, instanceId: null, totalInstances: 17, claimedInstances: 17, estimatedWallClockDuration: null } +41ms
cypress:server:run final results of all runs: { startedTestsAt: undefined, endedTestsAt: undefined, totalDuration: 0, totalSuites: 0, totalTests: 0, totalFailed: 0, totalPassed: 0, totalPending: 0, totalSkipped: 0, runs: [], browserPath: 'google-chrome', browserName: 'chrome', browserVersion: '98.0.4758.102', osName: 'linux', osVersion: 'Debian - 10.11', cypressVersion: '9.5.0', runUrl: 'https://dashboard.cypress.io/projects/hxd1hz/runs/2985', config: { configFile: '***/packages/apps/vue/cypress.json', env: { codeCoverageTasksRegistered: true, CACHE_FOLDER: '/root/.cache/Cypress', INSTALL_BINARY: 0, RECORD_KEY: '1287a...03971' }, testingType: 'e2e', version: '9.5.0', baseUrl: 'https://localhost:8080/', projectId: 'hxd1hz', chromeWebSecurity: false, integrationFolder: '***/packages/apps/vue/cypress/specs', defaultCommandTimeout: 10000, reporter: 'junit', viewportWidth: 1920, viewportHeight: 1080, retries: { runMode: 3, openMode: 0 }, numTestsKeptInMemory: 0, projectRoot: '***/packages/apps/vue', projectName: 'vue', rawJson: { baseUrl: 'https://localhost:8080/', projectId: 'hxd1hz', chromeWebSecurity: false, integrationFolder: 'cypress/specs', defaultCommandTimeout: 10000, reporter: 'junit', viewportWidth: 1920, viewportHeight: 1080, retries: [Object], numTestsKeptInMemory: 15, envFile: {}, projectRoot: '***/packages/apps/vue', projectName: 'vue' }, morgan: false, isTextTerminal: true, socketId: 'r7vti', report: true, browsers: [ [Object], [Object] ], animationDistanceThreshold: 5, blockHosts: null, clientCertificates: [], component: {}, componentFolder: '***/packages/apps/vue/cypress/component', downloadsFolder: '***/packages/apps/vue/cypress/downloads', e2e: {}, execTimeout: 60000, exit: true, experimentalFetchPolyfill: false, experimentalInteractiveRunEvents: false, experimentalSessionSupport: false, experimentalSourceRewriting: false, experimentalStudio: false, fileServerFolder: '***/packages/apps/vue', fixturesFolder: '***/packages/apps/vue/cypress/fixtures', ignoreTestFiles: '*.hot-update.js', includeShadowDom: false, keystrokeDelay: 0, modifyObstructiveCode: true, pageLoadTimeout: 60000, pluginsFile: '***/packages/apps/vue/cypress/plugins/index.js', port: 45145, redirectionLimit: 20, reporterOptions: null, requestTimeout: 5000, resolvedNodePath: '/__t/node/16.14.0/x64/bin/node', resolvedNodeVersion: '16.14.0', responseTimeout: 30000, screenshotOnRunFailure: true, screenshotsFolder: '***/packages/apps/vue/cypress/screenshots', slowTestThreshold: 10000, scrollBehavior: 'top', supportFile: '***/packages/apps/vue/cypress/support/index.js', supportFolder: '***/packages/apps/vue/cypress/support', taskTimeout: 60000, testFiles: '**/*.*', trashAssetsBeforeRuns: true, userAgent: null, video: true, videoCompression: 32, videosFolder: '***/packages/apps/vue/cypress/videos', videoUploadOnPasses: true, waitForAnimations: true, watchForFileChanges: false, autoOpen: false, clientRoute: '/__/', devServerPublicPathRoute: '/__cypress/src', hosts: null, isInteractive: true, namespace: '__cypress', reporterRoute: '/__cypress/reporter', socketIoCookie: '__socket.io', socketIoRoute: '/__socket.io', xhrRoute: '/xhrs/', cypressEnv: 'production', resolved: { animationDistanceThreshold: [Object], baseUrl: [Object], blockHosts: [Object], chromeWebSecurity: [Object], clientCertificates: [Object], component: [Object], componentFolder: [Object], defaultCommandTimeout: [Object], downloadsFolder: [Object], e2e: [Object], env: [Object], execTimeout: [Object], exit: [Object], experimentalFetchPolyfill: [Object], experimentalInteractiveRunEvents: [Object], experimentalSessionSupport: [Object], experimentalSourceRewriting: [Object], experimentalStudio: [Object], fileServerFolder: [Object], fixturesFolder: [Object], ignoreTestFiles: [Object], includeShadowDom: [Object], integrationFolder: [Object], keystrokeDelay: [Object], modifyObstructiveCode: [Object], numTestsKeptInMemory: [Object], pageLoadTimeout: [Object], pluginsFile: [Object], port: [Object], projectId: [Object], redirectionLimit: [Object], reporter: [Object], reporterOptions: [Object], requestTimeout: [Object], resolvedNodePath: [Object], resolvedNodeVersion: [Object], responseTimeout: [Object], retries: [Object], screenshotOnRunFailure: [Object], screenshotsFolder: [Object], slowTestThreshold: [Object], scrollBehavior: [Object], supportFile: [Object], supportFolder: [Object], taskTimeout: [Object], testFiles: [Object], trashAssetsBeforeRuns: [Object], userAgent: [Object], video: [Object], videoCompression: [Object], videosFolder: [Object], videoUploadOnPasses: [Object], viewportHeight: [Object], viewportWidth: [Object], waitForAnimations: [Object], watchForFileChanges: [Object], browsers: [Object], hosts: [Object], isInteractive: [Object], configFile: [Object], testingType: [Object], version: [Object] }, parentTestsFolder: '***/packages/apps/vue/cypress', parentTestsFolderDisplay: 'vue/cypress', scaffoldedFiles: [ [Object] ], proxyUrl: 'http://localhost:45145', browserUrl: 'https://localhost:8080/__/', reporterUrl: 'https://localhost:8080/__cypress/reporter', xhrUrl: '__cypress/xhrs/', proxyServer: 'http://localhost:45145', state: {} }, status: 'finished' } +45ms
cypress:server:plugins plugin event registered? { event: 'after:run', isRegistered: false } +48ms

====================================================================================================

(Run Finished)
cypress:server:cypress about to exit with code 0 +2s
cypress:server:browsers browsers.kill called with no active instance +372ms
cypress:proxy:http:util:prerequests metrics: { browserPreRequestsReceived: 0, proxyRequestsReceived: 0, immediatelyMatchedRequests: 0, eventuallyReceivedPreRequest: [], neverReceivedPreRequest: [] } +0ms
2022-02-22T16:49:57.237Z cypress:cli child event fired { event: 'exit', code: 0, signal: null }
2022-02-22T16:49:57.238Z cypress:cli child event fired { event: 'close', code: 0, signal: null }
2022-02-22T16:49:57.239Z cypress:cli Stopping Xvfb
Done in 5.17s.
robertguss commented 2 years ago

I will look further into this. In the meantime you can disable publicly sharing your Cypress runs.

robertguss commented 2 years ago

can you turn on debugging and paste in the output here for me to take a look at it? https://github.com/cypress-io/github-action#debugging

There is also some additional information in the older issue you linked originally: https://github.com/cypress-io/github-action/issues/147

mmsjd commented 2 years ago

@robertguss thank you so much for looking into this! is there a process in particular you are looking for? i'd like to narrow the scope as the debug log is roughly 28k lines.

robertguss commented 2 years ago

oh my. I am not looking for the Cypress debug log , 

env:
  DEBUG: 'cypress:*'

I am wanting to see the debug log from the action itself

- name: Cypress tests with debug logs
  uses: cypress-io/github-action@v2
  env:
    DEBUG: '@cypress/github-action'

like we have it in the readme and example here: https://github.com/cypress-io/github-action#debugging

feel free to attach it as a file instead of pasting it all in a comment.

mmsjd commented 2 years ago

Here you go. this.txt

robertguss commented 2 years ago

@mmsjd looking through the logs now. I am going to try and see if I can recreate this same behavior on my end and will keep you posted. I am not quite sure what is going just yet. Thank you for your help and patience. I will keep you posted.

robertguss commented 2 years ago

@mmsjd can you do me a favor and modify your test:regression script:

cypress run --group specs/regression --spec 'cypress/specs/regression/**/*' --config-file cypress.json --tag 'regression' --browser chrome --parallel --record

Can you please remove the --ci-build-id $GIT_COMMIT-$BUILD_NUMBER from your script. You mentioned this is returning an empty string or -.

Let me know the results.

Also, within this script you are passing in a lot of flags, which you also have enabled via the GHA workflow config which does this for you. I don't think this would cause your issue, but we can clean that up later. For example, in the your script you have --parallel --record but this is handled by doing this. 

  with:
    record: true
    parallel: true

So a lot of this is redundant, but again I don't think is the cause of your issue. just wanted to point that out.

robertguss commented 2 years ago

@mmsjd I was able to recreate the issue in this PR: https://github.com/cypress-io/cypress-realworld-app/pull/1200

I am trying some things to see if I can fix it.

mmsjd commented 2 years ago

Good morning!

Reducing test:regression down to cypress run --group specs/regression --spec 'cypress/specs/regression/**/*' --config-file cypress.json --tag 'regression' resulted in errors:

This project has been configured to record runs on our Dashboard.

It currently has the projectId: hxd1hz

You also provided your Record Key, but you did not pass the --record flag.

This run will not be recorded.

If you meant to have this run recorded please additionally pass this flag:

  $ cypress run --record

If you don't want to record these runs, you can silence this warning:

  $ cypress run --record false

https://on.cypress.io/recording-project-runs
You passed the --ci-build-id, --group, --tag, or --parallel flag without also passing the --record flag.

The --group flag you passed was: specs/regression

These flags can only be used when recording to the Cypress Dashboard service.

https://on.cypress.io/record-params-without-recording
error Command failed with exit code 1.
robertguss commented 2 years ago

@mmsjd I was able to solve the issue by removing the --ci-builid from the test:regression script, like I mentioned in an earlier comment. So hopefully this will solve it for you. Please let me know if works for you.

robertguss commented 2 years ago

Good morning!

Reducing test:regression down to cypress run --group specs/regression --spec 'cypress/specs/regression/**/*' --config-file cypress.json --tag 'regression' resulted in errors:

This project has been configured to record runs on our Dashboard.

It currently has the projectId: hxd1hz

You also provided your Record Key, but you did not pass the --record flag.

This run will not be recorded.

If you meant to have this run recorded please additionally pass this flag:

  $ cypress run --record

If you don't want to record these runs, you can silence this warning:

  $ cypress run --record false

https://on.cypress.io/recording-project-runs
You passed the --ci-build-id, --group, --tag, or --parallel flag without also passing the --record flag.

The --group flag you passed was: specs/regression

These flags can only be used when recording to the Cypress Dashboard service.

https://on.cypress.io/record-params-without-recording
error Command failed with exit code 1.

ok sorry to confuse you, but update your test:regression script to be this.

cypress run --group specs/regression --spec 'cypress/specs/regression/**/*' --config-file cypress.json --tag 'regression' --browser chrome --parallel --record

mmsjd commented 2 years ago

Sans --ci-build-id $GIT_COMMIT-$BUILD_NUMBER still results in passing on re-run. 

Using custom test command: yarn test:regression
run tests command "yarn test:regression"
current working directory "/__w/***/***"
2022-02-23T16:04:02.627Z @cypress/github-action parsed command: yarn test:regression
2022-02-23T16:04:02.628Z @cypress/github-action found command "/usr/local/bin/yarn"
2022-02-23T16:04:02.628Z @cypress/github-action with arguments test:regression
2022-02-23T16:04:02.628Z @cypress/github-action running "/usr/local/bin/yarn" test:regression in /__w/***/***
2022-02-23T16:04:02.628Z @cypress/github-action waiting for the command to finish? true
/usr/local/bin/yarn test:regression
yarn run v1.22.17
$ yarn workspace ***/apps.vue test:regression
$ cypress run --group specs/regression --spec 'cypress/specs/regression/**/*' --config-file cypress.json --tag 'regression' --browser chrome --parallel --record
[388:0223/160405.486423:ERROR:bus.cc(392)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
[388:0223/160405.489600:ERROR:bus.cc(392)] Failed to connect to the bus: Address does not contain a colon
[388:0223/160405.489628:ERROR:bus.cc(392)] Failed to connect to the bus: Address does not contain a colon
[548:0223/160405.521220:ERROR:gpu_init.cc(453)] Passthrough is not supported, GL is swiftshader, ANGLE is 

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        9.5.0                                                                          │
  │ Browser:        Chrome 98 (headless)                                                           │
  │ Node Version:   v16.14.0 (/__t/node/16.14.0/x64/bin/node)                                      │
  │ Specs:          17 found (regression/alerts_spec.js, regression/blast_cell_active_spec.js, reg │
  │                 ression/blast_cell_available_spec.js, regression/blast_cell_schedule_spec.js,  │
  │                 regression/blast_cell_setup_spec.js, regression/blast_energy_rates_spec.js, re │
  │                 gression/blast_navigati...)                                                    │
  │ Searched:       cypress/specs/regression/**/*                                                  │
  │ Params:         Tag: regression, Group: specs/regression, Parallel: true                       │
  │ Run URL:        https://dashboard.cypress.io/projects/hxd1hz/runs/3001                         │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

====================================================================================================

  (Run Finished)
Done in 4.67s.
2022-02-23T16:04:07.450Z @cypress/github-action all done, exiting
robertguss commented 2 years ago

oh boy. in my PR in our Real World App I was able to recreate your issue and then by removing that flag, The re-runs work as expected.

check this out: https://github.com/cypress-io/cypress-realworld-app/pull/1200

mmsjd commented 2 years ago

oh boy. in my PR in our Real World App I was able to recreate your issue and then by removing that flag, The re-runs work as expected.

check this out: cypress-io/cypress-realworld-app#1200

I only see initial run attempt. I don't see dropdown to toggle between initial run vs re-run. Assuming because I don't have the right permissions? this Is there a run before/after https://dashboard.cypress.io/projects/***/runs/518?

robertguss commented 2 years ago

I am on this link: https://github.com/cypress-io/cypress-realworld-app/runs/5305766039?check_suite_focus=true Screen Shot 2022-02-23 at 11 19 51 AM

re-run all jobs

mmsjd commented 2 years ago

For https://github.com/cypress-io/cypress-realworld-app/runs/5306703582?check_suite_focus=true#step:5:117 run-attempt #2 is also showing Run URL: https://dashboard.cypress.io/projects/***/runs/518 (same runUrl as initial)

robertguss commented 2 years ago

@mmsjd still looking into this and will be in touch when I have an update

robertguss commented 2 years ago

@mmsjd is there any way you can create a public repo, that recreates the issue you are seeing? It is really hard to debug this without actually seeing your code and the problem firsthand. Can you create a simple sample project, with your GitHub Action config so I can see this issue firsthand?

mmsjd commented 2 years ago

Hello! I posted the entire Cypress portion of our Github Actions yaml here. For https://github.com/cypress-io/cypress-realworld-app you need to setup one of your tests to always fail then attempt re-run. You will find the runUrl in the initial test run attempt and all subsequent runs share the same runUrl. This should not be the case. There should be a new runUrl if I trigger a re-run in GHA.

robertguss commented 2 years ago

this is helpful. thank you.

BioCarmen commented 2 years ago

i am having the same issue.

mmsjd commented 2 years ago

Good evening! I just wanted to follow up. Please let me know if there's anything I can possibly provide to help assist in debugging. Thank you!

robertguss commented 2 years ago

Sorry I haven't been around the past couple of days, busy with work. I will try and look into this again today and keep you posted.

robertguss commented 2 years ago

@mmsjd have you tried this? https://github.com/cypress-io/github-action#custom-build-id

mmsjd commented 2 years ago

Hi! No, I have not. I can try. Will report back (likely early next week).

mmsjd commented 2 years ago

Hello! Happy to report using cutom-build-id solution worked. Thank you so much for all your help.

robertguss commented 2 years ago

@mmsjd awesome! glad it worked and my pleasure.

ahsanfarooq3423 commented 2 years ago

For anyone who is looking to add a new custom-build-id: https://medium.com/@blakenewman/hey-no-problem-actually-github-has-made-an-alternative-way-to-achieve-this-possible-and-is-a-bit-e08bd574f6c6

eoin-roll-out commented 1 month ago

Hi - this doesn't work if you're using multiple containers. It just runs the test suite multiple times as they all have a different id, do we have a solution for that?

MikeMcC399 commented 1 month ago

@eoin-roll-out

You've replied to an old closed issue, so I'm not sure what you are referring to exactly when you say it doesn't work.