Re-running a build in Codeship does not generate new Dashboard run

[lifenstein]

Current behavior:

Cypress tests running on Codeship and recorded to the Dashboard with multiple pipelines (with cypress run --parallel) are skipped if the Codeship build is restarted. This means that even if there are failing Cypress tests in the first build run, the second test command (cypress run) returns a success code during the second build run. This results in the failing code to be deployed as per our CI logic.

Desired behavior:

Cypress dashboard should detect that the second run and re-run all the tests, returning an appropriate success/failure code.

Versions

Cypress v4.

Note

This is similar to #6450, except for Codeship. There are no known workarounds, and not even a way forward as of now, as per Cypress support. But I thought I'd add a bug ticket just to flag this as an ongoing issue.

[jennifer-shehane]

Hey @lifenstein, we are aware of a situation where this can happen during Codeship reruns.

Current behavior

If this is the same issue before, basically your output on rerun looks similar to this:

It looks like this is your first time using Cypress: 3.8.1

[15:37:20] Verifying Cypress can run /home/jane/.cache/Cypress/3.8.1/Cypress [started]

[15:37:22] Verifying Cypress can run /home/jane/.cache/Cypress/3.8.1/Cypress [completed]

====================================================================================================

(Run Starting)

┌────────────────────────────────────────────────────────────────────────────────────────────────┐

│ Cypress: 3.8.1 │

│ Browser: Electron 78 (headless) │

│ Specs: 1 found │

│ Params: Tag: false, Group: false, Parallel: true │

│ Run URL: https://dashboard.cypress.io/projects/abc123/runs/1 │

└────────────────────────────────────────────────────────────────────────────────────────────────┘

=========================================

Then the run just exits with Code 0 without running any tests.

Why

This issue here is basically that Codeship does not provide any environment variables that Cypress can use to identify a rerun from the first run. I investigated this with their available env vars below:

Nothing here to distinguish a rerun from a non rerun

CI=true
CI_BRANCH=codeship-updates
CI_BUILD_APPROVED=false
CI_BUILD_ID=1f9487c9-3dd6-493a-ac64-438196b615bf
CI_COMMITTER_EMAIL=jennifer@cypress.io
CI_COMMITTER_NAME=Jennifer Shehane
CI_COMMITTER_USERNAME=jennifer-shehane
CI_COMMIT_DESCRIPTION=v1.8.1-39-g104b
CI_COMMIT_ID=104bdd366c91ab79b8fdfb8a618a9607cc7f5078
CI_COMMIT_MESSAGE=trying to run codeship in parallel
CI_NAME=codeship
CI_PROJECT_ID=8d6a20c0-b70e-0133-41c6-56e5cd60fbd0
CI_PR_NUMBER=
CI_PULL_REQUEST=false
CI_REPO_NAME=cypress-example-kitchensink
CI_STRING_TIME=2020-01-07T06:06:32Z
CI_TIMESTAMP=1578377192

I contacted Codeship directly to make sure there were no other env vars to pull from

Unfortunately we don't have a feature to match exactly what you're looking for. When restarting builds it is meant only for very small changes the most common reason being when a 3rd party software messes up or has an incorrect setting. If you want to change actual variables then you'll need to trigger a new build with a commit or PR depending on your project.

Way to reproduce

Unfortunately - I tried a lot to reproduce it and was never able to pin down the configuration required in this case. But I don't doubt this is happening. We've seen this in other CI providers.

We run our kitchensink repo in Codeship, so you can see our setup there. If you can, please provide any insight on your setup.

• https://github.com/cypress-io/cypress-example-kitchensink

Workaround

There is a known workaround. Similar to the GH action issue, as a workaround, we suggest pushing an empty commit to trigger a new check with a different SHA.

The Fix?

Well, we don't have one unless Codeship exposes a unique environment variable we can use to identify a rerun from a regular run unfortunately.

---

Related to https://github.com/cypress-io/cypress-services/issues/1969 (private repo so not accessible)

[lifenstein]

@jennifer-shehane I don't have access to that Codeship project, unfortunately.

How we test

Our codeship setup runs scripts under https://github.com/uqlibrary/fez-frontend/tree/master/bin/ - this is a React project.

codeship-setup.sh runs during the setup stage of Codeship tests. We then have two pipelines; each exporting a different value for the variable PIPE_NUM and then running codeship-test.sh. Within the test file, we run one of two batches of tests based on PIPE_NUM.

Batch 1 runs Jest unit tests, after which it runs our Cypress tests. Batch 2 runs linting checks, after which it runs Cypress tests - this starts running Cypress first since unit tests take much longer.

We run the cypress command cypress run --record --parallel --config video=true after spinning up webpack-dev-server.

Responses to other things mentioned

1. Re: Codeship's response, what they're saying is that reruns can be used when someone has a wrong config in the first run, so they fix it and run again. But if the first run started Cypress tests, then the same issue would occur where the tests which ran (and failed) during the first run would be skipped in the second one!

2. The workaround of pushing a commit assumes that developers (especially new ones) are aware of this issue and remembering to do this instead of re-starting the Codeship build.... This is hard to enforce and unpredictable.

Possible fix

Is the "Run Completion Delay" working as intended? I had set it to 0 seconds (changed from 60) and the later to 1 second for fear of 0 being a special case.

This is how I expect it to work: If one or more of the parallel tests fail, and the rest finish running, then the test run would be considered complete, and if a new machine joins the queue, it would be treated as a new run. Is this how it actually works?

[lifenstein]

Forgot to mention another scenario where the issue crops up: Codeship runs can be stopped and restarted. In this case, Cypress waits for a very long time for new test results to be uploaded from Codeship scripts before timing out. This means that the "Run Completion Delay" isn't in effect since at least one of the machines will not have finished running.

It would be fantastic if there was a way to manually kill a run on the Cypress dashboard so that we could ensure that a rebuild would result in a new Cypress run.

[jennifer-shehane]

@lifenstein The rerun not working has nothing to do with the run completion delay. But, yes, runs are considered completed if no new cypress run calls are made to the Cypress API within the delay after the last run completed.

Reducing this will not fix the issue because the 'rerun' will still be regarded as the same run as it has the exact same generated ciBuildId.

The ciBuildId is the unique ID we use to distinguish runs. We auto generate it off of env vars sent from each CI. CodeShip though does not send us anything unique between an original run and rerun for us to generate a unique ciBuildId though.

So, theoretically, if you can find a way to pass a different ciBuildId (using --ci-build-id flag) to the 'rerun' that is different from the original run, then it would be recorded as a new run. Sorry I should have mentioned this previously.

Read more: https://on.cypress.io/parallelization#CI-Build-ID-environment-variables-by-provider

There is not currently a way to cancel a run from the Dashboard directly, but this is on our roadmap.

[lifenstein]

Thanks @jennifer-shehane, good to hear that the "stop run" feature is in the roadmap!

Re: changing variables during a re-run doesn't seem to be possible without turning off parallel execution, as the variable will need to be the same in both threads during a run. Without parallel execution, the tests take too long, so we'd need to stop running tests on commits and instead do it as a nightly or so. For now, we are still ramping up the Cypress tests, and might need to do that anyway at some point since we're limited to two pipelines on our plan.

Could you please clarify the run completion delay a bit further? When the last machine in the pool has finished/failed and the set run delay time has passed (which in our case is just 1s), any new requests from Codeship should start a new Cypress run, should it not? Do the post-run-completion-delay run (which should be in "completed" state) get restarted if a new request with the same build ID come through?

[jennifer-shehane]

Could you please clarify the run completion delay a bit further? When the last machine in the pool has finished/failed and the set run delay time has passed (which in our case is just 1s), any new requests from Codeship should start a new Cypress run, should it not? Do the post-run-completion-delay run (which should be in "completed" state) get restarted if a new request with the same build ID come through?

Any new requests from Codeship in the case of the runCompletionDelay having passed where the Codeship build has the same generated ciBuildId will error because the run has already been completed - we've already reported a status for that run.

A new run can only be created if it has a newly generated or passed ciBuildID.

Diagram and explanation of runCompletionDelay: https://on.cypress.io/parallelization#Run-completion-delay

[lifenstein]

Any new requests from Codeship in the case of the runCompletionDelay having passed where the Codeship build has the same generated ciBuildId will error because the run has already been completed - we've already reported a status for that run.

Perhaps this could be the fix! If there is either a toggle-able setting or commandline switch to allow a new build to be started if a machine tries to join with an already completed ciBuildID, then reruns would work as expected! Combine this with the ability to manually stop a run from the dashboard, and all the bases would be covered.

... OR, Codeship could actually provide a different identifier - because they do show previous runs in the UI, so there's clearly an identifier for different runs - they just need to expose it. I'll request this from them and mention this issue here.

[CypressCecelia]

@lifenstein Just an update, canceling a run from the Dashboard is now available: https://docs.cypress.io/guides/dashboard/runs.html#Cancel-run

[lifenstein]

Thanks @CypressCecelia, this is helpful when shutting down a build manually in CodeShip.

A rerun still needs to be trigger a new Cypress run. I did ask Codeship to add an environment variable or two to detect reruns from a script and to identify the rerun across pipelines, and they have accepted it as a feature request.

The switch I suggested in my last comment is still a possible fix for the issue, of course, now that manual stopping is available.

[CypressCecelia]

Thanks for the update on the Codeship side @lifenstein. We'll post updates to this issue as they occur.

[ayush]

@CypressCecelia any update on this?

[CypressCecelia]

Hi @ayush! I'm going to escalate this to our product team and get an update for you. Stay tuned!

[cypress-app-bot]

This issue has not had any activity in 180 days. Cypress evolves quickly and the reported behavior should be tested on the latest version of Cypress to verify the behavior is still occurring. It will be closed in 14 days if no updates are provided.

[cypress-app-bot]

This issue has been closed due to inactivity.