Percy builds not working

[andybalaam]

A few days ago I removed the PERCY_TOKEN from the Repository settings (after the Percy builds failed because of wrong token) and that has clearly not worked - I was hoping it would pick up the right token from the environment, but that didn't work:

https://github.com/matrix-org/matrix-react-sdk/actions/runs/6464714059/job/17550187611

Run npx -p @percy/cli percy build:finalize
npm WARN exec The following package was not found and will be installed: @percy/cli@1.27.3
[percy] Finalizing parallel build...
[percy] Error: Missing Percy token
Error: Process completed with exit code 1.

[andybalaam]

Reading the docs, it seems like we have the right stuff set up: we have a Cypress environment containing the secret, and we access it by making a variable in the env: section in our workflow. The step it set up to use the Cypress environment.

So investigating why it is not found.

[andybalaam]

It looks like (maybe - I'm not sure I understand) Environment secrets are not allowed in reusable workflows: https://github.com/actions/runner/issues/1490

This workflow is reusable, but as far as I can see, it's not actually being "called" in this case, but maybe the restriction still applies.

[andybalaam]

Trying a Repository secret

[andybalaam]

OK, that worked, in that a Percy build was created here: https://percy.io/dfde73bd/matrix-react-sdk/builds/30565506/failed

But it did report the error we were seeing before all this started:

Build failed

No snapshots were uploaded to this build, so there was nothing to compare. This could happen if your tests failed in your CI environment and nothing was uploaded to Percy.

This build can be safely ignored. It will not be used as a baseline for visual diffs. If these failures happen frequently, check your CI or client logs for failures.

I'm not sure whether this is because I re-kicked an old build. Trying to kick off a new one.

[andybalaam]

Deleting the environment secret PERCY_TOKEN since it does not appear to work.

[andybalaam]

I just noticed that the other secret, CYPRESS_RECORD_KEY is mentioned in the secrets: section of workflow_call:, so it is possible that if we listed PERCY_TOKEN in there too it would start working. Not investigating further since it seems unlikely: this run of the workflow was not due to a workflow_call event but a workflow_run, so it would be very strange if the workflow_call config affected it.

[andybalaam]

The re-run failed with the same "no snapshots" message: https://percy.io/dfde73bd/matrix-react-sdk/builds/30565683/

Time to investigate why.

[andybalaam]

Ah, it's because all the test runs succeeded with No specs executed: https://github.com/matrix-org/matrix-react-sdk/actions/runs/6464714059/job/17562398290

[andybalaam]

This URL looks wrong:

Connecting to cloud orchestration service...
🎥 Run URL: http://localhost:8080/run/162ab26c50c6ae801eb17174563c3f25

[andybalaam]

Seems like the CURRENTS_API_URL is not getting through somehow.

[andybalaam]

It is added as a variable in the Environment - trying it as a Repository variable.

[andybalaam]

OK, that was a blind alley - the Run URL is fine - that is not the CURRENTS_API_URL.

So some tests ran recently and they actually tested some things: https://github.com/matrix-org/matrix-react-sdk/actions/runs/6467934419/job/17558989088

So it's working, but that run didn't report to Percy so we can't see whether that is going to work.

[andybalaam]

Waiting for tomorrow morning to find out whether we will get Percy results now.

Meanwhile, trying to figure out why the Kiwi reporting is failing. So far, I know that we don't create an XML report on the tests to upload to Kiwi, even though we provide {"reporter":"cypress-multi-reporters", "reporterOptions": { "configFile": "cypress-ci-reporter-config.json" } } as an argument to cypress-cloud, and that config file contains:

{
    "reporterEnabled": "spec, mocha-junit-reporter",
    "mochaJunitReporterReporterOptions": {
        "mochaFile": "cypress/results/junit/results-[hash].xml",
        "useFullSuiteTitle": true
    }
}

The Upload reports log shows:

Run actions/upload-artifact@v3
Warning: No files were found with the provided path: matrix-react-sdk/cypress/results/junit. No artifacts will be uploaded.

[andybalaam]

When there are failures that were retried though, it does find files in that directory, and uploads and downloads them successfully.

[andybalaam]

When I turned on debug logging in the github action, I saw that the config option shown above was being overwritten, presumably somewhere inside the cypress-cloud script.

[andybalaam]

When run cypress-cloud locally, I see an XML report. I run it like this:

$ CURRENTS_API_URL=http://localhost:1234 CURRENTS_RECORD_KEY=y4 CURRENTS_PROJECT_ID=x DEBUG="cypress:*" npx cypress-cloud run --ci-build-id=4 --browser=chromium --spec cypress/e2e/read-receipts/redactions.spec.ts --config '{"reporter":"cypress-multi-reporters", "reporterOptions": { "configFile": "cypress-ci-reporter-config.json" } }'

$ ls -l cypress/results/junit/results-8887827be30d5ca46bc5c2128a5a7cba.xml 
-rw-rw-r-- 1 andy andy 4226 Oct 10 15:24 cypress/results/junit/results-8887827be30d5ca46bc5c2128a5a7cba.xml

[andybalaam]

So presumably the config we are providing in the github action is either garbled somehow, or overridden.

[andybalaam]

The overnight build ran and created a Percy build succesfully: https://github.com/matrix-org/matrix-react-sdk/actions/runs/6477961847/job/17589448684 so the PERCY_TOKEN problem is fixed.

However, it still says

Build failed

No snapshots were uploaded to this build, so there was nothing to compare. This could happen if your tests failed in your CI environment and nothing was uploaded to Percy.

[andybalaam]

Ah, the command config overrides command-prefix: https://github.com/cypress-io/github-action#custom-test-command

So we can't use both at the same time. I think this explains the lack of Percy results.

[andybalaam]

Hopeful fix for Percy not showing results: https://github.com/matrix-org/matrix-react-sdk/pull/11730

[t3chguy]

@andybalaam could you please undo your changes the Environment secrets, they are used for security isolation, given our need to use workflow_run a lot, repo secrets are significantly less protected.

[andybalaam]

(Respnding to above over DM and will update here)

[andybalaam]

(We concluded it would be worth trying the idea I toyed with of adding PERCY_TOKEN to workflow_call: secrets: in case it makes the environment secrets work)

[andybalaam]

I think the problem with the reporter is that the config we are supplying is Cypress config, but it's being consumed by the cypress-cloud exe and treated as Currents config.

[andybalaam]

Nope, the cypress-cloud docs say --config is for Cypress config, not its own config.

Trying out adding this config to the main Cypress config file.

[andybalaam]

We have Percy screenshots :tada: https://percy.io/dfde73bd/matrix-react-sdk/builds/30623219/unchanged/1691907360?browser=safari&viewLayout=overlay&viewMode=new&width=1920

[t3chguy]

@andybalaam should this be closed then?

[andybalaam]

Yeah, was keeping it for the Kiwi results, but I can track that separately if my latest change doesn't fix it.