chromaui / chromatic-cli

Chromatic CLI: `npx chromatic`
https://www.chromatic.com/docs/cli
MIT License
289 stars 70 forks source link

Chromatic with Cypress archive Error #1080

Closed rambod-rahmani closed 1 month ago

rambod-rahmani commented 1 month ago

Bug report

Issue Description:

I'm integrating Chromatic with Cypress for end-to-end testing on a sample Vue.js project. The Storybook builds and Chromatic integration work as expected without Cypress. I'm able to block my PRs via GitHub Actions until changes are reviewed and accepted on Chromatic's web interface. However, when trying to run Cypress tests along with Chromatic using:

ELECTRON_EXTRA_LAUNCH_ARGS=--remote-debugging-port=9222 pnpm cypress run

and successfully generating chromatic-archives, I run into the following error while executing:

npx chromatic --cypress --project-token=${{ secrets.CHROMATIC_PROJECT_TOKEN }}

Error Logs:

Here are the relevant logs from the command:

rr@Rambods-MacBook-Pro cypress-test % npx chromatic --cypress --force-rebuild --project-token=chpt_8425323d910bbc6

Chromatic CLI v11.10.4
https://www.chromatic.com/docs/cli

  ✔ Authenticated with Chromatic
    → Using project token '****************bbc6'
  ✔ Retrieved git information
    → Commit '4362f44' on branch 'feat/storybook'; found 1 parent build; TurboSnap disabled
  ✔ Collected Storybook metadata
    → Storybook 8.3.5 for Vue3; using the @storybook/vue3-vite builder (8.3.5); supported addons found: Links, Essentials, Actions, Interactions
  ✔ Initialized build
    → Build 97 initialized
  ✖ Building your Storybook
    → Command failed: node /Users/rr/DevOps/cypress-test/node_modules/.pnpm/@chromatic-com+cypress@0.8.0_esbuild@0.23.1_typescript@5.6.2_webpack-sources@3.2.3/node_modules/@chromatic
…
    Publish your built Storybook
    Verify your Storybook
    Test your stories

⚠ TurboSnap disabled due to rebuild
You appear to be rerunning an earlier build, because the baseline build has the same commit and branch name.
Comparing against the same commit would yield zero changed files, so we would end up running a build with no snapshots.
That's probably not what you want when rerunning a build, so we're just going to run a full build instead.

  ✖ Failed to run `chromatic --cypress`:

  Command failed with exit code 1: node /Users/rr/DevOps/cypress-test/node_modules/.pnpm/@chromatic-com+cypress@0.8.0_esbuild@0.23.1_typescript@5.6.2_webpack-sources@3.2.3/node_modules/@chromatic-com/cypress/dist/bin/build-archive-storybook.js --output-dir=/var/folders/q6/pl8jqmxd2jq14np3zp4hy1sm0000gn/T/chromatic--91774-tq3uA3q1goTN
/Users/rr/DevOps/cypress-test/node_modules/.pnpm/@chromatic-com+cypress@0.8.0_esbuild@0.23.1_typescript@5.6.2_webpack-sources@3.2.3/node_modules/@chromatic-com/cypress/dist/bin/build-archive-storybook.js:15
Please make sure that you have run your E2E tests, or have set the CHROMATIC_ARCHIVE_LOCATION env var if the output directory for the tests is not in the standard location.`)}i(d,"h");o(d,"checkArchivesDirExists");function O(e){fs.existsSync(e)||fs.mkdirSync(e,{recursive:!0});}i(O,"$");o(O,"ensureDir");async function T(e,r){return O(c__default.default.dirname(e)),promises.writeFile(e,r,{mode:511})}i(T,"C");o(T,"outputFile");async function A(e,r){return T(e,JSON.stringify(r))}i(A,"w");o(A,"outputJSONFile");async function C(e){let r=await promises.readFile(e);return JSON.parse(r.toString())}i(C,"x");o(C,"readJSONFile");function p(e){return `w${e.width}h${e.height}`}i(p,"a");o(p,"viewportToString");function D(e){let r=e.match(/w(\d+)h(\d+)/);return {width:Number(r[1]),height:Number(r[2])}}i(D,"N");o(D,"parseViewport");var X="snapshot.json";function x(e){let r=e.split(".");return r.slice(0,r.length-3).join(".")}i(x,"F");o(x,"snapshotIdFromFileName");function E(e){let r=e.split("."),t=r[r.length-3];return D(t)}i(E,"S");o(E,"viewportFromFileName");async function N(e){return (await promises.readdir(e)).filter(r=>r.endsWith(`.${X}`))}i(N,"v");o(N,"listSnapshotFiles");var B="stories.json";function _(e,r){let{stories:t}=e,n=t.map(s=>{let h=s.parameters.server.id,a=r[h];return {...s,parameters:{...s.parameters,chromatic:{...s.parameters.chromatic,modes:V(a)},viewport:{viewports:P(a),defaultViewport:p($(a))}}}});return {...e,stories:n}}i(_,"g");o(_,"addViewportsToStories");async function I(e){return (await promises.readdir(e)).filter(r=>r.endsWith(`.${B}`))}i(I,"E");o(I,"listStoriesFiles");function V(e){return e.reduce((r,t)=>{let n=p(t);return r[n]={viewport:n},r},{})}i(V,"X");o(V,"buildStoryModesConfig");function P(e){return e.reduce((r,t)=>{let n=p(t);return r[n]={name:n,styles:{width:`${t.width}px`,height:`${t.height}px`}},r},{})}i(P,"W");o(P,"buildStoryViewportsConfig");function $(e){let r=o((t,n)=>t.width<n.width?1:t.width>n.width?-1:0,"compareFn");return e.sort(r)[0]}i($,"z");o($,"findDefaultViewport");async function f(e){let r=k(e),t=await N(r),n=q(t),s=l(e),h=(await I(s)).map(a=>c__default.default.resolve(s,a));await Promise.all(h.map(async a=>{let j=await C(a),L=_(j,n);await A(a,L);}));}i(f,"l");o(f,"addViewportsToStoriesFiles");function q(e){let r={};return e.forEach(t=>{let n=x(t),s=r[n]||[];s.push(E(t)),r[n]=s;}),r}i(q,"G");o(q,"buildSnapshotViewportsLookup");function K(e,r,t){d(t),f(t).then(()=>{child_process.execFileSync("node",[v(),"dev",...e,"-c",r],{stdio:"inherit"});});}i(K,"Lt");o(K,"archiveStorybook");function m(e,r,t){d(t),f(t).then(()=>{child_process.execFileSync("node",[v(),"build",...e,"-c",r],{stdio:"inherit"});});}i(m,"kt");o(m,"buildArchiveStorybook");function v(){let e=w("storybook/package.json");return c.resolve(c.dirname(w.resolve("storybook/package.json")),e.bin.storybook)}i(v,"I");o(v,"binPath");var R="cypress/downloads";var Y=process.argv.slice(2),Z=c__default.default.resolve(__dirname,"../storybook-config");try{m(Y,Z,R);}catch(e){console.error(e.message),process.exitCode=1;}
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ^

TypeError: Cannot read properties of null (reading '1')
at parseViewport (/Users/rr/DevOps/cypress-test/node_modules/.pnpm/@chromatic-com+cypress@0.8.0_esbuild@0.23.1_typescript@5.6.2_webpack-sources@3.2.3/node_modules/@chromatic-com/cypress/dist/bin/build-archive-storybook.js:15:772)
at viewportFromFileName (/Users/rr/DevOps/cypress-test/node_modules/.pnpm/@chromatic-com+cypress@0.8.0_esbuild@0.23.1_typescript@5.6.2_webpack-sources@3.2.3/node_modules/@chromatic-com/cypress/dist/bin/build-archive-storybook.js:15:1017)
at /Users/rr/DevOps/cypress-test/node_modules/.pnpm/@chromatic-com+cypress@0.8.0_esbuild@0.23.1_typescript@5.6.2_webpack-sources@3.2.3/node_modules/@chromatic-com/cypress/dist/bin/build-archive-storybook.js:15:2341
at Array.forEach (<anonymous>)
at buildSnapshotViewportsLookup (/Users/rr/DevOps/cypress-test/node_modules/.pnpm/@chromatic-com+cypress@0.8.0_esbuild@0.23.1_typescript@5.6.2_webpack-sources@3.2.3/node_modules/@chromatic-com/cypress/dist/bin/build-archive-storybook.js:15:2300)
at addViewportsToStoriesFiles (/Users/rr/DevOps/cypress-test/node_modules/.pnpm/@chromatic-com+cypress@0.8.0_esbuild@0.23.1_typescript@5.6.2_webpack-sources@3.2.3/node_modules/@chromatic-com/cypress/dist/bin/build-archive-storybook.js:15:2080)

Node.js v20.13.1

I’ve ensured that my chromatic-archives folder is being generated correctly:

ls -l cypress/downloads/chromatic-archives
total 24
drwxr-xr-x 7 rr staff 224 Oct 4 10:15 archive
-rw-r--r-- 1 rr staff 188 Oct 4 10:15 find-elements-using-css-selectors.stories.json
-rw-r--r-- 1 rr staff 172 Oct 4 10:15 find-elements-using-xpath.stories.json
-rw-r--r-- 1 rr staff 134 Oct 4 10:15 passes.stories.json

Without the --cypress option, everything works as expected:

npx chromatic --force-rebuild --project-token=chpt_8425323d910bbc6

The Storybook is built and published, and visual changes are reviewed via the Chromatic web interface as usual.

Local Debugging:

I have also followed your suggestion to run the following commands:

npm run build-storybook
npx http-server ./storybook-static

The Webpage is blank.

rr@Rambods-MacBook-Pro cypress-test % npx http-server ./storybook-static                                          
Starting up http-server, serving ./storybook-static

http-server version: 14.1.1

http-server settings: 
CORS: disabled
Cache: 3600 seconds
Connection Timeout: 120 seconds
Directory Listings: visible
AutoIndex: visible
Serve GZIP Files: false
Serve Brotli Files: false
Default File Extension: none

Available on:
  http://127.0.0.1:8081
  http://192.168.1.6:8081
Hit CTRL-C to stop the server

[2024-10-04T08:51:29.676Z]  "GET /%3C%=%20BASE_URL%20%%3Efavicon.ico" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
(node:95835) [DEP0066] DeprecationWarning: OutgoingMessage.prototype._headers is deprecated
(Use `node --trace-deprecation ...` to show where the warning was created)
image

Without Cypress integration everything works fine

rr@Rambods-MacBook-Pro cypress-test % npx chromatic --force-rebuild --project-token=chpt_8425323d910bbc6          

Chromatic CLI v11.10.4
https://www.chromatic.com/docs/cli

  ✔ Authenticated with Chromatic
    → Using project token '****************bbc6'
  ✔ Retrieved git information
    → Commit '4362f44' on branch 'feat/storybook'; found 1 parent build; TurboSnap disabled
  ✔ Collected Storybook metadata
    → Storybook 8.3.5 for Vue3; using the @storybook/vue3-vite builder (8.3.5); supported addons found: Links, Essentials, Actions, Interactions
  ✔ Initialized build
    → Build 100 initialized
  ✔ Storybook built in 4 seconds
    → View build log at /Users/rr/DevOps/cypress-test/build-storybook.log
  ✔ Publish complete in 4 seconds
    → Uploaded 12 files (530.97 kB), skipped 77 files
  ✔ Started build 100
    → View build details at https://www.chromatic.com/build?appId=66feb692fd101b012ed2da6c&number=100
  ✔ Build 100 completed
    → Tested 9 stories across 4 components; captured 9 snapshots in 35 seconds

⚠ TurboSnap disabled due to rebuild
You appear to be rerunning an earlier build, because the baseline build has the same commit and branch name.
Comparing against the same commit would yield zero changed files, so we would end up running a build with no snapshots.
That's probably not what you want when rerunning a build, so we're just going to run a full build instead.

✔ Storybook published
We found 4 components with 9 stories.
ℹ View your Storybook at https://66feb692fd101b012ed2da6c-kzezurxaya.chromatic.com/

ℹ Speed up Continuous Integration
Your project is linked to GitHub so Chromatic will report results there.
This means you can pass the --exit-once-uploaded flag to skip waiting for build results.
Read more here: https://www.chromatic.com/docs/cli#chromatic-options

✖ Found 1 visual change: Review the changes at https://www.chromatic.com/build?appId=66feb692fd101b012ed2da6c&number=100

ℹ For CI/CD use cases, this command failed with exit code 1
Pass --exit-zero-on-changes to succeed this command regardless of changes.
Pass --auto-accept-changes to succeed and automatically accept any changes.

Additional Notes:

.storybook/main.ts

import type { StorybookConfig } from "@storybook/vue3-vite";

const config: StorybookConfig = {
  stories: [
    "../src/**/*.mdx",
    "../src/**/*.stories.@(js|jsx|mjs|ts|tsx)",
    "../src/components/LocatorTest.stories.ts",
    "../src/**/*.stories.ts",
  ],
  addons: [
    "@storybook/addon-onboarding",
    "@storybook/addon-links",
    "@storybook/addon-essentials",
    "@storybook/addon-actions",
    "@chromatic-com/storybook",
    "@storybook/addon-interactions",
  ],
  framework: {
    name: "@storybook/vue3-vite",
    options: {},
  },
};
export default [config;](
[tsconfig.json](https://github.com/user-attachments/files/17255694/tsconfig.json)
[package.json](https://github.com/user-attachments/files/17255693/package.json))
rr@Rambods-MacBook-Pro cypress-test % ls -l cypress/downloads/chromatic-archives                                                                        
total 24
drwxr-xr-x  7 rr  staff  224 Oct  4 10:37 archive
-rw-r--r--  1 rr  staff  188 Oct  4 10:37 find-elements-using-css-selectors.stories.json
-rw-r--r--  1 rr  staff  172 Oct  4 10:37 find-elements-using-xpath.stories.json
-rw-r--r--  1 rr  staff  134 Oct  4 10:37 passes.stories.json

build-storybook.log package.json tsconfig.json

codykaup commented 1 month ago

That error looks like it's coming from the viewport parser. How are you configuring your viewports?

rambod-rahmani commented 1 month ago

Hello @codykaup ,

here is the story with the viewport configuration:

import { Meta, StoryFn } from "@storybook/vue3";
import LocatorTest from "./LocatorTest.vue";

export default {
  title: "Components/LocatorTest",
  component: LocatorTest,
  parameters: {
    chromatic: { viewports: [320, 1200] },
  },
} as Meta<typeof LocatorTest>;

const Template: StoryFn<typeof LocatorTest> = (args) => ({
  components: { LocatorTest },
  setup() {
    return { args };
  },
  template: '<LocatorTest v-bind="args" />',
});

export const Default = Template.bind({});
Default.args = {};

for my sample Vue component

sample Vue Component

<template>
    <div class="locator-test">
        <h1 class="header" id="main-title">Locator Test Page</h1>

        <div class="form-group">
            <label for="username">Username:</label>
            <input type="text" id="username" placeholder="Enter username" class="form-input" />
        </div>

        <div class="form-group">
            <label for="password">Password:</label>
            <input type="password" id="password" placeholder="Enter password" class="form-input" />
        </div>

        <div class="button-group">
            <button id="submit-button" class="primary-button">Submit</button>
            <button id="cancel-button" class="secondary-button">Cancel</button>
        </div>

        <p class="info-text">Test your CSS and XPath locators on this page.</p>

        <ul class="item-list">
            <li class="item">Item 1</li>
            <li class="item">Item 2</li>
            <li class="item">Item 3</li>
        </ul>
    </div>
</template>

<script>
export default {
    name: "LocatorTest",
};
</script>

<style scoped>
.locator-test {
    padding: 20px;
    background-color: #f5f5f5;
    max-width: 400px;
    margin: 0 auto;
}

.header {
    color: #333;
    margin-bottom: 20px;
}

.form-group {
    display: flex;
    align-items: center;
    margin-bottom: 15px;
}

.form-group label {
    margin-right: 10px;
    min-width: 80px;
}

.form-input {
    padding: 10px;
    width: 100%;
    max-width: 300px;
}

.button-group {
    margin-top: 20px;
    display: flex;
    justify-content: center;
    /* Center the buttons */
}

.primary-button {
    background-color: #42b983;
    color: white;
    padding: 10px 20px;
    border: none;
    cursor: pointer;
    margin-right: 10px;
}

.secondary-button {
    background-color: #e0e0e0;
    color: #333;
    padding: 10px 20px;
    border: none;
    cursor: pointer;
}

.info-text {
    margin-top: 30px;
    color: #666;
}

.item-list {
    list-style-type: none;
    padding: 0;
}

.item {
    padding: 10px;
    background-color: #fff;
    margin-bottom: 10px;
    border: 1px solid #ddd;
}
</style>
codykaup commented 1 month ago

That all looks like Storybook to me so you shouldn't need --cypress there! The --cypress flag should be trying to read Cypress tests which parses viewports like these examples: https://docs.cypress.io/api/commands/viewport

Is the repo public so I can take a look? I'd love a reproduction environment to see if it's a bug on our end!

rambod-rahmani commented 1 month ago

Hello @codykaup , I made the repository public for you: https://github.com/Aladia-Frontend/cypress-test

I managed to fix the issue, I guess the problem was outdated libraries perhaps? You can go back to this commit if you want a reproduction environment though: https://github.com/Aladia-Frontend/cypress-test/pull/1/commits/4362f4408aba36456903a4a6a675250c559d3b7b

Anyway, now I ma facing the following issue that you might be able to help me with. I configured my project to perform visual regression tests using stories as well as Cypress E2E tests. Is that possible? You fill find two github workflows to this end:

The problema I am facing is that on Chromatic I can only review the latest build for any given project. Also given that the builds run consecutively, everytime all tests are detected as unviewed.

So for example in the following screenshot, in build 205 I accepted all changes. Then I pushed a change and all my tests are marked as changed in build 206 (storybook-regression-tests.yml) and 207 (e2e-regression-tests.yml)

image

codykaup commented 1 month ago

Thanks @rambod-rahmani!

Perhaps I'm missing something but I don't see any Cypress tests in that repo. Regardless, I'm glad you got it working!

I configured my project to perform visual regression tests using stories as well as Cypress E2E tests. Is that possible?

I would recommend creating a separate project in Chromatic for your E2E tests then using each project for their intended target

FYI they'll have different project tokens so you'll have to set up the right one in CI when you're running the build!


Another suggestion regarding Cypress.

I'm sure you have a good reason for going that route but I wanted to point out that Storybook also has interaction testing which can cover a pretty good amount of what Cypress would offer with much less complexity (since you're already using Storybook). It even integrates directly into Chromatic so the regression test fails if the interaction test fails.

It's your call on how you want to set things up, just wanted to make you aware!

rambod-rahmani commented 1 month ago

Thank you very much @codykaup ! All your suggestions were We can close this if it is ok with you.

codykaup commented 1 month ago

Sounds good @rambod-rahmani, glad to help out! Let us know if you run into any other issues!