Percy setup - E2E tests

[shuther]

Could you confirm how the E2E tests should be launched? In the readme, there is a mention about npm run test:e2e; if I set the export for PERCY_TOKEN, I get:

Running 7 tests using 4 workers
  ✓  1 [chromium] › e2e/Sanity.check.spec.ts:20:9 › Sanity › Static pages › should display the homepage (4.7s)
  ✓  2 [chromium] › integration/Guestbook.spec.ts:8:9 › Guestbook › Basic CRUD operations › should create a new entry in the guestbook and delete it (7.1s)
  ✓  3 [chromium] › e2e/Visual.spec.ts:6:9 › Visual testing › Static pages › should take screenshot of the homepage (4.8s)
  ✓  4 [chromium] › e2e/Guestbook.spec.ts:6:9 › Guestbook › CRUD operation › should browse to guestbook, crate a new entry, read, update and remove the newly created (8.2s)
[percy] Percy is not running, disabling snapshots
  ✓  5 [chromium] › e2e/Sanity.check.spec.ts:30:9 › Sanity › Static pages › should navigate to the about page (3.2s)
  ✓  6 [chromium] › e2e/Visual.spec.ts:18:9 › Visual testing › Static pages › should take screenshot of the about page (3.7s)
  ✓  7 [chromium] › e2e/Sanity.check.spec.ts:41:9 › Sanity › Static pages › should navigate to the portfolio page (3.2s)

So it looks that percy is not recognized.

However in the github action, the statement is: npx percy exec -- npm run test:e2e; now it seems percy is working but I get many errors while I just started with the boilerplate:

  ✓  1 [chromium] › e2e/Sanity.check.spec.ts:20:9 › Sanity › Static pages › should display the homepage (6.5s)
[percy] Invalid snapshot options:
  ✓  2 [chromium] › integration/Guestbook.spec.ts:8:9 › Guestbook › Basic CRUD operations › should create a new entry in the guestbook and delete it (9.5s)
[percy] Snapshot taken: Homepage
  ✓  3 [chromium] › e2e/Visual.spec.ts:6:9 › Visual testing › Static pages › should take screenshot of the homepage (7.9s)
  ✘  4 [chromium] › e2e/Guestbook.spec.ts:6:9 › Guestbook › CRUD operation › should browse to guestbook, crate a new entry, read, update and remove the newly created (8.7s)
  ✓  5 [chromium] › e2e/Sanity.check.spec.ts:30:9 › Sanity › Static pages › should navigate to the about page (3.6s)
[percy] Invalid snapshot options:
[percy] Snapshot taken: About
  ✓  6 [chromium] › e2e/Visual.spec.ts:18:9 › Visual testing › Static pages › should take screenshot of the about page (3.3s)
  ✓  7 [chromium] › e2e/Sanity.check.spec.ts:41:9 › Sanity › Static pages › should navigate to the portfolio page (2.8s)

  1) [chromium] › e2e/Guestbook.spec.ts:6:9 › Guestbook › CRUD operation › should browse to guestbook, crate a new entry, read, update and remove the newly created 

    Error: locator.fill: Error: strict mode violation: getByTestId('guestbook-list').getByText('Username') resolved to 2 elements:
        1) <span class="text-gray-500">RANDOM_USERNAME:</span> aka getByText('RANDOM_USERNAME:')
        2) <label for="username-2" class="text-sm font-bold text-gray-700">…</label> aka getByTestId('guestbook-list').getByText('Username', { exact: true })

    Call log:
      - waiting for getByTestId('guestbook-list').getByText('Username')

      27 |       // Update
      28 |       await guestbookList.locator('button[aria-label=edit]').last().click();
    > 29 |       await guestbookList.getByText('Username').fill(updatedUsername);
         |                                                 ^
      30 |       await guestbookList.getByRole('button', { name: 'Save' }).click();
      31 |
      32 |       // Verify after update

        at /Users/shuther/Documents/devP/my-project-name/tests/e2e/Guestbook.spec.ts:29:49

  1 failed
    [chromium] › e2e/Guestbook.spec.ts:6:9 › Guestbook › CRUD operation › should browse to guestbook, crate a new entry, read, update and remove the newly created 
  6 passed (28.5s)

[ixartz]

I use only Percy in GitHub Actions/CI. It should pass without any issue, here is the latest PR and Percy is passing: https://github.com/ixartz/Next-js-Boilerplate/pull/290

I have never tried Percy in a local environment.

[shuther]

so running percy requires: npx percy exec -- npm run test:e2e, npm run test:e2e is useful only if percy is not used. I am not sure who can update the documentation. Now it returns no error but I ran previously: npm run test-storybook:ci, not sure if this is the reason why the tests are working fine now. I still get some warning message however [percy] Invalid snapshot options:

[percy] Successfully downloaded Chromium 1262509
[percy] Percy has started!
[percy] Running "npm run test:e2e"

> next-js-boilerplate@3.54.0 test:e2e
> playwright test

Running 7 tests using 4 workers

  ✓  1 [chromium] › e2e/Sanity.check.spec.ts:20:9 › Sanity › Static pages › should display the homepage (8.4s)
[percy] Invalid snapshot options:
  ✓  2 [chromium] › integration/Guestbook.spec.ts:8:9 › Guestbook › Basic CRUD operations › should create a new entry in the guestbook and delete it (10.2s)
[percy] Snapshot taken: Homepage
  ✓  3 [chromium] › e2e/Visual.spec.ts:6:9 › Visual testing › Static pages › should take screenshot of the homepage (10.2s)
  ✓  4 [chromium] › e2e/Guestbook.spec.ts:6:9 › Guestbook › CRUD operation › should browse to guestbook, crate a new entry, read, update and remove the newly created (11.8s)
  ✓  5 [chromium] › e2e/Sanity.check.spec.ts:30:9 › Sanity › Static pages › should navigate to the about page (5.7s)
[percy] Invalid snapshot options:
[percy] Snapshot taken: About
  ✓  6 [chromium] › e2e/Visual.spec.ts:18:9 › Visual testing › Static pages › should take screenshot of the about page (6.3s)
  ✓  7 [chromium] › e2e/Sanity.check.spec.ts:41:9 › Sanity › Static pages › should navigate to the portfolio page (4.4s)

  Slow test file: [chromium] › e2e/Sanity.check.spec.ts (18.4s)
  Slow test file: [chromium] › e2e/Visual.spec.ts (16.5s)
  Consider splitting slow test files to speed up parallel execution
  7 passed (32.8s)
[percy] Finalized build #2: https://percy.io/326c6016/proj-web-test1/builds/35535039

[ixartz]

npm run test:e2e just run without Percy. And, if you want to run with Percy, you can use npx percy exec -- npm run test:e2e

I just run Percy locally and it works perfectly, no issue with Percy on local. I just never tried it locally. Usually, I only use it with GitHub Actions.

About the warning [percy] Invalid snapshot options:, it's a new issue from Percy cli: https://github.com/percy/cli/issues/1674. Several other people have the same issue.