microsoft / playwright

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
https://playwright.dev
Apache License 2.0
67.11k stars 3.69k forks source link

[Bug]: .toMatchSnapshot() CLI output unusable with large text files #33692

Open karlhorky opened 1 day ago

karlhorky commented 1 day ago

Version

1.48.2

Steps to reproduce

  1. Create a large text snapshot, eg. comparing all list items on a very long list (eg. (await page.locator('h2:text("Schedule") + div.grid a').allInnerTexts()).join('\n'))
  2. Make an intentional error in the text snapshot
  3. ๐Ÿ’ฅ Run Playwright CLI and see very hard to parse output (see below) - especially hard in logs for GitHub Actions workflow failures
  1) playwright/schedulerCreatesCohort.spec.ts:115:1 โ€บ Scheduler creates cohort โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€

    Error: Snapshot comparison failed:

      [
        {
          "title": "Intro to Course, Remote Collaboration",
          "startTimestamp": "2024-09-09T07:00:00.000Z",
          "endTimestamp": "2024-09-09T09:00:00.000Z"
        },

        ... many more lines here, potentially hundreds or thousands of lines ...

      ]

At some point, this visual diff using colored text via ANSI escape codes shows up in the output:

Image

This is really easy to miss, even if you quickly scroll through all lines.

Expected behavior

  1. Show a concise diff, with only a few lines of context on each side of each difference
  2. Maybe don't only show the changes via color, but also via text (accessibility)

Eg. the same type of diff as in .toEqual([ ... ])

Actual behavior

  1. Shows the full file content
  2. Only shows diff via hard-to-spot colors

Additional context

--

Environment

System:
    OS: macOS 15.1.1
    CPU: (8) arm64 Apple M1
    Memory: 81.94 MB / 16.00 GB
  Binaries:
    Node: 22.11.0 - /opt/homebrew/bin/node
    Yarn: 1.22.22 - /opt/homebrew/bin/yarn
    npm: 10.9.0 - /opt/homebrew/bin/npm
    bun: 1.1.30 - /opt/homebrew/bin/bun
  IDEs:
    VSCode: 1.95.3 - /opt/homebrew/bin/code
  Languages:
    Bash: 5.2.0 - /opt/homebrew/bin/bash
karlhorky commented 1 day ago

Workaround

Use toEqual() like this:

expect(await page.locator('h2:text("Schedule") + div.grid a').allInnerTexts()).toEqual([ ... ])

This will display a better, more usable, accessible diff with fewer lines of context:

ยท
  1) playwright/pernExtensiveImmersiveStudentBrowses.spec.ts:33:3 โ€บ PERN Extensive (Immersive) student browses โ€บ PERN Extensive (Immersive) student browses 
    Error: expect(received).toEqual(expected) // deep equality
    - Expected  - 1
    + Received  + 1
    @@ -64,11 +64,11 @@
      09:30 - 11:00",
        "Project Time
      11:00 - 17:00",
        "Retrospective and Review
      17:00 - 18:00",
    -   "๐Ÿง‘โ€๐Ÿซ Intro to Cour, Remote Collaboration",
    +   "๐Ÿง‘โ€๐Ÿซ Intro to Course, Remote Collaboration",
        "๐Ÿง‘โ€๐Ÿซ Job Application Processes",
        "๐Ÿง‘โ€๐Ÿซ LinkedIn and GitHub Profile Setup",
        "๐Ÿง‘โ€๐Ÿซ CV Setup and Job Market",
        "Daily Scrum",
      326 |       expect(await allScheduleAnchors.count()).toBeGreaterThan(200);
      327 |     }).toPass();
    > 328 |     expect(await allScheduleAnchors.allInnerTexts()).toEqual([
          |                                                      ^
        at 

Image