[BUG] not.toBeVisible() assertion incorrectly fails at mobile screen resolution (width <= 575px)

[DvaKompota]

Context:

• Playwright Version: 1.27.1
• Operating System: Linux Mint 20.3 Cinnamon 5.2.7
• Node.js version: 16.15.0
• Browser: All

<!-- CLI to auto-capture this info -->

    Running 1 test using 1 worker
    1) [chromium] › ngx-admin/homepage.spec.ts:62:6 › Validate Home Page loads correctly in different resolutions 

    Error: expect(received).not.toBeVisible()
    Call log:
      - expect.toBeVisible with timeout 5000ms
      - waiting for selector "li.menu-item:has-text("FEATURES") >> nth=0"
      -   selector resolved to <li nbmenuitem="" _ngcontent-rsv-c116="" class="menu…>…</li>
      -   unexpected value "true"
      -   selector resolved to <li nbmenuitem="" _ngcontent-rsv-c116="" class="menu…>…</li>
      -   unexpected value "true"
      -   selector resolved to <li nbmenuitem="" _ngcontent-rsv-c116="" class="menu…>…</li>
      -   unexpected value "true"
      -   selector resolved to <li nbmenuitem="" _ngcontent-rsv-c116="" class="menu…>…</li>
      -   unexpected value "true"
      -   selector resolved to <li nbmenuitem="" _ngcontent-rsv-c116="" class="menu…>…</li>
      -   unexpected value "true"
      -   selector resolved to <li nbmenuitem="" _ngcontent-rsv-c116="" class="menu…>…</li>
      -   unexpected value "true"
      -   selector resolved to <li nbmenuitem="" _ngcontent-rsv-c116="" class="menu…>…</li>
      -   unexpected value "true"

      70 |     await expect(page.locator('li.menu-item:has-text("FEATURES")').first()).not.toBeVisible();
      71 |     await page.setViewportSize({ width: 375, height: 812 }); // iPhone X
    > 72 |     await expect(page.locator('li.menu-item:has-text("FEATURES")').first()).not.toBeVisible();
         |                                                                                 ^
      73 | });
      74 |

<!-- npx envinfo --preset playwright --markdown -->
## System:
 - OS: Linux 5.4 Linux Mint 20.3 (Una)
 - Memory: 19.67 GB / 29.40 GB
 - Container: Yes
## Binaries:
 - Node: 16.15.0 - /usr/local/bin/node
 - Yarn: 1.22.19 - /usr/bin/yarn
 - npm: 8.11.0 - /usr/local/bin/npm
## Languages:
 - Bash: 5.0.17 - /usr/bin/bash
## npmPackages:
 - playwright: ^1.27.1 => 1.27.1 

Code Snippet Validating www.akveo.com/ngx-admin application homepage at different screen resolutions:

test('Validate Home Page loads correctly in different resolutions', async ({ page }) => {
    await page.goto('https://www.akveo.com/ngx-admin/themes')
    await page.locator('nb-card-body img[alt="Material Light Theme"]').click()
    await page.setViewportSize({ width: 1920, height: 1080 }); // Full HD
    await expect(page.locator('li.menu-item:has-text("FEATURES")').first()).toBeVisible();
    await page.setViewportSize({ width: 1280, height: 800 }); // macbook-13
    await expect(page.locator('li.menu-item:has-text("FEATURES")').first()).toBeVisible();
    await page.setViewportSize({ width: 810, height: 1080 }); // iPad
    await expect(page.locator('li.menu-item:has-text("FEATURES")').first()).not.toBeVisible();
    await page.setViewportSize({ width: 375, height: 812 }); // iPhone X
    await expect(page.locator('li.menu-item:has-text("FEATURES")').first()).not.toBeVisible();
});

Describe the bug The www.akveo.com/ngx-admin is used as a demo application under test If you run the code from the snippet, the first 3 validations will pass, and the last one will fail — that is an incorrect, because the li.menu-item:has-text("FEATURES") element is definitely not visible.

NOTE: the 3rd validation (at the iPad resolution) passes correctly, as the element is already not visible at this resolution. But as soon as the width of the viewport becomes 575 pixels or less — Playwrigth starts to see this element again despite it's absence on the screen.

The bug was discovered while building a pet test automation project for 1:1 comparison with Cypress (which didn't have this issue)

[pavelfeldman]

There are two and according to the accepted visibility definitions they are visible: attached and having non-0 size. A whole vertical strip of the nb-menu seems to be on screen for some reason.

[DvaKompota]

Thank you @pavelfeldman for such a swift response! So you think it is an application's under test bug, that reproduces specifically for mobile viewport? And that for some reason (perhaps different visibility criteria?) it is not caught by Cypress, but caught by Playwright?

[pavelfeldman]

I don't think it is an app bug per se - menu area is occluded by an opaque element, so the user taps won't trigger the menu. But we can't known that before we start clicking - the element is technically visible, everything above it can be transparent and pointer-events: none-y. I would not know why it is not caught by Cypress.

[DvaKompota]

Thank you @pavelfeldman — closing the ticket then

[SkaterRambo]

https://www.akveo.com/ngx-admin/themes @skater_rambo.com