search

vaadin / web-components

A set of high-quality standards based web components for enterprise web applications. Part of Vaadin 20+
https://vaadin.com/docs/latest/components
466 stars 83 forks source link

Investigate flaky tests in CI #3538

Open web-padawan opened 2 years ago

web-padawan commented 2 years ago

This ticket is a list of tests that fail in GitHub Actions occasionally. We need to investigate if they can be fixed, or skipped.

Unit tests

Chrome

packages/date-picker/test/keyboard-navigation.test.js:

 ❌ keyboard navigation > overlay > should focus the closest allowed date with pagedown when selected date is disabled
      TypeError: Cannot read properties of null (reading 'date')
        at qy.<anonymous> (packages/date-picker/test/keyboard-navigation.test.js:383:19)

Firefox

packages/date-picker/test/keyboard-navigation.test.js:

 ❌ keyboard navigation > overlay > should focus min date with shift and pageup
      TypeError: cell is null
        at packages/date-picker/test/keyboard-navigation.test.js:363:7
packages/date-picker/test/keyboard-navigation.test.js:

 ❌ keyboard navigation > overlay > should focus the closest allowed date with shift pageup when selected date is disabled
      TypeError: cell is null
        at packages/date-picker/test/keyboard-navigation.test.js:397:7
packages/date-picker/test/keyboard-input.test.js:

 ❌ keyboard > change event > should fire change on selecting date with Enter
      AssertionError: expected false to be true
      + expected - actual

      -false
      +true

      at r (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:9574:12)
      at n.exports<[3]</t.exports/i.prototype.assert (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:250:12)
      at n.exports<[5]</t.exports/< (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:1037:9)
      at e (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:7959:28)
      at e (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:9372:21)
      at packages/date-picker/test/keyboard-input.test.js:524:13
packages/form-layout/test/form-layout.test.js:

 ❌ form layout > container overflow > no spacing on edges (rtl) > should not have spacing on the right edge
      AssertionError: expected 288 to be close to 300 +/- 0.5
        at r (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:9574:12)
        at n.exports<[3]</t.exports/i.prototype.assert (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:250:12)
        at S (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:[33](https://github.com/vaadin/web-components/runs/5445517447?check_suite_focus=true#step:6:33)50:9)
        at c (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:7884:24)
        at packages/form-layout/test/form-layout.test.js:278:61
        at packages/form-layout/test/form-layout.test.js:277:59
packages/menu-bar/test/menu-bar.test.js:

 ❌ item components > should teleport the same component to overflow sub-menu and back
      AssertionError: expected undefined to equal <vaadin-context-menu-item theme="menu-bar-item" aria-selected="false" role="menuitem" class="" aria-haspopup="false" tabindex="0"><div style="width: 100px;">Item 3</div></vaadin-context-menu-item>
        at r (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:9574:12)
        at n.exports<[3]</t.exports/i.prototype.assert (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:250:12)
        at p (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:1409:11)
        at c (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:7884:24)
        at handleDom (node_modules/@open-wc/semantic-dom-diff/chai-dom-diff.js:102:16)
        at f (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:9164:32)
        at packages/menu-bar/test/menu-bar.test.js:577:33

WebKit

packages/date-picker/test/keyboard-input.test.js:

 ❌ keyboard > focused date > should display focused date while overlay focused
      Error: Timeout of 10000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.
        at createTimeoutError (node_modules/@web/test-runner-mocha/dist/autorun.js:1:243204)
        at node_modules/@web/test-runner-mocha/dist/autorun.js:1:257230
packages/upload/test/upload.test.js:

 ❌ upload > Abort Files > should fire `file-remove` and remove from files
      AssertionError: expected false to be true
      + expected - actual

      -false
      +true

      at r (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:9576:21)
      at node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:2[51](https://github.com/vaadin/web-components/runs/5446269040?check_suite_focus=true#step:6:51):8
      at node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:1037:9
      at e (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:7959:33)
      at packages/upload/test/upload.test.js:[52](https://github.com/vaadin/web-components/runs/5446269040?check_suite_focus=true#step:6:52)0:35
packages/date-picker/test/keyboard-navigation.test.js:

 ❌ keyboard navigation > overlay > should focus the closest allowed date with shift pageup when selected date is disabled
      AssertionError: expected Thu, 10 Jun 1999 00:00:00 GMT to deeply equal Sat, 25 Dec 1999 00:00:00 GMT
      + expected - actual

      -[Date: 1999-06-10T00:00:00.000Z]
      +[Date: 1999-12-25T00:00:00.000Z]

      at qy.<anonymous> (packages/date-picker/test/keyboard-navigation.test.js:397:28)
packages/grid/test/scrolling-mode.test.js:

 ❌ scrolling mode > overflow attribute > update on resize
      Error: Timeout of 10000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.
        at createTimeoutError (node_modules/@web/test-runner-mocha/dist/autorun.js:1:243204)
        at node_modules/@web/test-runner-mocha/dist/autorun.js:1:257230
packages/upload/test/upload.test.js:

 🚧 Browser logs:
      Error: Cannot clear timer: timer created with requestAnimationFrame() but cleared with clearTimeout()
        at clearTimer (node_modules/sinon/pkg/sinon-esm.js:4809:32)
        at packages/upload/src/vaadin-upload.js:606:19
        at progress (packages/upload/test/common.js:89:30)

 ❌ upload > Abort Files > should fire `file-remove` and remove from files
      Error: Error: Cannot clear timer: timer created with requestAnimationFrame() but cleared with clearTimeout() (http://localhost:8000/node_modules/sinon/pkg/sinon-esm.js:4815)
        at node_modules/@web/test-runner-mocha/dist/autorun.js:1:326426

Visual tests

Menu Bar

packages/menu-bar/test/visual/material/menu-bar.test.js:

 ❌ menu-bar > ltr > theme > contained
      Error: Visual diff failed. New screenshot is 0.003600230414746544% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/menu-bar/test/visual/material/screenshots/menu-bar/failed/ltr-contained-diff.png
        at async o.<anonymous> (packages/menu-bar/test/visual/material/menu-bar.test.js:[94](https://github.com/vaadin/web-components/runs/5495396052?check_suite_focus=true#step:5:94):11)

 ❌ menu-bar > rtl > theme > contained
      Error: Visual diff failed. New screenshot is 0.001350086405529[95](https://github.com/vaadin/web-components/runs/5495396052?check_suite_focus=true#step:5:95)4% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/menu-bar/test/visual/material/screenshots/menu-bar/failed/rtl-contained-diff.png
        at async o.<anonymous> (packages/menu-bar/test/visual/material/menu-bar.test.js:94:11)
vursen commented 2 years ago

Here is an updated list of still flaky tests after several improvements have been made (see the referenced PRs).

The tests are not stable in WebKit and Firefox.

menu-bar

packages/menu-bar/test/menu-bar.test.js:

 ❌ item components > should teleport the same component to overflow sub-menu and back
      AssertionError: expected undefined to equal <vaadin-context-menu-item theme="menu-bar-item" aria-selected="false" role="menuitem" class="" aria-haspopup="false" tabindex="0"><div style="width: 100px;">Item 3</div></vaadin-context-menu-item>
        at r (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:9574:12)
        at n.exports<[3]</t.exports/i.prototype.assert (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:250:12)
        at p (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:1409:11)
        at c (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:7884:24)
        at handleDom (node_modules/@open-wc/semantic-dom-diff/chai-dom-diff.js:102:16)
        at f (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:9164:32)
        at packages/menu-bar/test/menu-bar.test.js:577:33

form-layout

packages/form-layout/test/form-layout.test.js:

 ❌ form layout > container overflow > no spacing on edges (rtl) > should not have spacing on the right edge
      AssertionError: expected 288 to be close to 300 +/- 0.5
        at r (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:9574:12)
        at n.exports<[3]</t.exports/i.prototype.assert (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:250:12)
        at S (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:[33](https://github.com/vaadin/web-components/runs/5445517447?check_suite_focus=true#step:6:33)50:9)
        at c (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:7884:24)
        at packages/form-layout/test/form-layout.test.js:278:61
        at packages/form-layout/test/form-layout.test.js:277:59

grid

packages/grid/test/scrolling-mode.test.js:

 ❌ scrolling mode > overflow attribute > update on resize
      Error: Timeout of 10000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.
        at createTimeoutError (node_modules/@web/test-runner-mocha/dist/autorun.js:1:243204)
        at node_modules/@web/test-runner-mocha/dist/autorun.js:1:257230
web-padawan commented 2 years ago

Here are some more flaky tests from the recent coverage builds (failed in Chrome):

date-picker

packages/date-picker/test/scroller.test.js:

 ❌ vaadin-infinite-scroller > should reflect currently visible item index as position scrolling up
      Error: Uncaught AssertionError: expected 1.4000000000000057 to be below 1 (http://localhost:8000/node_modules/@esm-bundle/chai/esm/chai.js:[98](https://github.com/vaadin/web-components/runs/7611221226?check_suite_focus=true#step:5:99))
        at t.onerror (node_modules/@web/test-runner-mocha/dist/autorun.js:1:326417)
packages/date-picker/test/keyboard-navigation.test.js:

 ❌ keyboard navigation > overlay > should focus previous year with shift and pageup
      AssertionError: expected Sat, 01 Jan 2000 00:00:00 GMT to deeply equal Fri, 01 Jan 1999 00:00:00 GMT
      + expected - actual

      -[Date: 2000-01-01T00:00:00.000Z]
      +[Date: 1999-01-01T00:00:00.000Z]

      at qy.<anonymous> (packages/date-picker/test/keyboard-navigation.test.js:274:38)

field-highlighter

packages/field-highlighter/test/user-tags.test.js:

 ❌ user-tags > default > should create user tags for each added user
      AssertionError: expected { entries: [Function: entries],
        keys: [Function: keys],
        values: [Function: values],
        forEach: [Function: forEach],
        length: 0,
        item: [Function: item] } to have a length of 2 but got 0
      + expected - actual

      -0
      +2

      at o.<anonymous> (packages/field-highlighter/test/user-tags.test.js:55:28)

UPD (27.10.2022): added some more flaky tests

Firefox:

packages/combo-box/test/lazy-loading.test.js:

 ❌ lazy loading > combo-box-light > dataProvider > async > should keep the focused item focused when loading more items
      AssertionError: expected -1 to equal 0
      + expected - actual

      --1
      +0

Chrome:

packages/grid/test/scrolling-mode.test.js:

 ❌ scrolling mode > overflow attribute > update on resize
      Error: Timeout of 10000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.
        at createTimeoutError (node_modules/@web/test-runner-mocha/dist/autorun.js:1:243195)
        at Gg._timeoutError (node_modules/@web/test-runner-mocha/dist/autorun.js:1:259145)
        at node_modules/@web/test-runner-mocha/dist/autorun.js:1:257217

WebKit:

packages/component-base/test/virtualizer-item-height.test.js:
 ❌ virtualizer - item height > should adjust the placeholder height
      AssertionError: expected 200 to be below 40
      + expected - actual

      -200
      +40

      at r (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:9575:6)
      at node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:251:8
      at y (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:1781:10)
      at c (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:7884:30)
sissbruecker commented 1 year ago

Update from 23.06.2023

Went through the workflow runs of the last few weeks and collected failed workflows. Pretty much none of the tests mentioned above caused a workflow to fail in the last four weeks. However I'm not sure if Github keeps track of workflows that have been manually re-run, the number of failed workflows was suspiciously low compared to what you would expect from having to re-run actions regularly.

Anyway, here are some tests that came up regularly in failed workflows, which have not been mentioned so far:

date-picker

Chrome, Firefox, Webkit

packages/date-picker/test/events.test.js:
    ❌ events > has-input-value-changed event > without value > with user input > should be fired when selecting a date with Space
    AssertionError: expected false to be true
    + expected - actual
    -false
    +true
    at n.<anonymous> (packages/date-picker/test/events.test.js:154:51)

integration

Chrome

integration/tests/grid-tooltip.test.js:
    ❌ tooltip > scrolling > should hide the tooltip when starting scrolling
    AssertionError: expected false to be true
    + expected - actual
    -false
    +true
    at n.<anonymous> (integration/tests/grid-tooltip.test.js:356:35)

Chrome

integration/tests/grid-tooltip.test.js:

 ❌ tooltip > mouse > should set tooltip target on cell mouseenter
      AssertionError: expected <vaadin-grid aria-describedby="vaadin-tooltip-4" style="touch-action: none;">
              <vaadin-grid-column path="firstName"></vaadin-grid-column>
              <vaadin-grid-column path="lastName"></vaadin-grid-column>
              <vaadin-tooltip slot="tooltip"></vaadin-tooltip>
            <vaadin-grid-cell-content slot="vaadin-grid-cell-content-0">First name</vaadin-grid-cell-content><vaadin-grid-cell-content slot="vaadin-grid-cell-content-1">Last name</vaadin-grid-cell-content><vaadin-grid-cell-content slot="vaadin-grid-cell-content-2"></vaadin-grid-cell-content><vaadin-grid-cell-content slot="vaadin-grid-cell-content-3"></vaadin-grid-cell-content><vaadin-grid-cell-content slot="vaadin-grid-cell-content-4"></vaadin-grid-cell-content><vaadin-grid-cell-content slot="vaadin-grid-cell-content-5"></vaadin-grid-cell-content><vaadin-grid-cell-content slot="vaadin-grid-cell-content-6">John</vaadin-grid-cell-content><vaadin-grid-cell-content slot="vaadin-grid-cell-content-7">Doe</vaadin-grid-cell-content><vaadin-grid-cell-content slot="vaadin-grid-cell-content-8">Jane</vaadin-grid-cell-content><vaadin-grid-cell-content slot="vaadin-grid-cell-content-9">Doe</vaadin-grid-cell-content></vaadin-grid> to equal <td id="vaadin-grid-cell-6" role="gridcell" tabindex="0" part="cell body-cell first-column-cell first-row-cell even-row-cell drag-disabled-row-cell drop-disabled-row-cell" first-column="" aria-selected="false" style="width: 100px; flex-grow: 1; order: 10000000;"><slot name="vaadin-grid-cell-content-6"></slot></td>
        at n.<anonymous> (integration/tests/grid-tooltip.test.js:105:[36](https://github.com/vaadin/web-components/actions/runs/5314201424/jobs/9621134295#step:5:37))
web-padawan commented 3 days ago

Update: there are now a few grid-pro test failing only in Firefox after recent change to use sendKeys():

packages/grid-pro/test/keyboard-navigation-polymer.test.js:

 ❌ keyboard navigation > when `singleCellEdit` is true > should focus previous cell available for editing on the previous in non-edit mode on Shift Tab
      AssertionError: expected false to be true
      + expected - actual

      -false
      +true

 ❌ keyboard navigation > when `singleCellEdit` is true > should focus correct editable cell when column is hidden
      AssertionError: expected false to be true
      + expected - actual

      -false
      +true