search

vaadin / web-components

A set of high-quality standards based web components for enterprise web applications. Part of Vaadin 20+
https://vaadin.com/docs/latest/components
470 stars 83 forks source link

Investigate flaky visual tests in Lumo builds #7823

Closed web-padawan closed 1 month ago

web-padawan commented 2 months ago

Problem

confirm-dialog

 ❌ confirm-dialog > basic
      Error: Screenshot is not the same width and height as the baseline. Baseline: { width: 1024, height: 650 } Screenshot: { width: 1024, height: 695 }
        at async n.<anonymous> (packages/confirm-dialog/test/visual/lumo/confirm-dialog.test.js:26:5)

 ❌ confirm-dialog > cancel
      Error: Visual diff failed. New screenshot is 0.61% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/confirm-dialog/test/visual/lumo/screenshots/confirm-dialog/failed/cancel-diff.png
        at async n.<anonymous> (packages/confirm-dialog/test/visual/lumo/confirm-dialog.test.js:31:5)

 ❌ confirm-dialog > reject
      Error: Visual diff failed. New screenshot is 0.41% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/confirm-dialog/test/visual/lumo/screenshots/confirm-dialog/failed/reject-diff.png
        at async n.<anonymous> (packages/confirm-dialog/test/visual/lumo/confirm-dialog.test.js:36:5)

 ❌ confirm-dialog > cancel reject
      Error: Visual diff failed. New screenshot is 0.41% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/confirm-dialog/test/visual/lumo/screenshots/confirm-dialog/failed/cancel-reject-diff.png
        at async n.<anonymous> (packages/confirm-dialog/test/visual/lumo/confirm-dialog.test.js:42:5)

crud

 ❌ crud > theme-no-border-edit
      Error: Visual diff failed. New screenshot is 0.88% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/crud/test/visual/lumo/screenshots/crud/failed/theme-no-border-edit-diff.png
        at async Ka.<anonymous> (packages/crud/test/visual/lumo/crud.test.js:40:5)

 ❌ crud > ltr > ltr-editor-position-default > editor-position-default-new
      Error: Visual diff failed. New screenshot is 0.37% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/crud/test/visual/lumo/screenshots/crud/failed/ltr-editor-position-default-new-diff.png
        at async n.<anonymous> (packages/crud/test/visual/lumo/crud.test.js:95:13)

 ❌ crud > ltr > ltr-editor-position-default > editor-position-default-edit
      Error: Visual diff failed. New screenshot is 0.17% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/crud/test/visual/lumo/screenshots/crud/failed/ltr-editor-position-default-edit-diff.png
        at async n.<anonymous> (packages/crud/test/visual/lumo/crud.test.js:100:13)

 ❌ crud > ltr > ltr-editor-position-fullscreen > editor-position-fullscreen-new
      Error: Visual diff failed. New screenshot is 0.42% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/crud/test/visual/lumo/screenshots/crud/failed/ltr-editor-position-fullscreen-new-diff.png
        at async n.<anonymous> (packages/crud/test/visual/lumo/crud.test.js:95:13)

 ❌ crud > ltr > ltr-editor-position-fullscreen > editor-position-fullscreen-edit
      Error: Visual diff failed. New screenshot is 1.26% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/crud/test/visual/lumo/screenshots/crud/failed/ltr-editor-position-fullscreen-edit-diff.png
        at async n.<anonymous> (packages/crud/test/visual/lumo/crud.test.js:100:13)

 ❌ crud > rtl > rtl-editor-position-default > editor-position-default-new
      Error: Visual diff failed. New screenshot is 0.37% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/crud/test/visual/lumo/screenshots/crud/failed/rtl-editor-position-default-new-diff.png
        at async n.<anonymous> (packages/crud/test/visual/lumo/crud.test.js:95:13)

 ❌ crud > rtl > rtl-editor-position-default > editor-position-default-edit
      Error: Visual diff failed. New screenshot is 0.88% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/crud/test/visual/lumo/screenshots/crud/failed/rtl-editor-position-default-edit-diff.png
        at async n.<anonymous> (packages/crud/test/visual/lumo/crud.test.js:100:13)

 ❌ crud > rtl > rtl-editor-position-fullscreen > editor-position-fullscreen-new
      Error: Visual diff failed. New screenshot is 0.44% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/crud/test/visual/lumo/screenshots/crud/failed/rtl-editor-position-fullscreen-new-diff.png
        at async n.<anonymous> (packages/crud/test/visual/lumo/crud.test.js:95:13)

field-highlighter

 ❌ field-highlighter > checkbox > default
      Error: Visual diff failed. New screenshot is 6.49% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/field-highlighter/test/visual/lumo/screenshots/field-highlighter/failed/checkbox-diff.png
        at async n.<anonymous> (packages/field-highlighter/test/visual/lumo/field-highlighter.test.js:34:7)

 ❌ field-highlighter > checkbox-group > default
      Error: Visual diff failed. New screenshot is 3.05% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/field-highlighter/test/visual/lumo/screenshots/field-highlighter/failed/checkbox-group-diff.png
        at async n.<anonymous> (packages/field-highlighter/test/visual/lumo/field-highlighter.test.js:62:7)

 ❌ field-highlighter > date-time-picker > default
      Error: Visual diff failed. New screenshot is 3.16% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/field-highlighter/test/visual/lumo/screenshots/field-highlighter/failed/date-time-picker-diff.png
        at async n.<anonymous> (packages/field-highlighter/test/visual/lumo/field-highlighter.test.js:80:7)

 ❌ field-highlighter > list-box > default
      Error: Visual diff failed. New screenshot is 4.77% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/field-highlighter/test/visual/lumo/screenshots/field-highlighter/failed/list-box-diff.png
        at async n.<anonymous> (packages/field-highlighter/test/visual/lumo/field-highlighter.test.js:110:7)

 ❌ field-highlighter > radio-button > default
      Error: Visual diff failed. New screenshot is 6.57% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/field-highlighter/test/visual/lumo/screenshots/field-highlighter/failed/radio-button-diff.png
        at async n.<anonymous> (packages/field-highlighter/test/visual/lumo/field-highlighter.test.js:129:7)

 ❌ field-highlighter > radio-group > default
      Error: Visual diff failed. New screenshot is 3.02% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/field-highlighter/test/visual/lumo/screenshots/field-highlighter/failed/radio-group-diff.png
        at async n.<anonymous> (packages/field-highlighter/test/visual/lumo/field-highlighter.test.js:157:7)

 ❌ field-highlighter > text-area > default
      Error: Visual diff failed. New screenshot is 3.80% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/field-highlighter/test/visual/lumo/screenshots/field-highlighter/failed/text-area-diff.png
        at async n.<anonymous> (packages/field-highlighter/test/visual/lumo/field-highlighter.test.js:184:7)

 ❌ field-highlighter > text-field > default
      Error: Visual diff failed. New screenshot is 5.74% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/field-highlighter/test/visual/lumo/screenshots/field-highlighter/failed/text-field-diff.png
        at async n.<anonymous> (packages/field-highlighter/test/visual/lumo/field-highlighter.test.js:202:7)

multi-select-combo-box

 ❌ multi-select-combo-box > error message
      Error: Screenshot is not the same width and height as the baseline. Baseline: { width: 212, height: 124 } Screenshot: { width: 212, height: 121 }
        at async Ka.<anonymous> (packages/multi-select-combo-box/test/visual/lumo/multi-select-combo-box.test.js:73:5)

text-area

 ❌ text-area > error message
      Error: Screenshot is not the same width and height as the baseline. Baseline: { width: 212, height: 146 } Screenshot: { width: 212, height: 138 }
        at async n.<anonymous> (packages/text-area/test/visual/lumo/text-area.test.js:74:5)

Proposed solution

We should look into disabling animations (especially in dialogs). Some not-animated-styles.js might need updates.

web-padawan commented 1 month ago

Should be fixed by #7284, #7830 and #7831. Closing.