search

ArnaudBuchholz / ui5-test-runner

A test runner for UI5 applications enabling parallel execution of tests.
https://arnaudbuchholz.github.io/ui5-test-runner/
MIT License
18 stars 9 forks source link

Address test instability #65

Closed ArnaudBuchholz closed 2 months ago

ArnaudBuchholz commented 9 months ago 
> ui5-test-runner@3.3.4 test:integration:playwright
> node . --capabilities --browser $/playwright.js

ui5-test-runner@3.3.4

00:01 │ Probing browser instantiation command
──────┴──────────────────────────────────────

00:01 │ Installing playwright...
──────┴─────────────────────────

added 2 packages in 925ms

00:02 │ Probing browser instantiation command
──────┴──────────────────────────────────────
playwright@1.38.1 in ...edtoolcache/node/18.18.0/x64/lib/node_modules/playwright
Browser capabilities :
 ├─ modules
 │  └─  playwright                                                              
 ├─ screenshot: ".png"
 ├─ scripts: true
 ├─ parallel: true
 └─ traces: ["console","network"]

00:14 │ Running tests
──────┴──────────────
Number of tests : 16
00:14 >> http://localhost:[358](https://github.com/ArnaudBuchholz/ui5-test-runner/actions/runs/6381606153/job/17318400182#step:5:359)95/tests/basic/index.html?listener=0  [lF0N6d4vHLk]
00:14 >> http://localhost:35895/tests/basic/ui5.html?listener=1    [Wcr9Vfa1y2s]
00:15 << http://localhost:35895/tests/basic/index.html?listener=0  [lF0N6d4vHLk]
✔️ Loads a page 913 ms
00:15 >> http://localhost:35895/tests/basic/iframe.html?listener=2 [GhJ1k1VrwNM]
00:15 << http://localhost:35895/tests/basic/ui5.html?listener=1    [Wcr9Vfa1y2s]
✔️ Loads a UI5 example 1756 ms
00:15 >> ...alhost:35895/tests/dynamic-include/one.html?listener=3 [oyJ7tN3TWuU]
00:16 << ...://localhost:35895/tests/basic/iframe.html?listener=2  [GhJ1k1VrwNM]
✔️ Loads a UI5 inside an iframe 1621 ms
00:16 >> ...alhost:35895/tests/dynamic-include/two.html?listener=4 [KZ41SOa41qU]
00:16 << ...lhost:35895/tests/dynamic-include/one.html?listener=3  [oyJ7tN3TWuU]
✔️ Dynamic include 814 ms
00:16 >> ...alhost:35895/tests/dynamic-include/mix.html?listener=5 [S7TRJZXc1JY]
00:18 << ...lhost:35895/tests/dynamic-include/mix.html?listener=5  [S7TRJZXc1JY]
✔️ Dynamic include mixed with a static one 1818 ms
00:18 >> ...5895/tests/local-storage/index.html?value=1&listener=6 [aV228gm5S9s]
00:18 << ...lhost:35895/tests/dynamic-include/two.html?listener=4  [KZ41SOa41qU]
✔️ Dynamic includes 1972 ms
00:18 >> ...5895/tests/local-storage/index.html?value=2&listener=7 [O0Jl_mkQ0Nw]
00:19 << ...895/tests/local-storage/index.html?value=1&listener=6  [aV228gm5S9s]
✔️ Local storage (1) 824 ms
00:19 >> ...localhost:35895/tests/screenshot/index.html?listener=8 [HoxyFGhb4fI]
00:19 << ...895/tests/local-storage/index.html?value=2&listener=7  [O0Jl_mkQ0Nw]
✔️ Local storage (2) 915 ms
00:19 >> ...://localhost:35895/tests/scripts/qunit.html?listener=9 [OYT3IaEkFEs]
00:20 << ...ocalhost:35895/tests/screenshot/index.html?listener=8  [HoxyFGhb4fI]
✔️ Screenshot 1210 ms
00:20 >> ...calhost:35895/tests/scripts/testsuite.html?listener=10 [uQ9egZ64sM4]
00:21 << ...//localhost:35895/tests/scripts/qunit.html?listener=9  [OYT3IaEkFEs]
✔️ Scripts (QUnit) 1581 ms
00:21 >> ...wser/webapp/test/unit/unitTests.qunit.html?listener=11 [GwmYbnvtwLE]
00:21 << ...alhost:35895/tests/scripts/testsuite.html?listener=10  [uQ9egZ64sM4]
✔️ Scripts (TestSuite) 933 ms
00:21 >> .../localhost:35895/tests/scripts/iframe.html?listener=12 [FB4lUxUKkhU]
00:22 << ...localhost:35895/tests/scripts/iframe.html?listener=12  [FB4lUxUKkhU]
✔️ Scripts (IFrame Coverage) 968 ms
00:22 >> ...ests/timeout/index.html?rate=100&wait=1000&listener=13 [WG81q9_9vJc]
00:23 << ...ser/webapp/test/unit/unitTests.qunit.html?listener=11  [GwmYbnvtwLE]
✔️ Scripts (External QUnit) 2127 ms
00:23 >> ...ests/timeout/index.html?rate=250&wait=1250&listener=14 [$JcvJ4XzIJY]
00:24 << ...sts/timeout/index.html?rate=100&wait=1000&listener=13  [WG81q9_9vJc]
✔️ Timeout (100ms) 2043 ms
00:24 >> ...://localhost:35895/tests/traces/index.html?listener=15 [WqAdCL23xak]
00:25 << ...//localhost:35895/tests/traces/index.html?listener=15  [WqAdCL23xak]
❌ Console logs [WqAdCL23xak] AssertionError [ERR_ASSERTION]: All expected logs were found
    at Object.endpoint (/home/runner/work/ui5-test-runner/ui5-test-runner/src/capabilities/tests/traces/index.js:79:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async EventEmitter.<anonymous> (/home/runner/work/ui5-test-runner/ui5-test-runner/src/capabilities/index.js:156:15) {
  generatedMessage: false,
  code: 'ERR_ASSERTION',
  actual: 5,
  expected: 6,
  operator: 'strictEqual'
}
00:25 << ...sts/timeout/index.html?rate=250&wait=1250&listener=14  [$JcvJ4XzIJY]
Report folder /home/runner/work/ui5-test-runner/ui5-test-runner/report not cleaned because of errors.
✔️ Timeout (250ms) 1959 ms
Done.
Error: Process completed with exit code 1.
ArnaudBuchholz commented 2 months ago

Not reproduced