search

ArnaudBuchholz / ui5-test-runner

A test runner for UI5 applications enabling parallel execution of tests.
https://arnaudbuchholz.github.io/ui5-test-runner/
MIT License
18 stars 9 forks source link

Provide more information about failures in the report console output #53

Closed ArnaudBuchholz closed 9 months ago

marianfoo commented 1 year ago

Would be great if browser output would be in the report as well, could be a seperate file? II could have used that yesterday. :)

ArnaudBuchholz commented 12 months ago

The browser console.log is actually saved in a csv file inside the folder associated to the test execution

sb-aigh commented 11 months ago

@ArnaudBuchholz We also would like to see more information about failures in the report console output. Would be great if you could improve this.

ArnaudBuchholz commented 10 months ago

I am thinking of adding another report generator that will be a text one and that will be called at the end of the process instead of the current output.

ArnaudBuchholz commented 9 months ago

Proposal :

┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│https://ui5.sap.com/test-resources/sap/m/non_working/unitTests.qunit.html                                🧨           │
│https://ui5.sap.com/test-resources/sap/m/demokit/orderbrowser/webapp/test/unit/unitTests.qunit.html      🧪     25/25 │
│...s://ui5.sap.com/test-resources/sap/m/demokit/orderbrowser/webapp/test/integration/opaTests.qunit.html 🥼     12/12 │
│...p.com/test-resources/sap/m/demokit/orderbrowser/webapp/test/integration/opaTestsNavigation.qunit.html 🥼 🐞   5/9  │
│https://ui5.sap.com/test-resources/sap/m/demokit/cart/webapp/test/unit/unitTests.qunit.html              🧪     39/39 │
│...://ui5.sap.com/test-resources/sap/m/demokit/cart/webapp/test/integration/opaTestsComponent.qunit.html 🥼 🐞   3/80 │
│...s://ui5.sap.com/test-resources/sap/m/demokit/teamCalendar/webapp/test/integration/opaTests.qunit.html 🥼 🐞   0/9  │
│...://ui5.sap.com/test-resources/sap/tnt/demokit/toolpageapp/webapp/test/integration/opaTests.qunit.html 🥼 🐞   0/10 │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
[https://ui5.sap.com/test-resources/sap/m/non_working/unitTests.qunit.html                                             ]
Unable to run the page (check the execution log)

[https://ui5.sap.com/test-resources/sap/m/demokit/orderbrowser/webapp/test/integration/opaTestsNavigation.qunit.html   ]
Desktop navigation ▶ Should navigate on press
The list does not have a selected item so nothing can be remembered
Opa timeout after 15 seconds
This is what Opa logged:
Found 0 blocking out of 177 tracked timeouts -  sap.ui.test.autowaiter._timeoutWaiter#hasPending
AutoWaiter syncpoint -  sap.ui.test.autowaiter._autoWaiter
Found 4 controls of type 'View' in page -  sap.ui.test.Opa5
Found 1 views with viewName 'sap.ui.demo.orderbrowser.view.Master' -  sap.ui.test.Opa5
Found view with ID 'undefined' and viewName 'sap.ui.demo.orderbrowser.view.Master' -  sap.ui.test.Opa5
Found control with ID 'list' and controlType 'undefined' in view 'sap.ui.demo.orderbrowser.view.Master' -  sap.ui.test.Opa5
Control 'Element sap.m.List#__component0---master--list' is not visible -  sap.ui.test.matchers.Visible
0 out of 1 controls met the matchers pipeline requirements -  sap.ui.test.pipelines.MatcherPipeline
Matchers found no controls so check function will be skipped -  sap.ui.test.Opa5
Callstack:
    at e.waitFor (https://ui5.sap.com/resources/sap/ui/test/PageObjectFactory.js:6:1145)
    at p.iRememberTheSelectedItem (https://ui5.sap.com/test-resources/sap/m/demokit/orderbrowser/webapp/test/integration/pages/Master.js:29:18)
    at Object.<anonymous> (https://ui5.sap.com/test-resources/sap/m/demokit/orderbrowser/webapp/test/integration/NavigationJourney.js:21:24)
    at Object.<anonymous> (https://ui5.sap.com/resources/sap/ui/test/opaQunit.js:6:1964)
    at n (https://ui5.sap.com/resources/sap/ui/thirdparty/qunit-2.js:11:13583)
    at V.e [as run] (https://ui5.sap.com/resources/sap/ui/thirdparty/qunit-2.js:11:13406)
    at https://ui5.sap.com/resources/sap/ui/thirdparty/qunit-2.js:11:16188

(4 errors occurred but it is recommended to focus on the first OPA error)

[https://ui5.sap.com/test-resources/sap/m/demokit/cart/webapp/test/integration/opaTestsComponent.qunit.html            ]
Welcome Journey ▶ Should press the product link and navigate to product view
The product list does not contain any entries
Opa timeout after 15 seconds
This is what Opa logged:
Found 0 blocking out of 263 tracked timeouts -  sap.ui.test.autowaiter._timeoutWaiter#hasPending
AutoWaiter syncpoint -  sap.ui.test.autowaiter._autoWaiter
Found 5 controls of type 'View' in page -  sap.ui.test.Opa5
Found 1 views with viewName 'sap.ui.demo.cart.view.Category' -  sap.ui.test.Opa5
Found view with ID 'undefined' and viewName 'sap.ui.demo.cart.view.Category' -  sap.ui.test.Opa5
Found control with ID 'productList' and controlType 'undefined' in view 'sap.ui.demo.cart.view.Category' -  sap.ui.test.Opa5
Control 'Element sap.m.List#__component0---category--productList' is not visible -  sap.ui.test.matchers.Visible
0 out of 1 controls met the matchers pipeline requirements -  sap.ui.test.pipelines.MatcherPipeline
Matchers found no controls so check function will be skipped -  sap.ui.test.Opa5
Callstack:
    at e.waitFor (https://ui5.sap.com/resources/sap/ui/test/PageObjectFactory.js:6:1145)
    at p.iShouldSeeSomeEntriesInTheProductList (https://ui5.sap.com/test-resources/sap/m/demokit/cart/webapp/test/integration/pages/Category.js:338:11)
    at Object.<anonymous> (https://ui5.sap.com/test-resources/sap/m/demokit/cart/webapp/test/integration/WelcomeJourney.js:39:22)
    at Object.<anonymous> (https://ui5.sap.com/resources/sap/ui/test/opaQunit.js:6:1964)
    at n (https://ui5.sap.com/resources/sap/ui/thirdparty/qunit-2.js:11:13583)
    at V.e [as run] (https://ui5.sap.com/resources/sap/ui/thirdparty/qunit-2.js:11:13406)
    at https://ui5.sap.com/resources/sap/ui/thirdparty/qunit-2.js:11:16188

(77 errors occurred but it is recommended to focus on the first OPA error)

[https://ui5.sap.com/test-resources/sap/m/demokit/teamCalendar/webapp/test/integration/opaTests.qunit.html             ]
Team Calendar ▶ Should see the Planning Calendar initially set
The Planning Calendar rows are not equal to the team members
Opa timeout after 15 seconds
This is what Opa logged:
Found 0 blocking out of 159 tracked timeouts -  sap.ui.test.autowaiter._timeoutWaiter#hasPending
AutoWaiter syncpoint -  sap.ui.test.autowaiter._autoWaiter
Found no control with the global ID 'PlanningCalendar' -  sap.ui.test.Opa5
Matchers found no controls so check function will be skipped -  sap.ui.test.Opa5
Callstack:
    at p.thePlanningCalendarShouldHaveAllTeamMembers (https://ui5.sap.com/test-resources/sap/m/demokit/teamCalendar/webapp/test/integration/pages/Main.js:93:18)
    at Object.<anonymous> (https://ui5.sap.com/test-resources/sap/m/demokit/teamCalendar/webapp/test/integration/MainJourney.js:33:22)
    at Object.<anonymous> (https://ui5.sap.com/resources/sap/ui/test/opaQunit.js:6:1964)
    at n (https://ui5.sap.com/resources/sap/ui/thirdparty/qunit-2.js:11:13583)
    at V.e [as run] (https://ui5.sap.com/resources/sap/ui/thirdparty/qunit-2.js:11:13406)
    at https://ui5.sap.com/resources/sap/ui/thirdparty/qunit-2.js:11:16188
    at Object.P [as advance] (https://ui5.sap.com/resources/sap/ui/thirdparty/qunit-2.js:11:10009)

(9 errors occurred but it is recommended to focus on the first OPA error)

[https://ui5.sap.com/test-resources/sap/tnt/demokit/toolpageapp/webapp/test/integration/opaTests.qunit.html            ]
Desktop navigation ▶ should press the error button and see a popover message
Unable to load the component with the name: sap.ui.demo.toolpageapp
Failure in Opa check function
Exception thrown by the testcode:'Error: Error while waiting for promise scheduled on flow, details: TypeError: Cannot read properties of undefined (reading 'getResourceBundle')
TypeError: Cannot read properties of undefined (reading 'getResourceBundle')
    at p.getBundleTextByModel (https://ui5.sap.com/test-resources/sap/tnt/demokit/toolpageapp/webapp/controller/BaseController.js:49:26)
    at p.getBundleText (https://ui5.sap.com/test-resources/sap/tnt/demokit/toolpageapp/webapp/controller/App.controller.js:270:16)
    at p._setToggleButtonTooltip (https://ui5.sap.com/test-resources/sap/tnt/demokit/toolpageapp/webapp/controller/App.controller.js:132:9)
    at p.onSideNavButtonPress (https://ui5.sap.com/test-resources/sap/tnt/demokit/toolpageapp/webapp/controller/App.controller.js:126:9)
    at p.onInit (https://ui5.sap.com/test-resources/sap/tnt/demokit/toolpageapp/webapp/controller/App.controller.js:50:10)
    at i.fireEvent (https://ui5.sap.com/resources/sap-ui-core.js:299:2318)
    at d.fireEvent (https://ui5.sap.com/resources/sap-ui-core.js:404:4071)
    at p.fireAfterInit (https://ui5.sap.com/resources/sap-ui-core.js:332:13496)
    at E (https://ui5.sap.com/resources/sap-ui-core.js:380:1775)
    at A.runAsOwner (https://ui5.sap.com/resources/sap-ui-core.js:380:5058)
Error: Error while waiting for promise scheduled on flow, details: TypeError: Cannot read properties of undefined (reading 'getResourceBundle')
TypeError: Cannot read properties of undefined (reading 'getResourceBundle')
    at p.getBundleTextByModel (https://ui5.sap.com/test-resources/sap/tnt/demokit/toolpageapp/webapp/controller/BaseController.js:49:26)
    at p.getBundleText (https://ui5.sap.com/test-resources/sap/tnt/demokit/toolpageapp/webapp/controller/App.controller.js:270:16)
    at p._setToggleButtonTooltip (https://ui5.sap.com/test-resources/sap/tnt/demokit/toolpageapp/webapp/controller/App.controller.js:132:9)
    at p.onSideNavButtonPress (https://ui5.sap.com/test-resources/sap/tnt/demokit/toolpageapp/webapp/controller/App.controller.js:126:9)
    at p.onInit (https://ui5.sap.com/test-resources/sap/tnt/demokit/toolpageapp/webapp/controller/App.controller.js:50:10)
    at i.fireEvent (https://ui5.sap.com/resources/sap-ui-core.js:299:2318)
    at d.fireEvent (https://ui5.sap.com/resources/sap-ui-core.js:404:4071)
    at p.fireAfterInit (https://ui5.sap.com/resources/sap-ui-core.js:332:13496)
    at E (https://ui5.sap.com/resources/sap-ui-core.js:380:1775)
    at A.runAsOwner (https://ui5.sap.com/resources/sap-ui-core.js:380:5058)
    at t.check (https://ui5.sap.com/resources/sap/ui/test/Opa.js:6:4614)
    at L._executeCheck (https://ui5.sap.com/resources/sap/ui/test/Opa5.js:6:6478)
    at u.check (https://ui5.sap.com/resources/sap/ui/test/Opa5.js:6:4786)
    at p.<anonymous> (https://ui5.sap.com/resources/sap/ui/test/Opa.js:6:3689)
    at n (https://ui5.sap.com/resources/sap/ui/test/Opa.js:6:573)
    at t (https://ui5.sap.com/resources/sap/ui/test/autowaiter/_timeoutWaiter.js:6:1502)'
This is what Opa logged:
Executing OPA check function on controls null -  sap.ui.test.Opa5
Check function is:
function(){if(!r.started){r.started=true;e.then(function(){r.done=true},function(e){r.errorMessage="Error while waiting for promise scheduled on flow"+(e?", details: "+k(e):"")})}if(r.errorMessage){throw new Error(r.errorMessage)}else{return!!r.done}} -  sap.ui.test.Opa5

(10 errors occurred but it is recommended to focus on the first OPA error)
ArnaudBuchholz commented 9 months ago

@marianfoo depending on the browser command, the console output is extracted inside a folder associated to the page execution. I modified the text output to provide the folder name (since it is a hash, it is difficult to guess).

For instance :

[https://ui5.sap.com/test-resources/sap/m/demokit/teamCalendar/webapp/test/integration/opaTests.qunit.html                                                   ]
Page execution folder name : JjE$q8NMfno
Team Calendar ▶ Should see the Planning Calendar initially set
The Planning Calendar rows are not equal to the team members
Opa timeout after 15 seconds
This is what Opa logged:
Found 0 blocking out of 159 tracked timeouts -  sap.ui.test.autowaiter._timeoutWaiter#hasPending