How to display failed tests from multiple Angular versions

[DrewML]

This is in regards to #97.

I currently have a branch where we can run ng-inspector again n versions of Angular. The problem that I'm having is, I can't come up with an ideal way to visualize the test failures.

Right now, by default, the Protractor runner will just pass the test results to stdout. The problem is, you'll have that output for each version of Angular, 1 by 1, and it's going to be a pain to scroll up and figure out which failures are from which version.

I currently have a setup in place that gives us the test results in the following format:

{
    "1.0.6": [
        {
            "name": "anonymous app",
            "failedAfterAlls": [
                {
                    "matcherName": "toBe",
                    "message": "Expected 1 to be 0.",
                    "stack": "Error: Failed expectation\n    at Env.<anonymous> (/Users/admin/Documents/ng-inspector/test/e2e/specs/anon.js:11:13)\n    at /Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/jasminewd2/index.js:95:14\n    at [object Object].promise.ControlFlow.runInFrame_ (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/promise.js:1877:20)\n    at [object Object].promise.ControlFlow.runEventLoop_ (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/promise.js:1755:8)\n    at [object Object].<anonymous> (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/promise.js:2056:12)\n    at goog.async.run.processWorkQueue (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/selenium-webdriver/lib/goog/async/run.js:125:21)\n    at runMicrotasksCallback (node.js:337:7)\n    at process._tickCallback (node.js:355:11)",
                    "passed": false,
                    "expected": 0,
                    "actual": 1
                }
            ],
            "passed": false,
            "failedSpecs": []
        },
        {
            "name": "require.js",
            "failedAfterAlls": [],
            "passed": false,
            "failedSpecs": [
                {
                    "id": "spec10",
                    "description": "should inspect app loaded with require.js",
                    "fullName": "require.js should inspect app loaded with require.js",
                    "failedExpectations": [
                        {
                            "matcherName": "",
                            "message": "Failed: Error while running testForAngular: asynchronous script timeout: result was not received in 2.5 seconds\n  (Session info: chrome=43.0.2357.124)\n  (Driver info: chromedriver=2.14.313457 (3d645c400edf2e2c500566c9aa096063e707c9cf),platform=Mac OS X 10.10.1 x86_64) (WARNING: The server did not provide any stacktrace information)\nCommand duration or timeout: 2.51 seconds\nBuild info: version: '2.45.0', revision: '5017cb8', time: '2015-02-26 23:59:50'\nSystem info: host: 'DrewMachine', ip: '192.168.1.7', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.10.1', java.version: '1.8.0_20'\nDriver info: org.openqa.selenium.chrome.ChromeDriver\nCapabilities [{applicationCacheEnabled=false, rotatable=false, mobileEmulationEnabled=false, chrome={userDataDir=/var/folders/1j/pmh_9yms2j5235ql_zptrdz40000gp/T/.org.chromium.Chromium.hchtoj}, takesHeapSnapshot=true, databaseEnabled=false, handlesAlerts=true, version=43.0.2357.124, platform=MAC, browserConnectionEnabled=false, nativeEvents=true, acceptSslCerts=true, locationContextEnabled=true, webStorageEnabled=true, browserName=chrome, takesScreenshot=true, javascriptEnabled=true, cssSelectorsEnabled=true}]\nSession ID: 7398971247663e7b95252a156e1652dc",
                            "stack": "Error: Failed: Error while running testForAngular: asynchronous script timeout: result was not received in 2.5 seconds\n  (Session info: chrome=43.0.2357.124)\n  (Driver info: chromedriver=2.14.313457 (3d645c400edf2e2c500566c9aa096063e707c9cf),platform=Mac OS X 10.10.1 x86_64) (WARNING: The server did not provide any stacktrace information)\nCommand duration or timeout: 2.51 seconds\nBuild info: version: '2.45.0', revision: '5017cb8', time: '2015-02-26 23:59:50'\nSystem info: host: 'DrewMachine', ip: '192.168.1.7', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.10.1', java.version: '1.8.0_20'\nDriver info: org.openqa.selenium.chrome.ChromeDriver\nCapabilities [{applicationCacheEnabled=false, rotatable=false, mobileEmulationEnabled=false, chrome={userDataDir=/var/folders/1j/pmh_9yms2j5235ql_zptrdz40000gp/T/.org.chromium.Chromium.hchtoj}, takesHeapSnapshot=true, databaseEnabled=false, handlesAlerts=true, version=43.0.2357.124, platform=MAC, browserConnectionEnabled=false, nativeEvents=true, acceptSslCerts=true, locationContextEnabled=true, webStorageEnabled=true, browserName=chrome, takesScreenshot=true, javascriptEnabled=true, cssSelectorsEnabled=true}]\nSession ID: 7398971247663e7b95252a156e1652dc\n    at stack (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/jasmine/node_modules/jasmine-core/lib/jasmine-core/jasmine.js:1441:17)\n    at buildExpectationResult (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/jasmine/node_modules/jasmine-core/lib/jasmine-core/jasmine.js:1411:14)\n    at Spec.Env.expectationResultFactory (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/jasmine/node_modules/jasmine-core/lib/jasmine-core/jasmine.js:533:18)\n    at Spec.addExpectationResult (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/jasmine/node_modules/jasmine-core/lib/jasmine-core/jasmine.js:293:34)\n    at Env.fail (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/jasmine/node_modules/jasmine-core/lib/jasmine-core/jasmine.js:837:25)\n    at Function.next.fail (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/jasmine/node_modules/jasmine-core/lib/jasmine-core/jasmine.js:1776:19)\n    at /Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/jasminewd2/index.js:102:16\n    at [object Object].promise.ControlFlow.runInFrame_ (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/promise.js:1877:20)\n    at [object Object].promise.Callback_.goog.defineClass.notify (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/promise.js:2464:25)\n    at [object Object].promise.Promise.notify_ (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/promise.js:563:12)",
                            "passed": false,
                            "expected": "",
                            "actual": ""
                        },
                        {
                            "matcherName": "",
                            "message": "Failed: Error while waiting for Protractor to sync with the page: \"root element (div.angular-root-element) has no injector. this may mean it is not inside ng-app.\"",
                            "stack": "Error: Failed: Error while waiting for Protractor to sync with the page: \"root element (div.angular-root-element) has no injector. this may mean it is not inside ng-app.\"\n    at stack (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/jasmine/node_modules/jasmine-core/lib/jasmine-core/jasmine.js:1441:17)\n    at buildExpectationResult (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/jasmine/node_modules/jasmine-core/lib/jasmine-core/jasmine.js:1411:14)\n    at Spec.Env.expectationResultFactory (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/jasmine/node_modules/jasmine-core/lib/jasmine-core/jasmine.js:533:18)\n    at Spec.addExpectationResult (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/jasmine/node_modules/jasmine-core/lib/jasmine-core/jasmine.js:293:34)\n    at Env.fail (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/jasmine/node_modules/jasmine-core/lib/jasmine-core/jasmine.js:837:25)\n    at Function.next.fail (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/jasmine/node_modules/jasmine-core/lib/jasmine-core/jasmine.js:1776:19)\n    at /Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/jasminewd2/index.js:102:16\n    at [object Object].promise.ControlFlow.runInFrame_ (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/promise.js:1877:20)\n    at [object Object].promise.Callback_.goog.defineClass.notify (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/promise.js:2464:25)\n    at [object Object].promise.Promise.notify_ (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/promise.js:563:12)",
                            "passed": false,
                            "expected": "",
                            "actual": ""
                        }
                    ],
                    "passedExpectations": [],
                    "status": "failed"
                }
            ]
        }
    ],
    "1.1.4": [
        {
            "name": "anonymous app",
            "failedAfterAlls": [
                {
                    "matcherName": "toBe",
                    "message": "Expected 1 to be 0.",
                    "stack": "Error: Failed expectation\n    at Env.<anonymous> (/Users/admin/Documents/ng-inspector/test/e2e/specs/anon.js:11:13)\n    at /Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/jasminewd2/index.js:95:14\n    at [object Object].promise.ControlFlow.runInFrame_ (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/promise.js:1877:20)\n    at [object Object].promise.ControlFlow.runEventLoop_ (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/promise.js:1755:8)\n    at [object Object].<anonymous> (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/promise.js:2056:12)\n    at goog.async.run.processWorkQueue (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/selenium-webdriver/lib/goog/async/run.js:125:21)\n    at runMicrotasksCallback (node.js:337:7)\n    at process._tickCallback (node.js:355:11)",
                    "passed": false,
                    "expected": 0,
                    "actual": 1
                }
            ],
            "passed": false,
            "failedSpecs": []
        }
    ],
    "1.2.0": [
        {
            "name": "anonymous app",
            "failedAfterAlls": [
                {
                    "matcherName": "toBe",
                    "message": "Expected 1 to be 0.",
                    "stack": "Error: Failed expectation\n    at Env.<anonymous> (/Users/admin/Documents/ng-inspector/test/e2e/specs/anon.js:11:13)\n    at /Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/jasminewd2/index.js:95:14\n    at [object Object].promise.ControlFlow.runInFrame_ (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/promise.js:1877:20)\n    at [object Object].promise.ControlFlow.runEventLoop_ (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/promise.js:1755:8)\n    at [object Object].<anonymous> (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/promise.js:2056:12)\n    at goog.async.run.processWorkQueue (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/selenium-webdriver/lib/goog/async/run.js:125:21)\n    at runMicrotasksCallback (node.js:337:7)\n    at process._tickCallback (node.js:355:11)",
                    "passed": false,
                    "expected": 0,
                    "actual": 1
                }
            ],
            "passed": false,
            "failedSpecs": []
        }
    ],
    "1.3.0": [
        {
            "name": "anonymous app",
            "failedAfterAlls": [
                {
                    "matcherName": "toBe",
                    "message": "Expected 1 to be 0.",
                    "stack": "Error: Failed expectation\n    at Env.<anonymous> (/Users/admin/Documents/ng-inspector/test/e2e/specs/anon.js:11:13)\n    at /Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/jasminewd2/index.js:95:14\n    at [object Object].promise.ControlFlow.runInFrame_ (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/promise.js:1877:20)\n    at [object Object].promise.ControlFlow.runEventLoop_ (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/promise.js:1755:8)\n    at [object Object].<anonymous> (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/promise.js:2056:12)\n    at goog.async.run.processWorkQueue (/Users/admin/Documents/ng-inspector/node_modules/protractor/node_modules/selenium-webdriver/lib/goog/async/run.js:125:21)\n    at runMicrotasksCallback (node.js:337:7)\n    at process._tickCallback (node.js:355:11)",
                    "passed": false,
                    "expected": 0,
                    "actual": 1
                }
            ],
            "passed": false,
            "failedSpecs": []
        }
    ]
}

I've had different ideas of how to display these failures to a developer, but none of them seem ideal. The challenge is displaying all the useful data (version, suite name, spec name, stack trace), but grouped in such a way that it's not confusing.

My current idea:

• When tests fail, print a small summary of each failed suite/spec name, and what version it failed in, to stdout.
• (local only, not on remote builds, when we get those ready) Build a tiny Angular app that can be used to explore the test results. Would just run with a local server. There would still be a challenge of finding the ideal way to display the data, but we have better options in a browser than we do in a terminal.

[SomeKittens]

A quick summary of each failure would be good - most of the time it's easy to see what went wrong. I also like the idea of a more advanced HTML output.

[DrewML]

Found a solution where Protractor still displays the results in a friendly way, without needing a custom summary. Hooray for easy solutions.