search

theintern / intern

A next-generation code testing stack for JavaScript.
https://theintern.io/
Other
4.36k stars 310 forks source link

From Travis CI: "Cannot read property 'numToReport' of undefined" #874

Closed inad9300 closed 4 years ago

inad9300 commented 6 years ago

When running the tests locally I actually don't have any problem, but when Travis CI runs them, the following error raises:

! Cannot read property 'numToReport' of undefined
TypeError: Cannot read property 'numToReport' of undefined
  at Pretty.suiteEnd  <src/lib/reporters/Pretty.ts:162:41>
  at <src/lib/reporters/Reporter.ts:94:31>
  at <src/lib/executors/Executor.ts:302:30>
  at <node_modules/@dojo/core/async/Task.ts:351:12>
  at handler  <node_modules/@dojo/core/async/ExtensiblePromise.ts:229:14>
  at process._tickCallback  <internal/process/next_tick.js:109:7>
! [POST http://localhost:4444/wd/hub/session / {"desiredCapabilities":{"idle-timeout":60,"name":"intern","build":"f735be36d579fa4ff583a7ec9a5f2faa0793c532","browserName":"chrome","chromeOptions":{"args":["headless"]}}}] unknown error: Chrome failed to start: crashed
  (Driver info: chromedriver=2.33.506092 (733a02544d189eeb751fe0d7ddca79a0ee28cce4),platform=Linux 4.14.12-041412-generic x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 60.18 seconds
Build info: version: '3.5.2', revision: '10229a9020', time: '2017-08-21T17:54:21.164Z'
System info: host: 'travis-job-inad9300-soil-333461114', ip: '172.17.0.8', os.name: 'Linux', os.arch: 'amd64', os.version: '4.14.12-041412-generic', java.version: '1.8.0_151'
Driver info: driver.version: unknown
UnknownError: [POST http://localhost:4444/wd/hub/session / {"desiredCapabilities":{"idle-timeout":60,"name":"intern","build":"f735be36d579fa4ff583a7ec9a5f2faa0793c532","browserName":"chrome","chromeOptions":{"args":["headless"]}}}] unknown error: Chrome failed to start: crashed

Here is how my Intern configuration:

{
    "environments": [{
        "browserName": "chrome",
        "chromeOptions": {"args": ["headless"]}
    }],
    "capabilities": {"fixSessionCapabilities": false},
    "loader": {"script": "dojo2"},
    "suites": ["build/**/*.test.js"],
    "coverage": ["build/!(testing)/**/!(*.test).js"],
    "reporters": [{
        "name": "pretty"
    }, {
        "name": "jsoncoverage",
        "options": {"filename": "coverage.json"}
    }]
}
jason0x43 commented 6 years ago

When you run tests locally, are you using the 'selenium' tunnel (the default), or just running them in Node?

inad9300 commented 6 years ago

I simply run intern from package.json, with no arguments, so I suppose I'm using the Selenium tunnel.

inad9300 commented 6 years ago

I'm not yet sure if it's related with Intern, as I removed the "pretty" reporter but still got some unexpected errors from Travis CI. I opened an issue on their side too -- https://github.com/travis-ci/travis-ci/issues/9142. In that case, the error read "Driver info: driver.version: unknown".

inad9300 commented 6 years ago

It finally was not related to Intern. Chrome was not available (not installed, or didn't have enough privileges to run), possibly because some updates they did on the Travis CI side recently.

jason0x43 commented 6 years ago

Ah. I apparently didn't scroll over far enough in the original error message, because it does say unknown error: Chrome failed to start: crashed.

inad9300 commented 6 years ago

Indeed, I also didn't realize initially 😄

inad9300 commented 6 years ago

I have stumbled upon this issue again. This is my configuration:

{
    "environments": ["firefox"],
    "capabilities": {
        "fixSessionCapabilities": "no-detect"
    },
    "suites": ["build/**/*.spec.js"],
    "coverage": ["build/**/!(*.spec).js"],
    "reporters": ["pretty"]
}

And this is the error:

! Cannot read property 'numToReport' of undefined
TypeError: Cannot read property 'numToReport' of undefined
  at Pretty.suiteEnd  <src/lib/reporters/Pretty.ts:162:41>
  at <src/lib/reporters/Reporter.ts:94:31>
  at <src/lib/executors/Executor.ts:302:30>
  at <node_modules/@dojo/core/async/Task.ts:351:12>
  at handler  <node_modules/@dojo/core/async/ExtensiblePromise.ts:229:14>
  at process._tickCallback  <internal/process/next_tick.js:109:7>
! [POST http://localhost:4444/wd/hub/session / {"desiredCapabilities":{"idle-timeout":60,"name":"intern","browserName":"firefox","moz:firefoxOptions":{"args":["headless"]}}}] Unable to create new service: GeckoDriverService
Build info: version: '3.5.2', revision: '10229a9020', time: '2017-08-21T17:54:21.164Z'
System info: host: 'dlaptop', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.0-116-generic', java.version: '1.8.0_151'
Driver info: driver.version: unknown
SessionNotCreatedException: [POST http://localhost:4444/wd/hub/session / {"desiredCapabilities":{"idle-timeout":60,"name":"intern","browserName":"firefox","moz:firefoxOptions":{"args":["headless"]}}}] Unable to create new service: GeckoDriverService
Build info: version: '3.5.2', revision: '10229a9020', time: '2017-08-21T17:54:21.164Z'
System info: host: 'dlaptop', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.0-116-generic', java.version: '1.8.0_151'
Driver info: driver.version: unknown
  at Server.post  <node_modules/src/Server.ts:367:14>
  at Server.createSession  <node_modules/src/Server.ts:412:14>
  at Suite.before  <src/lib/executors/Node.ts:471:8>
  at <src/lib/Suite.ts:388:19>
  at new Task  <node_modules/@dojo/core/async/Task.ts:239:3>
  at runLifecycleMethod  <src/lib/Suite.ts:355:10>
  at before  <src/lib/Suite.ts:458:10>
  at Suite.run  <src/lib/Suite.ts:477:6>
  at <src/lib/executors/Node.ts:799:20>
  at FunctionQueue.next  <src/lib/executors/Node.ts:923:16>

I'm using elementary OS (0.4.1 Loki), and have Firefox 58 installed.

mmarsella commented 6 years ago

I am on OS X 10.11.6 and Chrome v64.0.3282.186. I am also getting the same error:

! Cannot read property 'numToReport' of undefined
TypeError: Cannot read property 'numToReport' of undefined
  at Pretty.suiteEnd  <src/lib/reporters/Pretty.ts:162:41>
  at <src/lib/reporters/Reporter.ts:94:31>
  at <src/lib/executors/Executor.ts:302:30>
  at <node_modules/@dojo/core/async/Task.ts:351:12>
  at handler  <node_modules/@dojo/core/async/ExtensiblePromise.ts:229:14>
  at <anonymous>
  at process._tickCallback  <internal/process/next_tick.js:188:7>

My config for the test is:

{
  "proxyPort": 9000,

  "capabilities": {
    "name": "single_test",
    "build": "intern-browserstack",
    "browserstack.local": false,
    "fixSessionCapabilities": false
  },

  "defaultTimeout": 300000,

  "environments": [
    {
      "browserName": "chrome"
    }
  ],

  "maxConcurrency": 1,

  // "tunnel": "browserstack",

  "tunnelOptions": {
    "verbose": true,
    "username": "myName",
    "accessKey": "myKey"
  },

  "reporters": ["pretty"],

  "loaderOptions": {
    "packages": null
  },

  "suites": null,

  "functionalSuites": ["dist/functional/tests/demo.js"],

  "excludeInstrumentation": true
}

This error only occurs when I comment out "tunnel: "browserstack" / using the default Selenium tunnel.

vladikoff commented 6 years ago

Same here:

! Cannot read property 'numToReport' of undefined
TypeError: Cannot read property 'numToReport' of undefined
  at Pretty.suiteEnd  <src/lib/reporters/Pretty.ts:162:41>
  at <src/lib/reporters/Reporter.ts:94:31>
  at <src/lib/executors/Executor.ts:302:30>
{ Error: One or more suite errors occurred during testing
    at /Users/vladikoff/mozilla/fxa-local-dev/fxa-content-server/node_modules/intern/lib/executors/Executor.js:400:45
    at /Users/vladikoff/mozilla/fxa-local-dev/fxa-content-server/node_modules/@dojo/core/async/Task.js:252:28
    at handler (/Users/vladikoff/mozilla/fxa-local-dev/fxa-content-server/node_modules/@dojo/core/async/ExtensiblePromise.js:146:37)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:228:7) reported: true }
jason0x43 commented 6 years ago

What version of Intern are you using?

Would it be possible for anyone to post a sample project that reproduces the issue? I have so far been unable to reproduce it.

gitgrimbo commented 6 years ago

I have just reproduced the error. Intern 4.3.2.

I attempted to run functional tests on my local PC with IE 11, but the "Protected Mode" was not set to the same value (I think a Windows update reset this setting).

This caused the Unexpected error launching Internet Explorer. Protected Mode settings are not the same for all zones. Enable Protected Mode must be set to the same value (enabled or disabled) for all zones. (WARNING: The server did not provide any stacktrace information) error,

which seems to lead to errors of the form Cannot read property 'xxx' of undefined thrown by Pretty.ts:

node
Tunnel: Ready

Total: Pending
Passed: 0  Failed: 0  Skipped: 0
BUG: suiteEnd was received for invalid session
(ノಠ益ಠ)ノ彡┻━┻
TypeError: Cannot read property 'numToReport' of undefined
  at Pretty.suiteEnd  <src\lib\reporters\Pretty.ts:161:44>
  at <src\lib\reporters\Reporter.ts:90:35>
  at <src\lib\executors\Executor.ts:308:35>
  at <node_modules\@theintern\common\index.js:16:7182>
  at process._tickCallback  <internal\process\next_tick.js:68:7>

(ノಠ益ಠ)ノ彡┻━┻
NoSuchDriver: [POST http://localhost:4444/wd/hub/session / {"desiredCapabilities":{"name":"intern","idle-timeout":60,"browserName":"internet explorer"}}] Unexpected error launching Internet Explorer. Protected Mode settings are not the same for all zones. Enable Protected Mode must be set to the same value (enabled or disabled) for all zones. (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 52 milliseconds
Build info: version: '3.12.0', revision: '7c6e0b3', time: '2018-05-08T15:15:08.936Z'
System info: host: 'RYZEN5', ip: '10.14.10.6', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_171'
Driver info: driver.version: unknown
  at Server.post  <node_modules\src\Server.ts:359:16>
  at Server.createSession  <node_modules\src\Server.ts:404:16>
  at Suite.before  <src\lib\executors\Node.ts:448:15>
  at <src\lib\Suite.ts:384:26>
  at new e  <node_modules\@theintern\common\index.js:16:5076>
  at runLifecycleMethod  <src\lib\Suite.ts:351:13>
  at before  <src\lib\Suite.ts:448:13>
  at Suite.run  <src\lib\Suite.ts:467:10>
! Cannot read property 'numToReport' of undefined
TypeError: Cannot read property 'numToReport' of undefined
  at Pretty.suiteEnd  <src\lib\reporters\Pretty.ts:161:44>
  at <src\lib\reporters\Reporter.ts:90:35>
  at <src\lib\executors\Executor.ts:308:35>
  at <node_modules\@theintern\common\index.js:16:7182>
  at process._tickCallback  <internal\process\next_tick.js:68:7>
! [POST http://localhost:4444/wd/hub/session / {"desiredCapabilities":{"name":"intern","idle-timeout":60,"browserName":"internet explorer"}}] Unexpected error launching Internet Explorer. Protected Mode settings are not the same for all zones. Enable Protected Mode must be set to the same value (enabled or disabled) for all zones. (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 52 milliseconds
Build info: version: '3.12.0', revision: '7c6e0b3', time: '2018-05-08T15:15:08.936Z'
System info: host: 'RYZEN5', ip: '10.14.10.6', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_171'
Driver info: driver.version: unknown
NoSuchDriver: [POST http://localhost:4444/wd/hub/session / {"desiredCapabilities":{"name":"intern","idle-timeout":60,"browserName":"internet explorer"}}] Unexpected error launching Internet Explorer. Protected Mode settings are not the same for all zones. Enable Protected Mode must be set to the same value (enabled or disabled) for all zones. (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 52 milliseconds
Build info: version: '3.12.0', revision: '7c6e0b3', time: '2018-05-08T15:15:08.936Z'
System info: host: 'RYZEN5', ip: '10.14.10.6', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_171'
Driver info: driver.version: unknown
  at Server.post  <node_modules\src\Server.ts:359:16>
  at Server.createSession  <node_modules\src\Server.ts:404:16>
  at Suite.before  <src\lib\executors\Node.ts:448:15>
  at <src\lib\Suite.ts:384:26>
  at new e  <node_modules\@theintern\common\index.js:16:5076>
  at runLifecycleMethod  <src\lib\Suite.ts:351:13>
  at before  <src\lib\Suite.ts:448:13>
  at Suite.run  <src\lib\Suite.ts:467:10>
  at <src\lib\executors\Node.ts:775:27>
  at FunctionQueue.next  <src\lib\executors\Node.ts:899:19>

node
Tunnel: Ready

Total: Pending
Passed: 0  Failed: 0  Skipped: 0

(ノಠ益ಠ)ノ彡┻━┻
TypeError: Cannot read property 'browserName' of undefined
  at Pretty._abbreviateEnvironment  <src\lib\reporters\Pretty.ts:340:45>
  at Pretty._drawSessionReport  <src\lib\reporters\Pretty.ts:318:11>
  at Pretty._render  <src\lib\reporters\Pretty.ts:385:13>
  at Pretty.runEnd  <src\lib\reporters\Pretty.ts:123:9>
  at <src\lib\reporters\Reporter.ts:90:35>
  at <src\lib\executors\Executor.ts:308:35>
  at <node_modules\@theintern\common\index.js:16:7182>
  at process._tickCallback  <internal\process\next_tick.js:68:7>

Stopped

Some console.logging in the transpiled Pretty.js:

BUG: suiteEnd was received for invalid session
suiteEnd suite.id=internet explorer on any platform
_getReport
_getReport suiteOrTest.sessionId="" id=""
_getReport this._reports={}
_getReport this._reports[id]=undefined
_getReport this._reports[id]=new Report {
  numTotal: 0,
  numPassed: 0,
  numFailed: 0,
  numSkipped: 0,
  results: [],
  environment: undefined,
  sessionId: undefined,
  coverageMap: CoverageMap { data: {} },
  suiteInfo: {} }
suiteEnd info=undefined
inad9300 commented 5 years ago

Any chance that you had the time to look into this? Any clues or workarounds?

jason0x43 commented 5 years ago

Sorry, have't gotten to it yet. Just to double check -- are you only seeing the issue when using the Pretty reporter on Travis, or is it happening locally now, too?

inad9300 commented 5 years ago

At the moment I am seeing this problem locally only. I ignored it for a long while because it was working in Travis CI. I came to this issue, however, because Travis CI fails now too, although with a different error message:

Error: Failed to load module intern from /intern.js (parent: build/arch/src/assert.test.js)
  at reportModuleLoadError @ node_modules/@dojo/loader/loader.ts:168:44
  at HTMLScriptElement.handler @ node_modules/@dojo/loader/loader.ts:1080:5

As you point out, the original issue ("Cannot read property 'numToReport' of undefined") seems to be related to the "pretty" reporter. However, I see different errors when trying "simple":

> intern

SUITE ERROR
UnknownError: [POST http://localhost:4444/wd/hub/session / {"desiredCapabilities":{"name":"intern","idle-timeout":60,"browserName":"chrome","goog:chromeOptions":{"args":["headless","disable-gpu"]}}}] unknown error: cannot find Chrome binary
  (Driver info: chromedriver=2.41.578700 (2f1ed5f9343c13f73144538f15c00b370eda6706),platform=Linux 4.15.0-50-generic x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 38 milliseconds
Build info: version: '3.14.0', revision: 'aacccce0', time: '2018-08-02T20:13:22.693Z'
System info: host: 'xxxx', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.15.0-50-generic', java.version: '11.0.3'
Driver info: driver.version: unknown
  at Server.post @ node_modules/src/Server.ts:361:16
  at Server.createSession @ node_modules/src/Server.ts:406:16
  at Suite.before @ src/lib/executors/Node.ts:448:15
  @ src/lib/Suite.ts:431:26
  at new e @ node_modules/@theintern/common/index.js:16:5076
  at runLifecycleMethod @ src/lib/Suite.ts:398:13
  at before @ src/lib/Suite.ts:495:13
  at Suite.run @ src/lib/Suite.ts:514:10
  @ src/lib/executors/Node.ts:769:27
  at FunctionQueue.next @ src/lib/executors/Node.ts:893:19
UnknownError: [POST http://localhost:4444/wd/hub/session / {"desiredCapabilities":{"name":"intern","idle-timeout":60,"browserName":"chrome","goog:chromeOptions":{"args":["headless","disable-gpu"]}}}] unknown error: cannot find Chrome binary
  (Driver info: chromedriver=2.41.578700 (2f1ed5f9343c13f73144538f15c00b370eda6706),platform=Linux 4.15.0-50-generic x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 38 milliseconds
Build info: version: '3.14.0', revision: 'aacccce0', time: '2018-08-02T20:13:22.693Z'
System info: host: 'xxxx', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.15.0-50-generic', java.version: '11.0.3'
Driver info: driver.version: unknown
  at Server.post @ node_modules/src/Server.ts:361:16
  at Server.createSession @ node_modules/src/Server.ts:406:16
  at Suite.before @ src/lib/executors/Node.ts:448:15
  @ src/lib/Suite.ts:431:26
  at new e @ node_modules/@theintern/common/index.js:16:5076
  at runLifecycleMethod @ src/lib/Suite.ts:398:13
  at before @ src/lib/Suite.ts:495:13
  at Suite.run @ src/lib/Suite.ts:514:10
  @ src/lib/executors/Node.ts:769:27
  at FunctionQueue.next @ src/lib/executors/Node.ts:893:19

Honestly, the whole Intern experience it's been a total pain in the ass from the beginning. I wanted to stick with it because the idea of it looked good, but I think I cannot stand it anymore. Without maturity and without stability, there is no point on having many amazing features.

jason0x43 commented 5 years ago

After looking into this a bit more, the pretty reporter error is a bit of a red herring. It's throwing an error because it's being asked to handle a suiteEnd event before a session has actually been created. The base problem is that in most cases (maybe all of them) the browser isn't starting up:

So the core issue is that the browser isn't able to start for some reason, and the Pretty reporter just isn't handling that gracefully (or at least it's not being particularly helpful).

As to why the browser is failing to start,

In any case, the reporter should be handling the failure more gracefully.

inad9300 commented 5 years ago

@jason0x43 I am sorry for my late-night mini-rant. Once you think to have tried everything you consider reasonable, you just go mad.

In my opinion, both error messages and the documentation has plenty of room for improvement. (As a suggestion, error messages should when possible point to the right section of the documentation.) The thing is, when trying to make things "just work", I'm flooded with questions which are hard or impossible to answer with the documentation, such as:

Perhaps my expectations were misguided. What I was expecting Intern to be able to do (i.e. the reason I chose it, and I believe I'm not alone) is to run tests in a browser that I can choose from a given list of supported browsers, both locally and in a CI environment. Knowing some of the details of how it works can interest me as a developer, but I don't want to be forced to care about them while acting as a mere user of the framework. That is, I expect to be able to say "firefox", and have the tests run in Firefox. And if some extra step is required in order to make this work (e.g. installing Firefox or setting a specific driver version), I expect to see what those extra steps are as Intern's output.

One last piece of feedback that I want to communicate is that Intern's configuration flexibility is no good at all. Of course, this might be only my personal opinion, but I have the impression that it is mentioned as a good feature in the docs, but the effect it has on me is no other than creating confusion. There should be a place for each property, not one thousand. I constantly ask myself, "is this the right place? will Intern be smart enough to pick it up?" Or if I'm copying some config from the web, "is this according to the latest Intern version? could this person have made a mistake?" And it really is not possible to answer these questions other than by trial and error, which is not a very rewarding way to work. (As a suggestion for this, it would be awesome to take forward TypeScript's support and allow defining the config in a "intern.ts" file, where you can have typed configuration.)

For the sake of completion (in case someone is browsing the web for this error message), here is what I see when trying to use "firefox" as the browser:

> intern

SUITE ERROR
SessionNotCreatedException: [POST http://localhost:4444/wd/hub/session / {"desiredCapabilities":{"name":"intern","idle-timeout":60,"browserName":"firefox","goog:chromeOptions":{"args":["headless","disable-gpu","window-size=1024,768"]},"moz:firefoxOptions":{"args":["-headless","--window-size=1024,768"]}}}] Unable to create new service: GeckoDriverService
Build info: version: '3.14.0', revision: 'aacccce0', time: '2018-08-02T20:13:22.693Z'
System info: host: 'xxxx', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.15.0-50-generic', java.version: '11.0.3'
Driver info: driver.version: unknown
  at Server.post @ node_modules/src/Server.ts:361:16
  at Server.createSession @ node_modules/src/Server.ts:406:16
  at Suite.before @ src/lib/executors/Node.ts:448:15
  @ src/lib/Suite.ts:431:26
  at new e @ node_modules/@theintern/common/index.js:16:5076
  at runLifecycleMethod @ src/lib/Suite.ts:398:13
  at before @ src/lib/Suite.ts:495:13
  at Suite.run @ src/lib/Suite.ts:514:10
  @ src/lib/executors/Node.ts:769:27
  at FunctionQueue.next @ src/lib/executors/Node.ts:893:19
SessionNotCreatedException: [POST http://localhost:4444/wd/hub/session / {"desiredCapabilities":{"name":"intern","idle-timeout":60,"browserName":"firefox","goog:chromeOptions":{"args":["headless","disable-gpu","window-size=1024,768"]},"moz:firefoxOptions":{"args":["-headless","--window-size=1024,768"]}}}] Unable to create new service: GeckoDriverService
Build info: version: '3.14.0', revision: 'aacccce0', time: '2018-08-02T20:13:22.693Z'
System info: host: 'xxxx', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.15.0-50-generic', java.version: '11.0.3'
Driver info: driver.version: unknown
  at Server.post @ node_modules/src/Server.ts:361:16
  at Server.createSession @ node_modules/src/Server.ts:406:16
  at Suite.before @ src/lib/executors/Node.ts:448:15
  @ src/lib/Suite.ts:431:26
  at new e @ node_modules/@theintern/common/index.js:16:5076
  at runLifecycleMethod @ src/lib/Suite.ts:398:13
  at before @ src/lib/Suite.ts:495:13
  at Suite.run @ src/lib/Suite.ts:514:10
  @ src/lib/executors/Node.ts:769:27
  at FunctionQueue.next @ src/lib/executors/Node.ts:893:19
inad9300 commented 5 years ago

After trying the "tunnelOptions" you suggested, I see the following error:

> intern

SUITE ERROR
UnknownError: [POST http://localhost:4444/wd/hub/session / {"desiredCapabilities":{"name":"intern","idle-timeout":60,"browserName":"chrome","goog:chromeOptions":{"args":["headless","disable-gpu","window-size=1024,768"]},"moz:firefoxOptions":{"args":["-headless","--window-size=1024,768"]}}}] unknown error: DevToolsActivePort file doesn't exist
  (Driver info: chromedriver=74.0.3729.6 (255758eccf3d244491b8a1317aa76e1ce10d57e9-refs/branch-heads/3729@{#29}),platform=Linux 4.15.0-50-generic x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 60.08 seconds
Build info: version: '3.14.0', revision: 'aacccce0', time: '2018-08-02T20:13:22.693Z'
System info: host: 'xxxx', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.15.0-50-generic', java.version: '11.0.3'
Driver info: driver.version: unknown
  at Server.post @ node_modules/src/Server.ts:361:16
  at Server.createSession @ node_modules/src/Server.ts:406:16
  at Suite.before @ src/lib/executors/Node.ts:448:15
  @ src/lib/Suite.ts:431:26
  at new e @ node_modules/@theintern/common/index.js:16:5076
  at runLifecycleMethod @ src/lib/Suite.ts:398:13
  at before @ src/lib/Suite.ts:495:13
  at Suite.run @ src/lib/Suite.ts:514:10
  @ src/lib/executors/Node.ts:769:27
  at FunctionQueue.next @ src/lib/executors/Node.ts:893:19
UnknownError: [POST http://localhost:4444/wd/hub/session / {"desiredCapabilities":{"name":"intern","idle-timeout":60,"browserName":"chrome","goog:chromeOptions":{"args":["headless","disable-gpu","window-size=1024,768"]},"moz:firefoxOptions":{"args":["-headless","--window-size=1024,768"]}}}] unknown error: DevToolsActivePort file doesn't exist
  (Driver info: chromedriver=74.0.3729.6 (255758eccf3d244491b8a1317aa76e1ce10d57e9-refs/branch-heads/3729@{#29}),platform=Linux 4.15.0-50-generic x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 60.08 seconds
Build info: version: '3.14.0', revision: 'aacccce0', time: '2018-08-02T20:13:22.693Z'
System info: host: 'xxxx', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.15.0-50-generic', java.version: '11.0.3'
Driver info: driver.version: unknown
  at Server.post @ node_modules/src/Server.ts:361:16
  at Server.createSession @ node_modules/src/Server.ts:406:16
  at Suite.before @ src/lib/executors/Node.ts:448:15
  @ src/lib/Suite.ts:431:26
  at new e @ node_modules/@theintern/common/index.js:16:5076
  at runLifecycleMethod @ src/lib/Suite.ts:398:13
  at before @ src/lib/Suite.ts:495:13
  at Suite.run @ src/lib/Suite.ts:514:10
  @ src/lib/executors/Node.ts:769:27
  at FunctionQueue.next @ src/lib/executors/Node.ts:893:19
inad9300 commented 5 years ago

OK, after installing Google Chrome on my machine, I see the same error as the one in Travis CI, which looks like some progress to me:

> intern

SUITE ERROR
Error: Failed to load module intern from /intern.js (parent: build/arch/src/assert.test.js)
  at reportModuleLoadError @ node_modules/@dojo/loader/loader.ts:168:44
  at HTMLScriptElement.handler @ node_modules/@dojo/loader/loader.ts:1080:5
0/0 tests failed in chrome 74.0.3729.157 on Linux

Edit I confirm the exact same problem occurs with @dojo/loader 0.1.0, 0.1.1 and 2.0.0.

inad9300 commented 5 years ago

Interestingly, when I run the tests as "functionalSuites" as opposed to "suites", many of them run, meaning the loader seems to be able to find "intern" in this case. Other tests can't pass because they depend on a global document variable being available, but that makes sense (perhaps there is a trick to make document available with this setting?). Hopefully this helps...

inad9300 commented 5 years ago

Holy heck, I can't believe I made it "work"!

I removed import 'intern' from all test files, and this way it works... I don't understand why it stopped working, though. However, this is not an ideal solution, because VS Code complains that Cannot find name 'intern'.ts(2304), and shows every Intern variable as having type any, giving no auto-completion support. This is despite having "types": ["intern"] in tsconfig.json. Nevertheless, when purposefully making a mistake, the compiler catches it.

I am happy enough with the current state of things, but suggestions to make VS Code aware of Intern's types are very much welcomed.

And Firefox continues not to work. This was only with "chrome".

jason0x43 commented 5 years ago

In my opinion, both error messages and the documentation has plenty of room for improvement.

You're not wrong.

I don't even have Chrome installed on my machine, but do I need to? Does Intern come with one? If so, why would I need "tunnel options" (whatever that means...) to make it work?

Intern is a tool that runs tests in Node or in browsers. It can manage browsers (opening and closing them and loading pages) using WebDriver, or you can open a browser window yourself and point it at Intern. In either case, though, Intern can only work with browsers that are available, it doesn't include one. (That would likely only work for Chrome and Firefox, and Intern attempts to be browser-agnostic as much as possible.)

If I want to use Chromium, can I? What do I need to do for this? What browser name must I use? Will this work in a CI environment?

Hmmm...that's a good question. You'd need to specify the location to Chromium on your system using the "binary" capability (that kind of thing is where WebDriver can be a pain to configure), something like:

"environments": {
  "browserName": "chrome",
  "goog:chromeOptions": {
    "binary": "/Applications/Chromium.app/Contents/MacOS/Google Chrome"
  }
}

Since you have to specify the binary location, you may need to a different config on your local system vs a CI system (unless Chromium is in the same location).

That is, I expect to be able to say "firefox", and have the tests run in Firefox. And if some extra step is required in order to make this work (e.g. installing Firefox or setting a specific driver version), I expect to see what those extra steps are as Intern's output.

That is a goal, although there are limitations to what Intern can easily do. For example, if you specify "firefox", Intern doesn't know what version of Firefox you're using. That version affects which version of geckodriver (Firefox's WebDriver server) you need. Using an invalid webdriver version might work, or it might partially work, or it might fail without providing a helpful error. Intern can certainly do better about informing the user of what's going on, but at some point a user will likely have to be involved.

One last piece of feedback that I want to communicate is that Intern's configuration flexibility is no good at all.

Intern's configuration system is declarative so that it can work work on Node and in browsers, and be easily serialized and transmitted from Node to remote browsers. A side benefit of this is that configuration properties can be provided on the command line or as query args, or specified in a config file, all using the same property names and with the same data format.

JSON isn't as flexible as an imperative script would be, but on the other hand it's easier to serialize and it avoids footguns like trying to run a config written in ES2015 on a browser that doesn't support it like IE.

I constantly ask myself, "is this the right place? will Intern be smart enough to pick it up?"

I'm not really sure what is meant here. Intern has a well defined config format, and config options will only be used when they're in the right place. That said, I agree that having some way of validating a config would be very helpful. Someone started writing a JSON schema for Intern's config format at one point, although I don't believe it was ever completed. Adding such a schema to Intern's source would definitely help out.

Or if I'm copying some config from the web, "is this according to the latest Intern version?

If it's JSON, it should be current (or current enough). Intern prior to v4 (which came out a couple of years ago) used AMD modules for config.

And it really is not possible to answer these questions other than by trial and error, which is not a very rewarding way to work. (As a suggestion for this, it would be awesome to take forward TypeScript's support and allow defining the config in a "intern.ts" file, where you can have typed configuration.)

I totally agree. Better documentation and a JSON schema would be very helpful here.

Interestingly, when I run the tests as "functionalSuites" as opposed to "suites", many of them run, meaning the loader seems to be able to find "intern" in this case.

Intern supports two types of test: functional and unit. Functional tests always run in Node and remotely drive a browser (tell it to go to addresses and click elements and such). Unit tests run directly in Node or a browser. Which property your test be listed in (functionalSuites or suite) depends on what type of test it is -- is it using remote to send commands to a browser, or is it just JavaScript code that is being directly run in a browser?

Other tests can't pass because they depend on a global document variable being available, but that makes sense (perhaps there is a trick to make document available with this setting?). Hopefully this helps...

If the test needs document it's presumably a unit test, and it should be run in a browser. In that case it should be listed in suites and should actually run in a browser. To run that kind of test in Node you'd need to use something like jsdom, or mock out the parts of document needed in the tests.

Having this distinction between test types does make things a bit complex, but it's not done without reason. Test modules that use Node modules or are written in ES6 generally can't be loaded directly in a browser, while tests that access browser resources may not load or execute in Node.

I removed import 'intern' from all test files, and this way it works... I don't understand why it stopped working, though.

Hmmm...I'm not really sure about this. VSCode should be happy enough with "types": ["intern"]. It would really be helpful (or at least I could give more useful input) if you could post your project, or a simplified version that contains the config and a few representative tests, to a gist or repo somewhere.

inad9300 commented 5 years ago

Thanks a lot for taking the time to reply to my comments. A couple of clarifications:

I'm not really sure what is meant here. Intern has a well defined config format, and config options will only be used when they're in the right place.

When I referred to Intern configuration's excessive flexibility, I was referring to the fact that there is a "normalization" process. For example, you can write "environments": "chrome", but also "environments": ["chrome"], and "environments": {"browserName": "chrome"}, and "environments": [{"browserName": "chrome"}]. The problem with this is that you haven't got a clue about which properties will be normalized and to which degree. Explaining in greater detail the normalization process for each property would help, but I see it as completely unnecessary.

Not sure if you would consider this part of the normalization, but an even bigger part of the problem is the unpredictable inheritance model that there is in place. For instance, there is a "capabilities" property which somehow will affect the "environments" part of the config, god knows how. And I have the feeling that this is the case for many other sets of properties, despite their names not indicating this in any way. If you were to just add support for configuration files written in JavaScript (possibly already exists? Hard to find), users would be able to put in simple variables whatever they want to avoid repeating, this way having full control over what's going on.

This whole process has no measurable benefit, but adds a great deal of confusion for users, and unnecessary complexity to the project.

Intern supports two types of test: functional and unit. Functional tests always run in Node and remotely drive a browser (tell it to go to addresses and click elements and such). Unit tests run directly in Node or a browser.

I know about the conceptual difference between the two, but we should not be expected to know about the implementation details of the two. It simply surprised me to see that one of them was able to deal with import "intern", while the other one didn't and gave an unrelated error message. It seems to me that this is a limitation of the so-called "loader"; and a bit of a contradiction to "TypeScript support", i.e. TypeScript is supported, but a valid TypeScript file doesn't run.

Hmmm...I'm not really sure about this. VSCode should be happy enough with "types": ["intern"]. It would really be helpful (or at least I could give more useful input) if you could post your project, or a simplified version that contains the config and a few representative tests, to a gist or repo somewhere.

The project I'm using Intern in is https://github.com/inad9300/Soil. If you check it out and open any *.test.ts file with VS Code, you should see that intern is underlined in red. If you look into it, thank you in advance.

jason0x43 commented 5 years ago

When I referred to Intern configuration's excessive flexibility, I was referring to the fact that there is a "normalization" process.

Ah, I see what you mean now. That behavior was intended to be a convenience, mostly to handle input from the command line (so, for example, you could type environments=chrome to do a one-off run in Chrome rather than having to type environments=[{"browserName":"chrome"}]). It could definitely stand to be more clearly documented, though.

For instance, there is a "capabilities" property which somehow will affect the "environments" part of the config, god knows how.

That's actually the weirdest one of those, and is a bit of a historical artifact. "capabilities" is what Selenium / WebDriver calls the config you're asking for from a Selenium server. "environments" is Intern's property for specifying which environments you want your test to run in. In Intern's config, capabilities are the baseline capabilities that will be applied to all remote sessions, and environments describes what the sessions will be. At test time, the value of capabilities is mixed into each of the environments, and each result is sent to a Selenium server as a capabilities descriptor.

That's definitely something else that could use more documentation, or a clearer structure.

If you were to just add support for configuration files written in JavaScript (possibly already exists? Hard to find), users would be able to put in simple variables whatever they want to avoid repeating, this way having full control over what's going on.

I'm not really clear how that would help (at least not for this particular issue). I mean, the config structure is what it is, so whether you're specifying it in a JSON object, or constructing that config object using JS, or setting individual config properties with JS, you'll still need some knowledge about what the config property values should be. There are definitely cases where allowing a JS config file would be useful, but it would also introduce problems that might not make for a worthwhile tradeoff.

It simply surprised me to see that one of them was able to deal with import "intern", while the other one didn't and gave an unrelated error message. It seems to me that this is a limitation of the so-called "loader"; and a bit of a contradiction to "TypeScript support", i.e. TypeScript is supported, but a valid TypeScript file doesn't run.

I'll have to take a look at your project to know what's going on there, but it likely is loader-related. For an import statement to work, whether you're using Intern or simply loading your own code, you need to be using a loader such as requirejs, the Dojo loader, the Node loader, webpack, a browser-native module loader, etc. Module imports always work with functional tests since they run in Node, which has a native loader. Module imports for unit tests running in Node will always work. Module imports for unit tests running in a browser will typically not work out-of-the-box; you have to provide a loader. This isn't a limitation of Intern, it's just how JS modules work.

The project I'm using Intern in is https://github.com/inad9300/Soil.

I'll take a look.

jason0x43 commented 5 years ago

@inad9300 I took a look at your project. It reminded me a bit of Polymer's lit-element and lit-html which I was just reading about yesterday. :)

In any case, here are some things I noticed.

Config

The environments block should be an array when you're describing more than one environment:

"environments": [
  {
    "browserName": "chrome",
    "goog:chromeOptions": {
      "args": ["headless", "disable-gpu", "window-size=1024,768"]
    }
  },
  {
    "browserName": "firefox",
    "moz:firefoxOptions": {
      "args": ["-headless", "--window-size=1024,768"]
    }
  }
],

If you want to set fixSessionCapabilities to false or no-detect for one browser, do that in its environment specifier. To set it for all browsers, use capabilities.

"environments": [
  {
    "browserName": "chrome",
    "fixSessionCapabilities": "no-detect"
  }
]

or

"capabilities": {
  "fixSessionCapabilities": "no-detect"
}

Also, since you're using Firefox, you'll need to tell Intern (actually the SeleniumTunnel in Leadfoot, which is Intern's WebDriver library) to download the Firefox driver.

"tunnelOptions": {
  "drivers": ["firefox", "chrome"]
}

(I know, having to specify the drivers and browsers separately seems terribly duplicative. It's historical, and that requirement will be removed at some point.)

Typings

The problem with typings is that VScode is using the default project config (tsconfig.json) when you're editing the test files. However, the tests are explicitly excluded from that config, so the compiler options (like "types": ["intern"]) aren't being applied.

To get around that you have a couple of options. One is to let the base tsconfig.json include all source files and specify common compilerOptions, and then create a separate tsconfig.src.json for building the source (just like you have tsconfig.tests.json for building the tests).

Another option is to include a type reference at the top of each of your test files, like:

/// <reference types="intern" />

This is preferable to import 'intern' because it only loads types. TS will turn the import 'intern' statement into an actual module load, which isn't what you want in this case.

Speaking of module loaders, I did see that you were using a module loader. The issue you were running into is that AMD loaders such as the Dojo loader or requireJS require some configuration; for example, you'd need to tell the loader where various packages are (like Intern). Intern's own self tests currently use the Dojo loader, and have this sort of config. In this case it's much easier to just reference the types, though, rather than to configure the loader to make an unnecessary module load.

inad9300 commented 5 years ago

Amazing feedback, thank you so much... I'm much more clear about the configuration after our discussion. And thanks for the reminder about VS Code picking only the tsconfig.json at the root! I wasn't thinking of this at all.

It is definitely similar to lit-element and lit-html. Funny enough, I had this same observation and mentioned it to them in https://github.com/Polymer/lit-html/issues/700 (although it was kindly ignored, which I totally understand).

jason0x43 commented 4 years ago

Closing this. I created #1002 to track improvements to the Pretty reporter.