`ember test --environment=production` fails on newly generated app

[kategengler]

ember test --environment=production fails on newly generated app using ember-cli 2.12.0 (also observed on ember-cli 2.11.1)

First checked that ember test passes as expected.

$ ember new test-this-app
$ cd test-this-app
$ ember test
cleaning up...
Built project successfully. Stored in "/Users/katie/dev/test-new-app/tmp/class-tests_dist-kzna4Pun.tmp".
ok 1 Chrome 57.0 - ESLint - app.js: should pass ESLint
ok 2 Chrome 57.0 - ESLint - helpers/destroy-app.js: should pass ESLint
ok 3 Chrome 57.0 - ESLint - helpers/module-for-acceptance.js: should pass ESLint
ok 4 Chrome 57.0 - ESLint - helpers/resolver.js: should pass ESLint
ok 5 Chrome 57.0 - ESLint - helpers/start-app.js: should pass ESLint
ok 6 Chrome 57.0 - ESLint - resolver.js: should pass ESLint
ok 7 Chrome 57.0 - ESLint - router.js: should pass ESLint
ok 8 Chrome 57.0 - ESLint - test-helper.js: should pass ESLint

1..8
# tests 8
# pass  8
# skip  0
# fail  0

# ok

Then tried ember test --environment=production, which has failures and hangs with the following output:

cleaning up...
Built project successfully. Stored in "/Users/katie/dev/test-new-app/tmp/class-tests_dist-WUxgPuKp.tmp".
not ok 1 PhantomJS 2.1 - Global error: TypeError: undefined is not an object (evaluating 't.default.Test.Adapter') at http://localhost:7357/assets/test-support-440a62dc7b8a879f77781fdf30e7c2da.js, line 2
    ---
        Log: |
            { type: 'error',
              text: 'TypeError: undefined is not an object (evaluating \'t.default.Test.Adapter\') at http://localhost:7357/assets/test-support-440a62dc7b8a879f77781fdf30e7c2da.js, line 2\n' }
    ...
not ok 2 PhantomJS 2.1 - Global error: TypeError: undefined is not an object (evaluating 't.default.Test.Adapter') at http://localhost:7357/assets/test-support-440a62dc7b8a879f77781fdf30e7c2da.js, line 2
    ---
        Log: |
            { type: 'warn',
              text: '\'jQuery.Deferred exception: undefined is not an object (evaluating \\\'t.default.Test.Adapter\\\')\' \'http://localhost:7357/assets/test-support-440a62dc7b8a879f77781fdf30e7c2da.js:2:22150\\nexports@http://localhost:7357/assets/vendor-0f35ceb1de77e7f57aced7f8fc9e4b4a.js:1:2820\\n_reify@http://localhost:7357/assets/vendor-0f35ceb1de77e7f57aced7f8fc9e4b4a.js:1:3368\\nreify@http://localhost:7357/assets/vendor-0f35ceb1de77e7f57aced7f8fc9e4b4a.js:1:3152\\nexports@http://localhost:7357/assets/vendor-0f35ceb1de77e7f57aced7f8fc9e4b4a.js:1:2792\\nrequireModule@http://localhost:7357/assets/vendor-0f35ceb1de77e7f57aced7f8fc9e4b4a.js:1:2100\\nhttp://localhost:7357/assets/test-support-440a62dc7b8a879f77781fdf30e7c2da.js:2:38769\\np@http://localhost:7357/assets/vendor-0f35ceb1de77e7f57aced7f8fc9e4b4a.js:2:26138\\nhttp://localhost:7357/assets/vendor-0f35ceb1de77e7f57aced7f8fc9e4b4a.js:2:39876\' null\n' }
            { type: 'error',
              text: 'TypeError: undefined is not an object (evaluating \'t.default.Test.Adapter\') at http://localhost:7357/assets/test-support-440a62dc7b8a879f77781fdf30e7c2da.js, line 2\n' }
    ...
ç^C%

I pushed the generated app here https://github.com/kategengler/ember-cli-2.12-generated-app for convenience.

---

Output from ember version --verbose && npm --version:

ember-cli: 2.12.0
http_parser: 2.7.0
node: 7.6.0
v8: 5.5.372.40
uv: 1.11.0
zlib: 1.2.11
ares: 1.10.1-DEV
modules: 51
openssl: 1.0.2k
icu: 58.2
unicode: 9.0
cldr: 30.0.3
tz: 2016j
os: darwin x64
4.1.2

[rwjblue]

Awesome, thank you for reporting. This might be a duplicate, but I vaguely recall the other issue being in ember-qunit or ember-mocha repo. I'll try to dig into this this weekend...

[trentmwillis]

@rwjblue last related update: https://github.com/ember-cli/ember-cli-qunit/issues/169#issuecomment-284129257

Seems like we need to include ember-testing.js in production test builds

[trentmwillis]

I began digging into this as I thought you should be able to workaround the issue with something like:

// ember-cli-build.js
app.import(app.bowerDirectory + '/ember/ember-testing.js', { type: 'test', prepend: true });

But this does not work, because ember-testing.js does not properly add things to the global Ember object. It requires the ember-testing module but doesn't assign the exports to anything.

In other words, it does this:

require('ember-testing');

when it needs to do this:

let testing = require('ember-testing');

Ember.Test = testing.Test;
Ember.Test.Adapter = testing.Adapter;
Ember.Test.QUnitAdapter = testing.QUnitAdapter;
Ember.setupForTesting = testing.setupForTesting;

Basically, it is missing this section of code.

I feel like this should be fixed upstream in ember.js, so that ember-testing.js can be used properly, but that won't fix previous releases. Another alternative is to write a shim that does the proper thing with ember-testing.js.

[rwjblue]

@trentmwillis - Yes, agreed

[trentmwillis]

I've opened a PR to fix part of this upstream: https://github.com/emberjs/ember.js/pull/15112 However, there will also need to be a fix in Ember-CLI since the path to ember-testing.js currently only works with the Bower distribution of Ember.

[trentmwillis]

Fix for including ember-testing when using ember-source: https://github.com/ember-cli/ember-cli/pull/6944.

After these fixes, ember-testing loads, but it errors out. I'm investigating why, looks to be related to feature flagging in Ember itself. Nevermind, I guess I had something borked with all my npm links.

[allthesignals]

I'm not sure if these is related but in ember-cli-qunit 4.0.0, I see a global failure:

beforeEach failed on global failure: Cannot read property 'setContext' of null

TypeError: Cannot read property 'setContext' of null
    at Object.beforeEach (http://localhost:7357/assets/test-support.js:8325:16)
    at callHook (http://localhost:7357/assets/test-support.js:4072:25)
    at runHook (http://localhost:7357/assets/test-support.js:4095:7)
    at Object.advance (http://localhost:7357/assets/test-support.js:3691:26)
    at Test.queue (http://localhost:7357/assets/test-support.js:4264:22)
    at test (http://localhost:7357/assets/test-support.js:4498:12)
    at Object.onError (http://localhost:7357/assets/test-support.js:5212:5)
    at window.onerror (http://localhost:7357/assets/test-support.js:6558:18)

Uncaught TypeError: Cannot read property 'pushResult' of undefined

Among others...

It's hard to understand where this is coming from - the tests all pass in headless Chrome.

[robydrupo]

Our pipeline is building first with --environment=production but then we have to rebuild again to execute our tests as part of a following step. Is there a workaround to avoid the second build and execute tests on a production build @trentmwillis at this point?

[trentmwillis]

@robydrupo I don't believe there is anyway to do a "tests-only" build (which is what it sounds like you want).

I'm going to close this issue as I believe the original problem has been solved (as noted in this comment). Other problems should be tracked in different issues.