Add tests

[steveszc]

This is a little tricky to test since the system under test is ember's test harness itself. Need to put some thought into how best test that the memory leak detection is actually working.

Ideally we could separately run ember test --filter="nonleaky" and ember test --filter="leaky" and assert that the nonleaky filter does not report a memory leak, and that the leaky filter does report a memory leak.

Unclear how best to accomplish this though.

[jelhan]

I recently created ember-addon-tests to help writing these kind of tests for Ember addons. It's very alpha. Only using it in ember-cli-content-security-policy so far, which was also motivating me to write it.

The idea itself isn't that new. A similar project exists since years: ember-cli-addon-tests But it's architecture got outdated and isn't used much.

Other addons, which need the same kind of tests, are using a monorepository including dozens packages only for testing. Famous examples are Ember Auto Import and Embroider.

The idea between both approaches is similar. Main difference is that with Ember Addon Tests the Ember addons to run the tests against are created in the test suite while Ember Auto Import and Embroider include them as fixtures in the repository.

[jelhan]

To get started quickly with some tests instrumenting ember test --filter="${scenario}" in a test suite build with mocha or jest sounds like a good idea.

import execa from 'execa';

test('test fail if triggering a memory leaks', async function() {
  try {
    await execa('ember', ['test', 'filter="leaky"']);

    // ember test should have thrown
    expect(true).toBe(false);
  } catch ({ code }) {
    // expect status code, stdout, ...
    expect(code).to.be(1);
  }
});

test('test pass if no memory leak is present', async function() {
   let { code } = await execa('ember', ['test', 'filter="nonleaky"']);
   expect(code).toBe(0);
});

To optimize performance you may want to do a build with ember build --environment test before and reuse it in all runs of ember test with --path option.

This approach should work out fine as long as you don't need to run test depending on a specific setup of the consuming Ember project.

[steveszc]

@jelhan Amazing! Thanks for your input here.

[steveszc]

@jelhan I was able to add some basic tests following your mocha example, and also got CI set up via github actions. Thanks! Definitely interested in using ember-addon-tests in the future.