Use consistent language when referring to acceptance tests in docs

[trabus]

Currently the docs use 'acceptance' and 'integration' interchangeably, which can lead to confusion. Originally, the default was 'integration', but it was proposed by @stefanpenner to change the built-in test blueprints from integration-test to acceptance-test, which is a more accurate description of the role they play as described in the docs. Unfortunately, the docs still have 14 instances of 'integration' vs 11 instances of 'acceptance', which makes things even more confusing because it makes it seem that integration should be the preferred language, but you can't generate an integration-test.

Also, as Ember 2.0 looms there will be more need for components to test integration between each other, and we'd likely want to use the more accurate 'integration-test' when referring to such tests. In such a case, having our language clear now would ease introduction of that concept.

TLDR - Use 'acceptance' exclusively moving forward when referring to app level tests, and update the docs to clarify the language.

[stefanpenner]

@trabus :+1: got time to push this forward?

[trabus]

@stefanpenner Absolutely. Will try and find some time later this week.

[trabus]

@stefanpenner So I decided to do some research tonight to make sure I'm not confusing things more when I make the changes, and I found something that gives me pause. Looking through the ember testing docs, they exclusively refer to 'integration' tests. I'm definitely of the opinion that the docs should refer to 'acceptance' tests, but I'm afraid that completely removing 'integration' from the language at this point would confuse things further. Maybe a note making the distinction would suffice, but I think it's an important detail to cover somehow.

I think eventually unifying the testing language across the board is necessary (and I'm sure already planned), especially if the ember docs will eventually be the final home for that part of the docs. I'm just not sure what the path to get there is. I guess I'm looking for any guidance on plans or where these docs should be going. I don't want to make things worse!

[stefanpenner]

the ember test docs are wrong, and ultimately confusing on this topic. I would prefer if we label them correctly, and then move efforts to improve them on the ember site as well.

for the purpose of discussion, my thoughts are as follows:

In order to communicate about these overlapping topics, we need a common language to describe and differentiate them.

• unit: testing algorithmic complexity, of individual units without additional collaborates
• integration: testing the interaction between multiple units (but still at a level that understands the implementation)
• acceptance: mimic the user, black box testing of the application, often tied directly to high level business use-cases.

[tomdale]

@trabus I'm happy to accept a PR for the Ember docs that unifies terminology across projects.

[trabus]

@tomdale Thanks, I'm glad to do it as long as it has support from the core team. I don't want to be the dude dropping in out of nowhere changing up the "generally accepted" (albeit technically incorrect) terminology that has been around for more than a year. Try googling for ember testing and you'll see integration and acceptance used interchangeably, with integration being the more common term used for what is actually acceptance testing. Obviously some tactful explanation will be in order, as some people tend to have opinions on testing.

Anyhow, I will likely take a crack at it in a week or so, starting with the ember-cli docs. If anyone else is interested in doing it, I wouldn't mind either. I'm trying to finish up work on my pod migration addon. :shipit:

[stefanpenner]

@trabus i believe we have core consensus on this topic. I have been advocating for this clarification for some time and have received favorable feedback, so I got your back if someone has concerns.

[jgwhite]

@trabus thanks for leading the way on this :grinning:

[trabus]

@jgwhite and I were talking about this today, and came to the consensus that an important part of this story will be to provide some concrete examples of what an integration test looks like in ember-cli, as well as creating a blueprint for integration tests. Just wanted to make a note so I don't forget.

[trabus]

I started work on this, but I'm a bit stuck trying to identify exactly what an integration test in ember-cli looks like. To be honest, now that I've started reading up on exactly what defines each type of test, I'm not quite sure I have a clear definition of unit tests in ember-cli either.

Unit tests in ember-cli seem a bit fuzzy in definition since many require dependencies pulled in through needs. Using needs pulls in dependencies that can change and doesn't provide the stability of using stubs. I'm guessing this is more a tradeoff since creating stubs for something complex can be time consuming. Still, this makes any test that uses needs look more like an integration test to me. Maybe we could recommend making use of sinon.js or something like it to provide more control for unit testing?

Regarding needs, does it allow proper setup for an integration test in ember-cli? Is it good enough to have each module doing a top down integration test, and does that provide the control to inspect side effects in the dependencies? Or should an integration test allow you to identify the modules brought into the test as individual units (something like moduleFor, but takes a hash or array of modules to instantiate. modulesFor?), and then allow you to hook them together in your test setup? Something like that would allow you to make use of stubs and spies in between the units, though I'm not sure exactly how it would look.

Anyhow, just some questions I'm a bit unclear on. I don't have a ton of experience with testing yet and I just want to make sure I'm understanding this all properly before I write about it further. Thanks!

[stefanpenner]

To be honest, now that I've started reading up on exactly what defines each type of test, I'm not quite sure I have a clear definition of unit tests in ember-cli either.

• acceptance test: you app is a black box, simulate the outside world. visit/click/fillIn/pretender
• unit test: testing algorithimic complexity one collaborator at a time, no additional collaborators.
• integration test: testing the interaction of multiple units

Unit tests in ember-cli seem a bit fuzzy in definition since many require dependencies pulled in through needs.

basically, by definition is needs is used it is an integration test.

I believe unit and acceptance tests are in a relatively alright place, but integration tests are not. They need lots of love. It is also unfortunate that many things, like ember-data models are not unit-test friendly.

Integration tests, likely should not have an isolated container, and may actually be better off booting up an app behind the scenes. This indicates deficiencies in various aspects of ember and ember's testing helpers

[tomdale]

Maybe just me, but I think of integration tests as also exercising the server/backend portion of the app.

[stefanpenner]

Maybe just me, but I think of integration tests as also exercising the server/backend portion of the app.

they can, but typically this should be isolated to adapters the communicate to the outside world. Again, integration vs acceptance is (black box vs open box) (emulating the user vs being a developer and using APIs to test multiple collaborators)

[trabus]

Maybe just me, but I think of integration tests as also exercising the server/backend portion of the app.

In ember-cli's case I imagine it is more important to stabilize the environment your modules are being integrated in rather than concerning actual interaction with the backend portion of your app. I see pretender or http-mocks providing that stability when running integration tests that interact with external data.

My confusion isn't really about the definition of the test types, but rather what an example of what each type of test looks like in ember-cli. I know for some types of modules this isn't clearly defined yet, and I imagine each type of module has constraints that either need exemptions or must be worked around to allow reasonable tests to be written.

I'm going to continue studying, and will likely try reaching out to some people to get ideas once I've defined the problems a little better.

[stefanpenner]

cc @joostdevries @chnn

[stefanpenner]

these docs will be moved to embers own docs.