tests/ in the repo's root, or in the package folder?

[consideRatio]

Questions

1. New projects: should we if in doubt put tests outside or inside the package folder?
2. Existing projects: should we in some situation relocate from outside to inside, or inside to outside?

Examples

• jupyterhub/jupyterhub: inside, since initial setup 2014 by Min
• jupyterhub/oauthenticator: inside, since initial setup 2017 by Min
• jupyterhub/kubespawner: outside, since initial setup 2016 by Yuvi

Related

• Pytest describes the outside / inside approaches, both considered acceptable. About outside

  Putting tests into an extra directory outside your actual application code might be useful if you have many functional tests or for other reasons want to keep tests separate from actual application code (often a good idea): About inside Inlining test directories into your application package is useful if you have direct relation between tests and application modules and want to distribute them along with your application:

[consideRatio]

My current answer is

1. New projects benefit from locating things outside initially until we know the benefits of locating them inside motivates it. It will not be a breaking change to relocate tests inside the package, but it will be to relocate tests outside as someone may depend on fixtures etc in conf.py. Due to that, it seems better to start out outside until its known that we want to provide fixtures or similar to end users via the package, perhaps because someone making a subclass want to rely on them.
2. I think it can be fine to relocate from inside to outside if we believe the pytest fixtures in conftest.py or tests as part of the package are unused, but since that would be a possibly breaking change, only opt for doing so as part of a major release anyhow just in case. This is done in https://github.com/jupyterhub/oauthenticator/pull/552, which I think could be fine as there are no important fixtures for subclasses there I think.

[choldgraf]

For what it's worth, I've grown accustomed to assuming that tests are in an outside folder (eg, docs, tests, and either src or packagename as top level folders)

While we're at it, we might also want to standardize on sphinx folder naming:

• doc or docs
• _build in sake folder, or build and source in different folders with conf.py in a parent docs folder

EDIT by Erik: See https://github.com/jupyterhub/team-compass/issues/590 for discussion about sphinx!

[GeorgianaElena]

For what it's worth, I've grown accustomed to assuming that tests are in an outside folder (eg, docs, tests, and either src or packagename as top level folders)

Me too! I never found the oauthenticator tests from the first try and this was the motivation behind relocating them.

While we're at it, we might also want to standardize on sphinx folder naming

Great point! Personally, my only preference is consistency, so either way is ok.

Once changing approach would be to have all the repos follow what's happening in the jupyterhub repo (assuming there aren't strong opinions about one or the other).

[manics]

One other factor is whether the tests are installed into Python's site-packages for all users, which will be the case if tests are under the package directory instead of the top-level. This then requires consideration of what changes can be made without breaking others who import or extend the tests.

We could choose to exclude a nested tests directory, but I think it's easier to keep tests outside and only put them under the package folder if we actively want to make them available at runtime.

[minrk]

The benefits of including them:

• the correct tests can be run against any installed version of the package, because the version of the test is tied to the version of the install
• installing the package means the tests can be run
• test utilities can be reused in other packages (jupyterhub test tools are imported in several spawners and authenticators)

Benefits of excluding them:

• saved space at install time
• 'private' testing utilities are forced to actually be private since nobody can import them (importable test utilities can still be made, they just need to be explicitly made importable by moving them to a totally different part of the repo in the package)

Ultimately, I don't think it matters much (not enough to suggest any migrations for existing projects to me), but following what seems to be the current standard in Python to put tests in their own folder for new projects makes sense.