Missing directory layout for tests

[guettli]

I am missing a directory layout recommendation for tests here:

https://packaging.python.org/tutorials/packaging-projects/

I know that there are several ways to put you tests. It would be very nice, if there would be an official recommendation for this.

Thank you.

[pradyunsg]

Hi @guettli! Thanks for filing this issue!

We just had a long and fairly polarizing discussion on the src/ vs flat layout over at #320, so I'd prefer that we defer this discussion to a later time. Folks like myself are a little tired of "how to layout your project" discussions right now.

While I do think we should look into this at some point, the aim here too should probably be to document the trade-offs and only after that is done, should we even visit whether we need to provide a recommendation.

[ncoghlan]

Similar to flat vs src tree loyouts, in-package vs out-of-package tests is a question of trade-offs. I think the structure we're now proposing for the tree layout documentation in general will also work there - separate guides for both options, paired with a discussion that covers the trade-offs between them.

(If we ever add a testing tutorial though, I'd suggest we use a separate tests directory for that - including your tests in your sdist is good, but including them in your wheel archives is dubious, and a separate directory is much easier to leave out of the build process. Tests in a separate directory are also typically easier to run against a fully installed package, rather than only running against a source checkout)

[guettli]

@ncoghlan

separate guides for both options, paired with a discussion that covers the trade-offs between them.

I think new comers want a to follow a sane default without reading a discussion that covers the trade-offs.

A footnote which links to the details makes sense.

If you look at this question: https://stackoverflow.com/questions/56426027/pycharm-create-test-target-directory

Imagine your cursor is in your IDE/editor in a python method. Now you want to create a test for this method. Wouldn't it be great if the IDE/editor could automatically create a basic skeleton (a file at a matching location in the directory tree). An IDE/editor can't parse a discussion that covers the trade-offs. You need a simple and clear rule.

[ncoghlan]

Nobody has the authority to dictate a particular answer as the "one true way", because all the options have downsides.

However, narrowing it to a couple of alternatives is enough for tool creators to pick one that they favour, and then they can either document that it's the only approach they support, or provide an option to enable the non-default layout.

Pushing for "there can be only one" at the community level turns what could have been productive collaboration on documentation of alternatives and trade-offs into an acrimonious contest of ideas that focuses more on keeping the "wrong" alternative from being selected as the default than that does on providing an accurate accounting of the strengths and weaknesses of the different approaches.

[guettli]

In 1996 I compiled the linux kernel for the first time. You need to answer a lot of questions for this. For almost every question there was a hint. Something like "if unsure choose ...". I had no clue what these questions were about. I managed to compete the task because these hints existed. That's what I have on my mind. One clear hint for new comers (without "or").

[ncoghlan]

Sure, and re-reading my earlier notes, I agree if we ever have a testing tutorial, out-of-package tests will probably make a good default.

However, it would be inappropriate to add such a tutorial without the supporting documentation that makes it clear that that isn't a universally preferred approach.

[davaya]

You don't need to write a whole tutorial in order to include a default layout for first-time packagers. As one myself, I'm frustrated by "The section does not aim to cover best practices for Python project development as a whole" in https://packaging.python.org/guides/distributing-packages-using-setuptools/.

How much harm could it possibly do to include a test folder in the first newbie example:

packaging_tutorial/
  example_pkg/
    __init__.py
  tests/
  setup.py
  LICENSE
  README.md

If there is reasonable consensus that unit tests should be included in source distributions but not installed in site-packages (https://stackoverflow.com/questions/14454744/does-it-make-sense-to-install-my-python-unit-tests-in-site-packages), then just put a test folder at the same level as setup.py and example_pkg, not under example_pkg where it would be included in the installation.

Everyone knows there are different approaches with tradeoffs, various camps with different opinions, and valid reasons for doing things different ways. That doesn't rule out suggesting a sane default for beginners. Encouraging package authors to include unit tests in the first place outweighs waiting until you can get agreement on the perfect way(s) to write them.

---

EDIT: fixed name of folder to tests.

[ncoghlan]

I've moderated my stance since writing https://github.com/pypa/packaging.python.org/issues/619#issuecomment-507863267 above (mostly prompted by @davaya's comment), so I went ahead and merged the PR that changed the status quo to be an unqualified recommendation for a top level tests directory.

I'm leaving the issue open, though, as I do think a full resolution would consist of:

• a new discussion document that explains why the top level tests directory is the default, but having tests in the package remains a popular alternative
• linking to that discussion in an appropriate way from the new subsection added in #698

[guettli]

@ncoghlan you wrote "unqualified recommendation". I think a qualified recommendation would make more sense. Who can do a qualified recommendation?

[bhrutledge]

FWIW: In my recent work on #866 (and related to #718), I wondered about adding an example of testing, possibly before Generating distribution archives. I thought it would help establish a testing habit, and also validate the project layout before running build. I've got what I think is a minimal example project up at https://github.com/bhrutledge/example-pkg-bhrutledge, which builds on the current recommended project layout.

I decided not to do it for #866 because it felt like scope creep. I also thought it might require introducing the concept of editable installs, which I think needs better documentation in the guide.

[ncoghlan]

@guettli Anyone can write the qualified recommendation (my comment is using qualified in the "with caveats" sense, not the "with relevant credentials" sense)

[bhrutledge]

If I can get a 👍 from a maintainer, I'm happy to open a PR based on my previous comment (incorporating any suggestions, of course).

[webknjaz]

@bhrutledge quick look at your repo seems fine except that I'd default to pytest — it's been a de facto go-to framework for many years now. I understand that it's not in stdlib but I argue that it's what many pythonistas will see in the wild. And it will also set them on the path of exploring better UX/DX and best community practices.

[bhrutledge]

I'd default to pytest — it's been a de facto go-to framework for many years now.

That sounds good to me. Does that imply adding and explaining extras_require in setup.cfg, so that readers could do pip install -e .[dev]? Or would it better for this audience to just python3 -m pip install pytest?

[webknjaz]

I wouldn't add extras simply because it's controversial: some people like test/docs/dev extras but I think this doesn't belong there. Those envs aren't really useful to the end-users of the published artifacts (you can't pip install just an extra w/o the package itself) + it's best to have such envs pinned with all the transitive deps (pip-tools style, for example) and such info wouldn't be a great fit for package metadata either, in my opinion. Plus, exact pins traditionally aren't used in the dist meta — it usually contains direct deps with relaxed constraints. FWIW this would be a nice thing to discuss in the pinning guide but is probably too much in the simple packaging tutorial which should be straightforward and not have extra branching/options.

[bhrutledge]

I updated my example project to use pytest: https://github.com/bhrutledge/example-pkg-bhrutledge/commit/f20bcbf5bda35deb1925d9e831cf7d2bb899bca4

[webknjaz]

@bhrutledge I've added a few comments on that commit page.