Revise "Testing" and "Software Maturity" standards

[jtniehof]

(Broader context at the end).

This issue is to discuss a potential PHEP to replace the "testing" and "software maturity" portions of the existing PyHC standards: to lay out what looks like the current standards, and allow for suggested changes.

I am lumping these together as potentially related. I am not sure what the appropriate division of these existing standards is. I suggest we focus the conversation in this issue on scoping: what standards belong together in a future document? Then we can have new issues to discuss the actual standards.

The relevant standards are:

• 1. Packaging: All code must be organized and provided as part of installable Python packages.
• 3. Releases: Projects should strive to have consistent and stable releases. Those releases should be made available through PyPI and Conda. Code that is not yet stable must have a release number that is less than 1.0 (e.g. 0.x). Packages should consider using semantic versioning.
• 4. Operating System Support: Packages must strive to support all major operating systems (e.g., OS X, Linux, Windows).
• 6. Version control: All code must use version control. It is strongly recommended that projects make use of a distributed version control system (e.g., git).
• 7. Coding Style: Projects must adopt the basic style recommendations of PEP 8 and static analysis tools should be used to identify deviations from the basic style recommendations (e.g. pylint, flake8, pycodestyle).
• 9. Testing: Stable packages must provide unit tests of individual components (e.g. functions, classes) as well as integration tests that test the interaction between components that covers most of the code. Testing coverage should be measured. Automated testing is recommended, in which tests are run before any code is merged. System[link] and acceptance[link] testing are also recommended.
• 10. Dependencies: Projects should import the minimum number of packages necessary. Adding new dependencies should be a considered decision.
• 14. Binaries: Binary files should be added to the package repository only when necessary in order to keep packages as light as possible. Jupyter notebooks can be binary files and should not be committed to the package repository but can be provided in other repositories.

(bullet list to fight the renumbering, apologies for roman numerals and suggestions welcome)

I am not seeing any relevant suggested updates in #16.

Standard 10 is somewhat in tension with Standard 12 (duplication of effort); 12 is discussed in #33.

Broader context: I envision a process of "patriating the constitution" where we revisit the existing standards documents and incorporate them into PHEPs, potentially updated, that are explicitly noted as replacing the relevant standards. We probably do not want one PHEP per standard. Our previous grouping in the review guidelines seems a good start.

[jtniehof]

My feeling here is that 1, 3, 4 relate to the release process and supported environments and belong together, along with other changes we've discussed (e.g. binary wheels and the like). They're user-facing.

I could see 6 potentially moving into the community section (#33) or staying here.

7, 9, 10, 14 are all pretty clearly development and repository practices (with 6, if it stays here). "Maturity" may not be the best name for this set.

[jtniehof]

In the interests of prodding discussion, I'm throwing out a strawman:

A "packaging standards" PHEP would incorporate (changes are bold):

• Packaging (standard 1), must be installable Python package. At least should use a PEP517/518 compliant build system. (Perhaps a must).
• Releases (standard 3), must make releases on a regular basis (not necessarily scheduled, wordsmithing welcome on this...basically not "disappear and tell people to build from main for years"). Must be on PyPI. Should be on conda. Should deliver wheels, including binary wheels if needs compiled code. Must have a consistent and documented versioning scheme.
• OS support (standard 4): Must support Linux, OSX, Windows. Must support current architectures. This needs some tuning--requiring support for Mac ARM is reasonable. Windows ARM isn't.

EDIT: I can see a "reference implementation" of a PyHC community-supported package that's just the framework for this....

Standard 6 moves to community (#33) and I'll explain why there.

A "development practices" PHEP would incorporate:

• Coding standards (standard 7): Must adopt PEP8, should static analysis, should static analysis in CI
• Testing (standard 9): Must include unit tests in repository. Must document preferred way of running unit tests. Should measure coverage and target reasonable code coverage while considering data coverage. Should use CI. Should have procedures for acceptance and system testing (this should be fleshed out more than in the previous standard, also see #18)
• Dependencies (standard 10): I'm really at a loss at the way forward here. There's a lot to be said for minimum dependencies. There's also a lot to be said for not reinventing the wheel. What is the goal here? How can it be evaluated? How can it be described?
• Binaries (standard 14): Should not commit binaries (not commit large binaries? how big is large?) Should include relevant source where possible. Should use jupytext for notebooks, either instead of or in addition to binary notebook formats.