stdlib-property-tests

Property-based tests for the Python standard library (and builtins)

Goal

Find and fix bugs in Python, before they ship to users.

CPython's existing test suite is good, but bugs still slip through occasionally. We think that using property-based testing tools - i.e. Hypothesis - can help with this. They're no magic bullet, but computer-assisted testing techniques routinely try inputs that humans wouldn't think of (or bother trying), and turn up bugs that humans missed.

Specifically, we propose adding these tests to CPython's CI suite, and gave a talk at the 2020 Language Summit to that effect. Doing so would mean:

• New code can be developed with property-based tests - some already is, but adding it to CPython CI would catch bugs.
• Tests for existing code can be augumented with property-based tests where this seems particularly valuable - e.g. tests/test_source_code.py in this repo has discovered several bugs.
• Property-based testing can be adopted incrementally. Replacing existing tests is an explicit non-goal of this project.

PyPy already uses Hypothesis, and sharing as much of the test suite as possible between implementations would be great. How this would work depends largely on CPython's decisions, though.

LICENSE

By contributing to this repository, you agree to license the contributed code under user's choice of the Mozilla Public License Version 2.0, and the Apache License 2.0.

This dual-licence is intended to make it as easy as possible for the tests in this repository to be used upstream by the CPython project, other implementations of Python, and the Hypothesis project and ecosystem.

Workflow

To run the tests against the current version of Python:

• pip install -r requirements.txt (or hypothesis hypothesmith)
• python -m unittest

For development, we use tox to manage an extensive suite of auto-formatters and linters, so:

• pip install tox
• tox

will set up a virtualenv for you, install everything, and finally run the formatters, linters, and test suite.

Contributors

• Zac Hatfield-Dodds
• Paul Ganssle
• Carl Friedrich Bolz-Tereick

Trophy Case

Bugs found via this specific project:

• BPO-40661, a segfault in the new parser, was given maximum priority and blocked the planned release of CPython 3.9 beta1.
• OverflowError in binascii.crc_hqx() under PyPy
• BPO-38953 tokenize.tokenize -> tokenize.untokenize does not round-trip as documented. Nor, for that matter, do the tokenize/untokenize functions in lib2to3.pgen.tokenize.
• PEP-615 (zoneinfo) fold detection failed for early transitions when the number of elapsed seconds is too large to fit in a C integer; and a fold inconsistency where first offset handling was broken in the C extension.
• BPO-40668, catastrophic loss of precision when attempting to round-trip YIQ-RGB-YIQ with the colorsys module - more than 10% of the possible range. (via #13)

Further reading

• Hypothesis' website, documentation, GitHub repo
• Introductory articles, simple properties, metamorphic properties
• Related thoughts from hardware testing and verification, testing a screencast editor, PBT in Erlang / Elixr, testing C code using Hypothesis
• python-afl or OSS-FUZZ could work very nicely with Hypothesis' fuzz support
• hypothesmith generates syntatically-valid but weird Python source code (e.g. BPO-38953 or psf/black#970). Using a syntax tree for semantic validity is the logical next step.