Fixes the coverage command in tox.ini so it works on Windows (#250), by using coverage run -m pytest, which is the approach to running pytest that coverage currently shows in its documentation.
Expands the CI test matrix to test on Ubuntu, macOS, and Windows, for a total of 15 CI test jobs (5 Python versions, 3 operating systems).
Rationale for testing with multiple operating systems on CI
This makes testing on CI more robust. Even when not deliberately relying on any platform-dependent features, it is possible to have a bug that only causes a test to error out or fail on some operating systems and not others. (Testing multiple OSes on CI may also be seen as regression testing for #250, but I think that is actually a less compelling motivation than the general benefits of doing so.)
However, it is admittedly not without its disadvantages. Although the jobs run in parallel, the total amount of time it takes tests to run on CI may still be increased, for two reasons. First, macOS and Windows runners tend to install and run Python code slightly slower than Ubuntu runners, at least in my experience. Second, the total number of jobs that can run at a time is rate-limited, so if tests are running automatically on numerous different commits, some may have to wait for runners.
I think this is well worth it. I tried it out in a throwaway PR internal to my fork, and the delays did not seem like a problem. More broadly, I think running CI tests on multiple OSes tends to be helpful for most projects. In this project, there are already many tox environments for the tests, for different versions of jsonschema and Markdown, and for different Python versions. Conceptually, I would view this as extending that to achieve more robustness along similar lines.
Because the number of jsonschema package versions only increases over time, it may make sense to add jsonschema versions to tox.ini at a higher rate than they are removed. For this reason, it's possible that tests will be slower in the future than now. But I don't think that should stand in the way. The CI test runners are upgraded from time to time, becoming faster. Furthermore, if necessary, the different tox environments within a CI test job (i.e., those for the same OS and Python version but differing by jsonschema and Markdown version) could themselves be parallelized by passing -p/--parallel to tox (and -p/--parallel-mode to coverage run, to avoid clashing on a shared coverage.xml). For now, I don't think parallelism within individual CI test jobs is called for, and I have not included such a change in this PR.
Closes #250
This makes two related changes:
coveragecommand intox.iniso it works on Windows (#250), by usingcoverage run -m pytest, which is the approach to runningpytestthatcoveragecurrently shows in its documentation.Rationale for testing with multiple operating systems on CI
This makes testing on CI more robust. Even when not deliberately relying on any platform-dependent features, it is possible to have a bug that only causes a test to error out or fail on some operating systems and not others. (Testing multiple OSes on CI may also be seen as regression testing for #250, but I think that is actually a less compelling motivation than the general benefits of doing so.)
However, it is admittedly not without its disadvantages. Although the jobs run in parallel, the total amount of time it takes tests to run on CI may still be increased, for two reasons. First, macOS and Windows runners tend to install and run Python code slightly slower than Ubuntu runners, at least in my experience. Second, the total number of jobs that can run at a time is rate-limited, so if tests are running automatically on numerous different commits, some may have to wait for runners.
I think this is well worth it. I tried it out in a throwaway PR internal to my fork, and the delays did not seem like a problem. More broadly, I think running CI tests on multiple OSes tends to be helpful for most projects. In this project, there are already many
toxenvironments for the tests, for different versions ofjsonschemaandMarkdown, and for different Python versions. Conceptually, I would view this as extending that to achieve more robustness along similar lines.Because the number of
jsonschemapackage versions only increases over time, it may make sense to addjsonschemaversions totox.iniat a higher rate than they are removed. For this reason, it's possible that tests will be slower in the future than now. But I don't think that should stand in the way. The CI test runners are upgraded from time to time, becoming faster. Furthermore, if necessary, the differenttoxenvironments within a CI test job (i.e., those for the same OS and Python version but differing byjsonschemaandMarkdownversion) could themselves be parallelized by passing-p/--paralleltotox(and-p/--parallel-modetocoverage run, to avoid clashing on a sharedcoverage.xml). For now, I don't think parallelism within individual CI test jobs is called for, and I have not included such a change in this PR.