Open tobiasdiez opened 3 years ago
tobiasdiez
commented
3 years ago Replying to @mkoeppe:
But having
test_associativitysupplied byTestFiniteSemigroupsurely can't be the final design?!
I think this was a mistake on my side, it should be in a new test file for the "Semigroup" category.
mkoeppe
commented
3 years ago Replying to @tobiasdiez:
Replying to @mkoeppe:
So in the end it does not matter at all in which method
category_instancesan object is put?No, it doesn't really matter. One could also have one big test file listing all these examples.
OK, this is an important clarification, thanks.
But I would advocate to have one test file for each category, because then one can easily also write additional tests (not related to the category framework).
Would it not be better to have one test file for each module (... whose doctest currently has calls to TestSuite)? I really don't see what is gained by grouping the test objects by category -- which is coarser than how things are grouped currently (i.e., by module) and also arbitrary (because for join categories you would be making an arbitrary choice into which test file to put an object).
tobiasdiez
commented
3 years ago Replying to @mkoeppe:
Would it not be better to have one test file for each module (... whose doctest currently has calls to
TestSuite)? I really don't see what is gained by grouping the test objects by category -- which is coarser than how things are grouped currently (i.e., by module) and also arbitrary (because for join categories you would be making an arbitrary choice into which test file to put an object).
It depends on what you want to test. If you want to test that the implementation of the category is correct, you put it in the test file of the category. If you want to test the implementation of an example, you put it in the test file of the example.
nthiery
commented
3 years ago Hello!
Thanks for bringing me in the conversation. It's been ten years since I designed TestSuite (with the help from many). Since that time, Python and the available testing tools have evolved a lot. I am glad to see efforts going into rethinking the design in this new context, questioning the implementation, and studying alternatives that would reach the same design goals but better suit the context. Especially toward using now standard infrastructure and practice to reduce both the technical debt and the entry barrier for new devs and/or for integration with other software.
nthiery
commented
3 years ago (the rant may be a bit long, so I am splitting this into several comments).
So, what were these design goals:
(I) Hardening library and user defined algebraic structures, and beyond
The tests are there for checking the internal consistency of category code and for checking that parents (and their elements, morphisms, ...) satisfy the specifications of their categories (e.g. axioms). Supporting user defined algebraic structures is a must. In my research area we routinely create new algebraic structures, and when doing so TestSuite is a daily tool: practice has told me that, once the implementation of that algebraic structure pass the testsuite, most of the time it's actually correct (certainly something that is research area dependent!).
In some cases, it even becomes a research tool: I am not sure whether the newly defined algebraic structure actually satisfy the axioms I want it to satisfy, and I use TestSuite to actually run checks. And use post-mortem instrospection with the debugger to actually recover the counter examples ot my conjectures.
The next step in this direction is to work toward merging
the _test_xxx methods and _is_xxx methods: most of the time,
the code is mostly the same: checking whether axiom xxx is
satisfied. What changes is how thorough the test is, and
how failures are reported. I have toyed around with the idea
with prototypes in sage-semigroups. Alas I never got to actually
converge to a proposal for Sage. But this is something we want
to keep in mind.
tscrim
commented
3 years ago Something else I want to clarify is that the category examples are generally meant to minimal working examples of what needs to be implemented for a particular category. However, some categories actually use other existing objects in Sage, which is something many of the categories do, but such a WME is not essential to defining a category.
nthiery
commented
3 years ago (II) Categories as bookshelves of code, documentation and tests
Each category models some algebraic realm. Like the bookshelf about Group Theory in a library, the category of groups in GAP or Sage ties together everything that applies to all groups: code, documentation, and tests. In a perfect world where the same system would be used for computations and proofs, I would include here mathematical statements (e.g. formulae for the axioms) and proofs! Then, there are subshelves for group elements, group morphisms, ... Subshelves for particular types of groups, and so on.
There are several ways to implement that tying together. In GAP, categories are a mere filters; each method is tied to a category through an individual declaration. And the language uses a bespoke method resolution that data. In SageMath instead, we decided to rely on standard OO method resolution, grouping together methods in classes. Both approaches have their advantages notably in terms of flexibility left to organize the code, and enable user extensions to existing categories.
Two features are required for testing purposes:
(a) Recovering all the tests that one wants to run to check an object in category A
(b) For each of them, resolving which test method is to be called. Indeed, a subcategory may overload a _test_xxx method when, for example, it has a more efficient way to check xxx.
Note that (b) is of the same nature as when resolving a usual method call. It thus seems highly desirable to use the exact same resolution mechanism as for code.
nthiery
commented
3 years ago (III) Scalability
Nothing much to say here: whatever design is chosen should scale to the rich hierarchy of categories in SageMath, with all the mantras (axioms, ...) that were implemented to support that richness.
A critical bit is to satisfy as much as possible the Single Source Of Truth principle. E.g. if at some point I want to refine the category of a parent (e.g. because a new category was created), I should only need to update a single location in the code.
nthiery
commented
3 years ago Sorry for the rant; I am trying to clarify for myself what guided -- implicitly or explicitly -- the design back then :-) Hopefully this is useful food for thought for a redesign.
nthiery
commented
3 years ago Oh, one piece of information in case it would be relevant: for the longest time it's been on my pile to instrument TestSuite so that, by running the Sage tests, one would get a relatively thorough list of all Sage parents together with their categories.
A typical applications would be to enable inverse lookup: given a category, suggest to the user a relatively thorough list of parents implementing that category; e.g. for documentation purposes.
tobiasdiez
commented
3 years ago Oh, that's wonderful feedback! This kind of deep insight is hard to get if you only read the existing code and don't know the story behind it's origin. Thanks a lot for your detailed account.
Please don't take this proposal as a critics of your work on the TestSuite. I was actually amazed when I discovered that Sage has it's own testing framework. The main idea behind this ticket was to outsource the cost of maintenance of such a framework to the broader Python community, so that we can focus on the math-related elements.
Concerning your points, I'm happy to report that the proposed design should fit all the points you mentioned. In short:
(I): It is flexible enough to accommodate new user-defined categories and examples.
(II): It relies on pytest's discovery mechanism, which makes it very transparent which tests are run for a given category object.
(III): Scalability is a bit hard to judge for me, since I'm not that familiar with all the possible edge cases. But so far I had no problems migrating the existing code. In particular, the "single source of truth principle" applies, since the testing code only relies on the calculated super categories.
Hence, I would like to invite you to play around with the new design and see if it's works for your use cases. Basic documentation about pytest is added in #31003. Let me know if you encounter any problems.
Merry Christmas!
kliem
commented
3 years ago Red branch.
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313
commented
3 years ago 7ed8c4ca-6d56-4ae9-953a-41e42b4ed313
commented
3 years ago Branch pushed to git repo; I updated commit sha1. New commits:
7692b09 | Merge branch 'develop' of git://github.com/sagemath/sage into public/refactoring/pytest |
tobiasdiez
commented
3 years ago New commits:
7692b09 | Merge branch 'develop' of git://github.com/sagemath/sage into public/refactoring/pytest |
tobiasdiez
commented
3 years ago Changed dependencies from #30748, #30901 to none
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313
commented
3 years ago 7ed8c4ca-6d56-4ae9-953a-41e42b4ed313
commented
3 years ago tobiasdiez
commented
3 years ago Merged develop branch and removed the (unnecessary) dependencies.
mkoeppe
commented
3 years ago Setting new milestone based on a cursory review of ticket status, priority, and last modification date.
fchapoton
commented
3 years ago red branch => needs work
tobiasdiez
commented
2 years ago Description changed:
---
+++
@@ -37,9 +37,7 @@
TOOD (as follow-up tickets):
- Complete migration of tests to pytest
-- Integrate `pytest` in `sage -t` and tox
-- Call `pytest` as part of the build process so that the newly added tests are found and checked
-- After the migration is done, remove invoking the `TestSuit` in the doctests and remove the `TestSuit` class.
+- After the migration is done, remove invoking the `TestSuite` in the doctests and remove the `TestSuite` class.
---
Description changed:
---
+++
@@ -4,47 +4,73 @@
- Reuse standard test infrastructure instead of a custom test discovery interface (this is a big plus as we do not have to maintain it ourself, especially given the large list of todo's in the TestSuite code)
- With this there also comes better error messages in case a test fails.
- Easier onboarding of new Python developers
-- Clear separation of test methods vs production code. For example, the `test_*.py` files can easily be excluded from distribution.
+- Clear separation of test methods vs production code. For example, the `*_test.py` files can easily be excluded from distribution.
- Good support by IDE (auto completion, test discovery and easier invocation). For example, VS code automatically discovers all pytest tests, provides a convenient interface to run them (all, subselection, only failed ones) and allows to directly start a debug session for a failing test. See https://code.visualstudio.com/docs/python/testing
- Less and clearer code (no hidden assumptions)
Disadvantages of this approach:
-- Current developer have to learn the new conventions (files need to be named `test_*.py` and tests methods need to start with `test_`) and pytest
+- Current developer have to learn the new conventions (files need to be named `*_test.py` and tests methods need to start with `test_`) and pytest
- Tests no longer live in the same file as the code (I would say this is an advantage, but some people may prefer to have them really close together).
In order to migrate the following changes are necessary:
-- Move `_test_xyz` from a category `module` to new file `test_module` (e.g. `test_sets_cat`)
+- Move `_test_xyz` from a category `module` to new file `<module>_test.py` (e.g. `finite_semigroups_test.py`)
- Rewrite test using pytest (which is straightforward and more or less only amounts to removing all the custom code involving `self_tester`)
-- Create a new test class for the parent class under test (e.g `test_finite_semigroups`)
+- Create a new test class for the parent class under test (e.g `TestFiniteSemigroup`)
- Create a static method `category_instances()` in this test class returning a list of instances to test.
For the moment, I did this only for two test methods in `sets_cat` to illustrate the process.
The new implementation is as follows:
-- The test class for the category (e.g. `test_sets_cat`) defines generic tests that should pass for every object in that category.
-- For every object in the category there is a test class (e.g. `test_finite_semigroups`) that provides instances to test via the method `category_instances`
-- Running pytest finds the test class `test_finite_semigroups`, then determines the categories of the object and adds all generic test methods for that category to the test class (this happens in `conftest.py`).
+- The test class for the category (e.g. in `sets_cat_test.py`) defines generic tests that should pass for every object in that category. Moreover, the test class may provides instances (objects) to test via the method `category_instances`.
+- Running pytest finds the test file `finite_semigroups_test.py`, then determines the categories of the objects (returned from `category_instances`) and adds all generic test methods for that category to the test class (this happens in `conftest.py`).
- Each test method can have the parameter `category_instance` which is bound to the return value of the `category_instances` method.
-- Thus, the output is as follows:
+
+Running `cd src && pytest --verbose` (from a virtual env with sage and pytest installed), the output is as follows:
-sage/categories/test_finite_semigroups.py::TestFiniteSemigroup::test_element_eq_reflexive[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED [ 25%] -sage/categories/test_finite_semigroups.py::TestFiniteSemigroup::test_element_eq_reflexive[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED [ 50%] -sage/categories/test_finite_semigroups.py::TestFiniteSemigroup::test_cardinality_return_type[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED [ 75%] -sage/categories/test_finite_semigroups.py::TestFiniteSemigroup::test_cardinality_return_type[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED [100%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_enumerated_set_contains[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED [ 2%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_enumerated_set_contains[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED [ 4%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_enumerated_set_contains[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED [ 6%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_enumerated_set_iter_list[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED [ 8%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_enumerated_set_iter_list[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED [ 10%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_enumerated_set_iter_list[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED [ 13%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_enumerated_set_iter_cardinality[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED [ 15%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_enumerated_set_iter_cardinality[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED [ 17%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_enumerated_set_iter_cardinality[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED [ 19%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_associativity[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED [ 21%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_associativity[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED [ 23%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_associativity[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED [ 26%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_an_element[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED [ 28%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_an_element[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED [ 30%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_an_element[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED [ 32%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_an_element_idempotent[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED [ 34%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_an_element_idempotent[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED [ 36%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_an_element_idempotent[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED [ 39%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_cardinality_return_type[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED [ 41%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_cardinality_return_type[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED [ 43%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_cardinality_return_type[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED [ 45%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_construction[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED [ 47%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_construction[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED [ 50%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_construction[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED [ 52%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_element_eq_reflexive[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED [ 54%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_element_eq_reflexive[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED [ 56%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_element_eq_reflexive[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED [ 58%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_elements_eq_symmetric[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED [ 60%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_elements_eq_symmetric[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED [ 63%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_elements_eq_symmetric[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED [ 65%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_elements_eq_transitive[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED [ 67%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_elements_eq_transitive[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED [ 69%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_elements_eq_transitive[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED [ 71%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_elements_neq[An example of a finite semigroup: the left regular band generated by ('a', 'b')] PASSED [ 73%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_elements_neq[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c')] PASSED [ 76%] +sage/categories/finite_semigroups_test.py::TestFiniteSemigroup::test_elements_neq[An example of a finite semigroup: the left regular band generated by ('a', 'b', 'c', 'd')] PASSED [ 78%] +...
+As one can see, each general test method is invoked for the three examples.
TOOD (as follow-up tickets):
- Complete migration of tests to pytest
- After the migration is done, remove invoking the `TestSuite` in the doctests and remove the `TestSuite` class.
----
-By default, pytest implements the following standard test discovery:
-
-- Search for `test_*.py` or `*_test.py` files (I would stick to the `test` prefix).
-- From those files, collect test items:
- - Functions or methods with prefix `test`
- - If the functions are inside of classes, then the class name has to start with `test` as well.
-Rebased, so ready-for-review again.
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313
commented
2 years ago Branch pushed to git repo; I updated commit sha1. New commits:
6d4152d | Merge branch 'develop' of https://github.com/sagemath/sage into public/refactoring/pytest |
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313
commented
2 years ago 7ed8c4ca-6d56-4ae9-953a-41e42b4ed313
commented
2 years ago 7ed8c4ca-6d56-4ae9-953a-41e42b4ed313
commented
2 years ago tobiasdiez
commented
2 years ago Description changed:
---
+++
@@ -18,7 +18,7 @@
- Create a new test class for the parent class under test (e.g `TestFiniteSemigroup`)
- Create a static method `category_instances()` in this test class returning a list of instances to test.
-For the moment, I did this only for two test methods in `sets_cat` to illustrate the process.
+For the moment, I did this only for a few test methods in `sets_cat` using finite semigroups as example to illustrate the process.
The new implementation is as follows:
- The test class for the category (e.g. in `sets_cat_test.py`) defines generic tests that should pass for every object in that category. Moreover, the test class may provides instances (objects) to test via the method `category_instances`.
slel
commented
2 years ago Description changed:
---
+++
@@ -1,21 +1,21 @@
-Currently, the sage code contains `_test_xyz` methods that are invoked via doctests and discovered via a custom `TestSuit` class. In this ticket, we are proposing to replace this custom designed test discovery by using the established pytest framework https://docs.pytest.org/ (this is inline with #28936 - using mainstraim Python test infrastructure).
+Currently, Sage code contains `_test_xyz` methods that are invoked via doctests and discovered via a custom `TestSuite` class. This ticket is to replace this custom test discovery by the use of the established **pytest** framework https://docs.pytest.org/ (this is in line with #28936 - using mainstream Python test infrastructure).
Advantages of this approach:
-- Reuse standard test infrastructure instead of a custom test discovery interface (this is a big plus as we do not have to maintain it ourself, especially given the large list of todo's in the TestSuite code)
-- With this there also comes better error messages in case a test fails.
-- Easier onboarding of new Python developers
+- Reuse standard test infrastructure instead of a custom test discovery interface (this is a big maintenance relief, especially given the large list of todos in the `TestSuite` code).
+- Better error messages in case a test fails.
+- Easier onboarding of new Python developers.
- Clear separation of test methods vs production code. For example, the `*_test.py` files can easily be excluded from distribution.
-- Good support by IDE (auto completion, test discovery and easier invocation). For example, VS code automatically discovers all pytest tests, provides a convenient interface to run them (all, subselection, only failed ones) and allows to directly start a debug session for a failing test. See https://code.visualstudio.com/docs/python/testing
-- Less and clearer code (no hidden assumptions)
+- Good support by IDEs (auto completion, test discovery and easier invocation). For example, VSCode automatically discovers all pytest tests, provides a convenient interface to run them (all, a selection, only failed ones) and allows to directly start a debug session for a failing test. See https://code.visualstudio.com/docs/python/testing
+- Less and clearer code (no hidden assumptions).
Disadvantages of this approach:
-- Current developer have to learn the new conventions (files need to be named `*_test.py` and tests methods need to start with `test_`) and pytest
-- Tests no longer live in the same file as the code (I would say this is an advantage, but some people may prefer to have them really close together).
+- Current developers have to learn pytest and its conventions (files must be named `*_test.py` and test methods must start with `test_`).
+- Tests no longer live in the same file as the code (I would call this an advantage, but some people may prefer to have them really close together).
-In order to migrate the following changes are necessary:
-- Move `_test_xyz` from a category `module` to new file `<module>_test.py` (e.g. `finite_semigroups_test.py`)
-- Rewrite test using pytest (which is straightforward and more or less only amounts to removing all the custom code involving `self_tester`)
-- Create a new test class for the parent class under test (e.g `TestFiniteSemigroup`)
+In order to migrate, the following changes are necessary:
+- Move `_test_xyz` from a category `module` to new file `<module>_test.py` (e.g. `finite_semigroups_test.py`).
+- Rewrite tests using pytest (which is straightforward and more or less only amounts to removing all the custom code involving `self_tester`).
+- Create a new test class for the parent class under test (e.g `TestFiniteSemigroup`).
- Create a static method `category_instances()` in this test class returning a list of instances to test.
For the moment, I did this only for a few test methods in `sets_cat` using finite semigroups as example to illustrate the process.
@@ -69,8 +69,8 @@
As one can see, each general test method is invoked for the three examples.
-TOOD (as follow-up tickets):
-- Complete migration of tests to pytest
-- After the migration is done, remove invoking the `TestSuite` in the doctests and remove the `TestSuite` class.
+TODO (in follow-up tickets):
+- Complete migration to pytest.
+- Once that is done, stop invoking `TestSuite` in doctests and remove the `TestSuite` class.
Changed keywords from none to testsuite
tobiasdiez
commented
2 years ago Why did you change the milestone? I don't see anything that would prevent this being merged soon (except for the missing review).
mkoeppe
commented
2 years ago There's been no activity, and it's not clear if there are any developers who would be interested in rewriting the entirety of our test suite.
tobiasdiez
commented
2 years ago While being a relatively huge project, I think, a completely migration can be done in a relatively straightforward way with an overseeable time investment. I would definitely up for doing this alone if necessary. So I'm appreciate if this ticket could be reviewed soon. Thanks!
mkoeppe
commented
2 years ago You have received review comments: comment:45, comment:48, etc.
mkoeppe
commented
2 years ago Description changed:
---
+++
@@ -73,4 +73,6 @@
- Complete migration to pytest.
- Once that is done, stop invoking `TestSuite` in doctests and remove the `TestSuite` class.
+See also:
+- #33546 Replace custom Sage doctest discovery by pytest (using pytest_collect_file)
Replying to @mkoeppe:
You have received review comments: comment:45, comment:48, etc.
I've fixed comment:45. To comment:48 I've already reacted in comment:49.
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313
commented
2 years ago Branch pushed to git repo; I updated commit sha1. New commits:
b047abe | Merge branch 'develop' into public/refactoring/pytest |
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313
commented
2 years ago mkoeppe
commented
1 year ago Branch has merge conflicts
Currently, Sage code contains
_test_xyzmethods that are invoked via doctests and discovered via a customTestSuiteclass. This ticket is to replace this custom test discovery by the use of the established pytest framework https://docs.pytest.org/ (this is in line with #28936 - using mainstream Python test infrastructure).Advantages of this approach:
TestSuitecode).*_test.pyfiles can easily be excluded from distribution.Disadvantages of this approach:
*_test.pyand test methods must start withtest_).In order to migrate, the following changes are necessary:
_test_xyzfrom a categorymoduleto new file<module>_test.py(e.g.finite_semigroups_test.py).self_tester).TestFiniteSemigroup).category_instances()in this test class returning a list of instances to test.For the moment, I did this only for a few test methods in
sets_catusing finite semigroups as example to illustrate the process.The new implementation is as follows:
sets_cat_test.py) defines generic tests that should pass for every object in that category. Moreover, the test class may provides instances (objects) to test via the methodcategory_instances.finite_semigroups_test.py, then determines the categories of the objects (returned fromcategory_instances) and adds all generic test methods for that category to the test class (this happens inconftest.py).category_instancewhich is bound to the return value of thecategory_instancesmethod.Running
cd src && pytest --verbose(from a virtual env with sage and pytest installed), the output is as follows:As one can see, each general test method is invoked for the three examples.
TODO (in follow-up tickets):
TestSuitein doctests and remove theTestSuiteclass.See also:
33546 Replace custom Sage doctest discovery by pytest (using pytest_collect_file)
CC: @mkoeppe @tscrim @nthiery @slel
Component: refactoring
Keywords: testsuite
Author: Tobias Diez
Branch/Commit: public/refactoring/pytest @
b047abeIssue created by migration from https://trac.sagemath.org/ticket/30738