[Django] Test suite cleanup

[mjtamlyn]

• Name and Contact of the Project Mentor: Marc Tamlyn (marc.tamlyn@gmail.com, mjtamlyn on twitter, github, IRC etc)
• Name of the Project: Test suite cleanup
• URL: djangoproject.com
• About the Project: Django has an extensive test framework for Python code, a suite of tools to make server-side testing easier, and a project policy that no new code is added without tests. This has been a significant contributor to the stability of Django as a project.

However, this now means that Django has a very large and powerful test suite without much separation or control from a user's perspective, so the goal of this project would be to add new options and suite types to allow running of specific types of tests, be they only a certain class (e.g. unit-tests only, selenium tests only) or excluding tests (such as the ones in contrib or third-party apps) from the main test run easily. In order to do this, we need a strong understanding of what tests we already have.

Django's test suite is also very large with over 1000 models. In some areas, the tests are poorly structured and it is not clear where similar related tests should be placed. It is likely there may be some duplication of features tested, and there are certainly edge cases which are not tested. For example, standardising all the unit tests necessary for a particular model field type would be beneficial.

• Suitable for beginners?

Whilst a lot of the actual work is likely to be quite straightforwards, it will require a good understanding of many features that Django has will be helpful when trying to ascertain which areas the tests are testing. Because the work can be split up into a wide range of small tasks with a cumulative large incremental gain, there is a high chance of some success even with a limited experience.

• What are issues/features students may work on?

Issues to consider:

• How would users declare which tests they want to run?
• Which tests should be enabled by default, and how hard should this be to change?
• How will it be app maintainers run their tests?
• Should there be additional hooks to, for example, allow tests to be run against different database backends in sequence?
• Are there tools similar to the new --debug-sql option which would help developers working on Django?
• How can we use coverage reports to find missing tests or redundant code?
• What more could we do with testing the front end code in the admin?
• Are their any benefits to using an alternative test runner like py.test?

Some easy things to do:

• Make use of the new setupTestData method
• Factor out long test cases or long tests.py files into logical separate tests and files, making use of the DiscoveryRunner's ability to find test_foo.py files.
• Combine some test_foo and test_foo_regress sections which are separate due to legacy purposes only
• Document the collection of test apps and their coverage of areas of Django

Some things not to consider:

• Splitting up at a high level between unit, regression and integration style tests
• Simply aiming to expand coverage

[alicetragedy]

hi @mjtamlyn, thank you for your submission! We're happy to accept this project for this year's Rails Girls Summer of Code. :tada: Feel free to go ahead and work on a project plan with any interested teams, as their applications need to be in by April 14th, 23:59 CET. I'll send you an email today with relevant information, but do spread the word among the Django/Python community if you can and think it would be effective to get interested students on board!

Note to Students

This project will probably require you to be quite familiar with Django and Python.