Feature request: pytest style testing

[yuenmh]

Summary

Testing system similar to pytest.

Motivation

I find that writing new tests in the grade_local.py file can be, to be frank, a little cumbersome, because of all the boilerplate code in the file deriving the base question class. I propose a system which uses the existing infrastructure to provide a more streamlined experience when writing tests, which would look something like the pytest API.

Details

The new addition is just a subclass of cse40.question.Question that loads all the functions in a file that are marked by a decorator. This can be added to the local grading script like this (taken from the HO3 one):

# grade_local.py
questions = [
    T1A("Task 1.A (drop_sparse_columns)", 1),
    T1B("Task 1.B (extract_numbers)", 1),
    T1C("Task 1.C (guess_types)", 1),
    T2A("Task 2.A (find_outliers)", 1),
    T2B("Task 2.B (merge_columns)", 1),
    T3A("Task 3.A (one_hot)", 1),
    T4A("Task 4.A (left_join)", 1),
    cse40.style.Style(path, max_points = 1),

    cse40.testing.TestFile(test_file_path), # <- this will run the tests in the file
    # The template could come with the path as Path(__file__).parent / "tests.py", which
    # could be an emptpy file in the template
]

The tests file would look something like this:

# tests.py
import cse40.testing
import pandas as pd
import numpy as np

# Not picked up as a test
def helper():
    ...

# This is picked up as a test
@cse40.testing.test
def drop_sparse_columns_returns_dataframe(submission, world_data):
    result = submission.drop_sparse_columns(world_data, 0.50)
    assert isinstance(result, pd.DataFrame)

The first argument to a test function is the submission, and all following args are filled with the corresponding values from additional_data. As you can see, the resulting API is actually not much different from the current one, it just lacks most of the visual noise, and also would use python asserts, which I feel are a bit more natural than testing with if statements.

My contribution

If the course staff/maintainers of this repo find this interesting, I would like to work on implementing it. Additionally if you would like to see a more concrete proposal I will be willing to clarify or code up a working example.

[ThachAndrew]

Thanks for the suggestion. This idea is good for a testing infrastructure, but probably not for scoring infrastructure. The local tests are like a testing infrastructure, where it's all or nothing. But the private tests use a scoring infrastructure, which rely on a non-pytest style, so we can give feedback, partial credit, and more control. We can refer to HO0, which gives a little peek at how the infrastructure is implemented.

However, please feel free to implement the pytest style testing on your own local end.

Thanks again, we encourage feature requests, so let us know if there are more!

[yuenmh]

Yes testing on the student's side is what I'm thinking of here. Writing tests quickly is probably more important for students when working through the problems than it is for the course staff writing cases for the autograder. ~I'll see what I can do, and link it here before making a PR because it will probably take a few iterations.~ Sorry it was late and I misread your message about implementing it.