Pre-test hooks (for creating files, etc)

Files and other stuff might require certain assets to be ready before the users program is even started. This would allow the test program to create those assets (and delete them after each test).

There are probably a couple of ways of achieving this syntax-wise. A few options in order of preference:

1. Add an extra decorator

@before(function, cleanup) which would execute the function just before the test and cleanup afterwards.

This could then be then be abstracted away into @before_test(manager_function), where manager_function is a function that can be used in with_statement header.

_Example_ (assume temporary_file is already defined in grader)

from grader import *

def create_temp_file(name, contents):
    return lambda: temporary_file(name, contents)

@test
@before_test(create_temp_file("hello.txt", "World!"))
def f(m):
     # hello.txt exists now in the current folder
     assert "World!" in m.stdout.read()

Pros Explicit, users can define their own hooks. Cons

~~This has the negative of forcing the order of the decorators - the pre-test decorators have to come before the @test decorator.~~ Not true, the manager decorator can take care of this.
It's also not very clear (requires nested function statements and fairly complex nesting of functions).
Can't think of a clear name for such a decorator
2. Modify the test decorator

If given arguments like create_temp_file("hello.txt", "World!"), it would wrap the tested function in a with statement in which header all the arguments are called:

The test would use this as following:

@test(create_temp_file("hello.txt", "World!"))
def some_test(m): 
    # some test

Pros Users can define and use their own hooks. Cons Implicit, not clear why that function should take such arguments

3. Provide in-test functions for file creation, execution

This seems like a bad choice since it would add quite a lot complexity to the system (adds the question, when exactly do we start executing users program).

4. Work around the issue

E.g. provide hooks to mock open() and so on.

_Note: the hard part here is figuring out a clean (and hopefully intuitive) syntax to use, that wouldn't seem too much like magic._

macobo / python-grader