Detect illegal constructs used

[macobo]

Some tasks require that certain constructs are "banned" (e.g. "write your own max function"). We should expose an api for dealing with this.

This should be separate from #3, since this cannot be 100% foolproof, so it shouldn't be used to block some sys calls etc.

[macobo]

I can think of 2 ways of approaching this atm.

1. Expose ast through ModuleContainer

Within the test, we can then write assertions that scan the ast for certain constructs. Macropy and quasiquotes make this simple enough.

This might not be robust enough. (e.g. three ways of accessing and importing function sin from some module). from math import sin, from math import * and import math

There are ways to sneak past this, e.g. exec, eval.

2. Manipulate imports and builtins

This would mean removing some functions from __builtins__ when a test is executed and writing an import hook that removes the desired functions from the module.

Patching the import system has a (serious?) drawback:

• It might (will) break other modules, e.g. when we ban math.sin and some other module relies on it.
• If we work around that (serve different modules to different people), then not all modules set __all__ defined and math.sin might be available via somemodule.math.sin.

This would be sensible if we could somehow discover where a function is being called from - if it's from the tested file, raise an exception. See http://stackoverflow.com/questions/1095543/get-name-of-calling-functions-module-in-python.

[macobo]

As the module has evolved, I think the correct thing to do is to make decorator (implemented as a pre-test hook( that exposes the AST of the users program as a parameter to the tester function.

@test
@expose_ast
def AST_test(m, AST):
    # test ast inside

This could be then extended with functions and wrappers that check the AST by walking through it.

[macobo]

As of 81e597f016397c5797e71cf436c8a6a7656fec7f, the syntax for exposing the AST works exactly as the above comment said. However, as there are no examples of walking the AST looking for nodes yet, I'm leaving this issue open.

[macobo]

Some things that need to be supported:

Given ast, find something that matches template. That could be a function, a while statement or whatever. Not sure if this would work everywhere, as it depends on the ast definition.