Hidden but still executable tests

[tedeler]

Hi.

I plan to use nbgrader in my electronics class. Many of the assignments require the students to give simple values (for current, voltage or resistance). I would like the students to validate this values directly in their notebook. But if I'd placed a correnspondent testcell below each single assignment this would already includes the answer: e.g. assert( abs(Id-0.32) < 0.02 ) The correct answer obviously is Id=0.32.

Is there a way to have hidden but still executable tests with nbgrader?

Regards, Torsten

[tedeler]

Hi.

I found a solution that works quiet good for my needs: https://github.com/tedeler/nbgrader/tree/e1ca78836e901ed8bef378caa9a861040ed46be1

Basically I added the preprocessor obfuscatetests (mostly copied from hiddentest) that checks for a marked section in a grading cell. The section gets compiled and the bytecode is put in place.

The cellcode

Id, Ud = solution()

### BEGIN OBFUSCATE TESTS
import math
assert( math.isclose(Id, 0.004376, abs_tol=1e-3) )
assert( math.isclose(Ud, 0.624, abs_tol=5e-3) )
### END OBFUSCATE TESTS

gets

Id, Ud = solution()

import marshal; exec(marshal.loads(b"\xe3\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0b\x00\x00\x00@\x00\x00\x00sx\x00\x00\x00y<d\x00d\x01l\x00Z\x00e\x00j\x01e\x02d\x02d\x03d\x04\x8d\x03s\x1et\x03\x82\x01e\x00j\x01e\x04d\x05d\x06d\x04\x8d\x03s2t\x03\x82\x01e\x05d\x07\x83\x01\x01\x00W\x00n6\x04\x00e\x03k\nrX\x01\x00\x01\x00\x01\x00e\x05d\x08\x83\x01\x01\x00Y\x00n\x1c\x04\x00e\x06k\nrr\x01\x00\x01\x00\x01\x00e\x05d\t\x83\x01\x01\x00Y\x00n\x02X\x00d\x01S\x00)\n\xe9\x00\x00\x00\x00NgJ]2\x8e\x91\xecq?g\xfc\xa9\xf1\xd2MbP?)\x01Z\x07abs_tolg+\x87\x16\xd9\xce\xf7\xe3?g{\x14\xaeG\xe1zt?z\x1cYour solution is correct :-)z\x1eYour solution is incorrect :-(zKPlease implement your solution above and remove 'raise NotImplementedError')\x07\xda\x04math\xda\x07iscloseZ\x02Id\xda\x0eAssertionErrorZ\x02Ud\xda\x05print\xda\x13NotImplementedError\xa9\x00r\x07\x00\x00\x00r\x07\x00\x00\x00\xfa\x08<string>\xda\x08<module>\x01\x00\x00\x00s\x10\x00\x00\x00\x02\x01\x08\x01\x14\x01\x14\x01\x0c\x01\x0e\x01\x0c\x01\x0e\x01") )

I'm aware that disassembling the bytecode would reveal the solution. But in my usecase the students would only use this feature for self testing and don't have any advantages if cheating.

The current implementation only reports "correct" or "incorrect" as text messages to the student and do not raise Exceptions. This has probably to change for not to break the validation feature.

I'm looking forward read you comments or improvements.

Regards, Torsten

Ps.: I know the tests are missing. I try to find time for this.

[jhamrick]

Oh, very cool!

I unfortunately do not think this is something that could be part of nbgrader proper, since it is python-specific (and nbgrader is language agnostic), but it definitely seems like a really useful extension. If you want to create a separate repository for this, then other people could download it and use the preprocessor as an extension, too, and I'd be happy to add it to the list of links in #662 (which will eventually become a proper page in the documentation).

[acbart]

We were talking about building an extension/module that would refer to an external server to run hidden tests.

On the client side, students would see something like this:

Problem: "Write a function that adds together two numbers and returns their sum."

def add(a, b):
    return a+b

Then in a hidden cell, it would have something like this:

import custom_instructor_module
custom_instructor_module.check_assignment("add_assignment")

On the server side, this module basically does something like this:

import requests
def check_assignment(assignment_id):
    code = requests.get("https://private_server.com/"+assignment_id).text
    eval(code)

And then that runs the instructor tests securely and independently from the user. Does this seem like a viable model, with the understanding that it puts some burden on us to manage the external server AND the custom_instructor_module gets exposed to the student (haven't thought much about how to sandbox the student code yet)?

[jhamrick]

That would be a very cool library for instructors to be able to use! (As before, I don't think it is in scope for nbgrader itself since it's not language agnostic, but I would love to be able to put it on the third-party resources page and point people to it if you end up implementing it)

I think that model seems mostly reasonable, though hidden cells aren't default in the notebook so you would have to ensure students have a notebook extension installed for that to work (like https://github.com/kirbs-/hide_code ). Though honestly, you probably wouldn't want that in a hidden cell---you would probably want students to see that they are running the tests (it's just that they can't see what the tests actually are 😄)

I think there may be some complexity there with ensuring you actually send all the relevant code to the remote server to be executed. For example, if the function relies on variables that have been declared earlier, you will need to detect this and send the value of those variables along too. I suppose you could do something like copy the globals dictionary just to have a snapshot of the environment and send the whole thing over to the remove server, though that seems a bit overkill.