comparing what student has done to a target

[machow]

In python and R, SCTs use a solution environment to allow exercise creators to specify what they want to test. (e.g. Ex().test_object('a') will compare the variable a across the student and solution environments). In shell, there is currently only one "environment" (the process running as user repl).

While we think about the right way to create a solution environment for shell exercises, it'd be useful to lay out how to perform SCTs comparing some result to a target. For example,

1. that the student has created all the files in a folder
2. that the student has created a file, matching some target file

currently, this can be done using test_expr_error. For example, if the target file is at /somedir/file-a.txt, and the student was supposed to create file-a.txt in their current directory, you could run

# cmp returns non-zero exit code (error) when files don't match
Ex().test_expr_error('cmp file-a.txt /somedir/file-a.txt')

Let me know of cases where test_expr_error feels clunky, and we can implement new SCTs / modify it.

cc @gvwilson

[gvwilson]

Thanks @machow - cmp will do what I need in the single-file case provided we can put the reference file containing the correct content somewhere that the learner won't see it (don't want to distract them). Unfortunately, it looks like there's no recursive option to cmp as there is to diff, so I'll either:

1. compare a small number of files by hand
2. use diff with appropriate flags

The first is probably the best option anyway, since I don't know how I'd give sensible error reporting in the second case.

So, where should I put the reference files? I presume I provision them in requirements.sh?

[machow]

provisioning in requirements.sh sounds like the right move.

For (1), if it's useful, you can put the SCTs in a loop...

# cmp returns non-zero exit code (error) when files don't match
list_of_files = ['a.txt', 'b.txt']
for fname in list_of_files:
    feedback = "some message"
    cmd = 'cmp {fname} /some_dir/{fname}'.format(fname = fname)
    Ex().test_expr_error(cmd, feedback)

alternatively, tests that don't have Ex before them don't run right away and could be used later...

# defines tests but doesn't run yet
tests = [test_expr_error('cmp {0} /some_dir/{0}'.format(fname)) for fname in list_of_files]
# runs tests
Ex().multi(tests)

[gvwilson]

Thanks @machow - if the tests are in a loop, will the error reports from each be queued up somehow for the user to read, and will the final pass/fail status automatically be the conjunction of all the tests that were run?

[machow]

Outside of logical tests like test_or and test_correct, SCTs stop running / give feedback at the first failure.

https://protowhat.readthedocs.io/en/latest/#module-protowhat.checks.check_logic

(edit: this happens because being able to run one SCT, often requires previous ones to pass)

[filipsch]

In the refactored version of shellwhat, there are examples of how has_expr_output() can be used. Once you get used to it, it's pretty elegant.

• Link to ref docs
• glossary example