direct access to the results of a notebook evaluation without creating the archive

[Feelx234]

Scenario: While writing your checker_script, you might want to see the output of the test directly without recreating the entire archive. Also creating the archive multiple times is problematic as a FileExistsError is raised when the same notebook is created multiple times for the same (dummy) Team. I would suggest providing a function like this:

def grade_notebook_no_archive(self, nb_path, target_dir=None, context=None):
    target_dir = target_dir or os.getcwd()    # prepare notebook
    with open(nb_path, mode='rb') as f:
        nb_data = f.read()    nb_hash = blake2b(nb_data, digest_size=4).hexdigest()    # prepare context and execute notebook
    with open('code.py', mode='wt') as c:
        # actual notebook execution
        try:
            logger.debug('execute notebook')
            state = exec_notebook(
                io.StringIO(nb_data.decode('utf-8')),
                file=c,
                ignore_errors=True,
                cell_timeout=self._cell_timeout
            )        except ValueError:
            state = {}    # infer meta information
    group = state.get('team_members', {})    if not group:
        logger.warning(f'Couldn\'t find valid information about team members in "{nb_path}"')    # execute tests
    logger.debug('execute tests')
    results, summary = self.apply_tests(state)
    enriched_results = dict(
        autograde_version=autograde.__version__,
        orig_file=str(nb_path),
        checksum=dict(
            md5sum=md5(nb_data).hexdigest(),
            sha256sum=sha256(nb_data).hexdigest(),
            blake2bsum=nb_hash
        ),
        team_members=group,
        test_cases=list(map(str, self._cases)),
        results=results,
        summary=summary,
    )
    return enriched_results

Note that this function still outputs the code.py file. Then in your python test script you can do

if __name__ == '__main__':
    target = Path("my_solutions.ipynb")
    result = grade_notebook_no_archive(nbt, nb_path = target.absolute())
    for val in result['results'].values():
        print("<<<<<  ", val['target'])
        print(val['message'])

Having such a file would also writing tests for autograde much easier as you can isolate the test_result from the archive.

[0b11001111]

Turns out this is a Windows specific problem. Now, an archive is deleted rather than overwritten which should solve the problem (see 8d1d59d9e733de810e2a61ba4d0e09f2ea1e8a3a). Further, test execution (NotebookTest.execute) returns the number of failed tests as exit code which eases scripting (see c0576fdeb402096215dea490059cad4e371a4de8). However, if you're really interested in test results while developing a test, call NotebookTest.grade_notebook directly in order to get the result dictionary.

As I don't have access to a Windows machine to test things, I leave this issues open until I get confirmation of this problem being solved. @Feelx234 your turn ;)

[Feelx234]

Solved, thanks!