jupyter / nbgrader

A system for assigning and grading notebooks
https://nbgrader.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
1.3k stars 317 forks source link

Validation errors are not shown correctly during autograding #1933

Open tmetzl opened 2 weeks ago

tmetzl commented 2 weeks ago

nbgrader_version=0.9.3 and older

When autograding a notebook with metadata errors, the error messages are not helpful and/or misleading. Here is an example notebook with a duplicate grade_id:

from nbformat.v4 import new_notebook, new_markdown_cell
from nbgrader.preprocessors import CheckCellMetadata
from nbgrader.nbgraderformat import MetadataValidator

nb = new_notebook()

cell = new_markdown_cell(
    metadata=dict(
        nbgrader=dict(
            grade_id="my_id",
            grade=False,
            solution=False,
            locked=True,
            schema_version=3
        )
    )
)

nb.cells.append(
    cell
)

nb.cells.append(
    cell
)

Now when the MetadataValidator is used to validate the notebook there is a descriptive error message that gives the correct information about the error:

MetadataValidator().validate_nb(nb)

Output:

---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
---> 27 MetadataValidator().validate_nb(nb)

File /opt/conda/lib/python3.11/site-packages/nbgrader/nbgraderformat/v3.py:110, in MetadataValidatorV3.validate_nb(self, nb)
    108 grade_id = cell.metadata['nbgrader']['grade_id']
    109 if grade_id in ids:
--> 110     raise ValidationError("Duplicate grade id: {}".format(grade_id))
    111 ids.add(grade_id)

ValidationError: Duplicate grade id: my_id

When the CheckCellMetadata preprocessor is used the error is undescriptive:

CheckCellMetadata().preprocess(nb, {})

Output:

---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
---> 28 CheckCellMetadata().preprocess(nb, {})

File /opt/conda/lib/python3.11/site-packages/nbgrader/preprocessors/checkcellmetadata.py:18, in CheckCellMetadata.preprocess(self, nb, resources)
     16     msg = "Notebook failed to validate; the nbgrader metadata may be corrupted."
     17     self.log.error(msg)
---> 18     raise ValidationError(msg)
     20 return nb, resources

ValidationError: Notebook failed to validate; the nbgrader metadata may be corrupted.

When autograding a notebook like this I actually get this misleading error originating from here: https://github.com/jupyter/nbgrader/blob/main/nbgrader/converters/base.py#L421

AutogradeApp | ERROR] One or more notebooks in the assignment use an old version 
    of the nbgrader metadata format. Please **back up your class files 
    directory** and then update the metadata using:

    nbgrader update .

This might be related to #1629.