jupyter / nbgrader

A system for assigning and grading notebooks
https://nbgrader.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
1.3k stars 317 forks source link

Student's solution with `input()` passes "Validate", but fails on "Autograde" #1844

Open lahwaacz opened 1 year ago

lahwaacz commented 1 year ago

Steps to reproduce the actual behavior

For example, give students this assignment (I'll omit the description as it is self-explanatory):

def count_digits(n, d):
    ### BEGIN SOLUTION
    n = abs(n)
    return str(n).count(str(d))
    ### END SOLUTION

Let's assume just one simple public test case:

assert count_digits(112233, 2) == 2

Let's say that a student changes the cell with the assignment as follows:

a = input("Enter an integer")
b = input("Enter a digit")

def count_digits(n, d):
    n = abs(n)
    return str(n).count(str(d))

s = count_digits(int(a), int(b))
print(s)

Now there is an issue: when the student clicks on the "Validate" button in JupyterLab to validate their assignment, nbgrader executes it just fine and says that the solution passes all the tests. However, when the instructor runs autograde, it fails with the following error:

StdinNotImplementedError: raw_input was called, but this frontend does not support input requests.

Expected behavior

Validate and autograde should behave consistently with respect to the input() function: either both should succeed, or both should fail with the same error.

Operating system

Arch Linux

nbgrader --version

Python version 3.11.5 (main, Sep  2 2023, 14:16:33) [GCC 13.2.1 20230801]
nbgrader version 0.9.1

jupyterhub --version (if used with JupyterHub)

4.0.2

jupyter notebook --version

7.0.6

jupyter lab --version

4.0.7
brichet commented 8 months ago

Thanks @lahwaacz for reporting this issue. After some inspection, it seems intentional that using the validate button only return errors on grade cells.

https://github.com/jupyter/nbgrader/blob/9876829b6be4fccce621ffee907bd5f9ac400c44/nbgrader/validator.py#L251

There is even an option to validate the whole notebook, c.Validator.validate_all = True (default to False).

https://github.com/jupyter/nbgrader/blob/9876829b6be4fccce621ffee907bd5f9ac400c44/nbgrader/validator.py#L92-L95

I don't have the history of these choices...