search

ucbds-infra / otter-grader

A Python and R autograding solution
https://otter-grader.readthedocs.io
BSD 3-Clause "New" or "Revised" License
122 stars 63 forks source link

Problem with variable name in test #817

Closed lrfinotti closed 1 month ago

lrfinotti commented 1 month ago

It seems that one cannot use a variable named ton a test.

If you have something like

t = 1  # SOLUTION

and then a test like

t == 1

you get an error. (Maybe a problem with conflicting variable names?) See the very simple attached file as an example. bug.zip (The ZIP file contains only a very simple notebook.)

Here is the output of trying to create an assignment:

$ otter assign bug.ipynb bug_dist
Traceback (most recent call last):
  File "/home/finotti/.venvs/DS/bin/otter", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/finotti/.venvs/DS/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/finotti/.venvs/DS/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/finotti/.venvs/DS/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/finotti/.venvs/DS/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/finotti/.venvs/DS/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/finotti/.venvs/DS/lib/python3.11/site-packages/otter/cli.py", line 32, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/home/finotti/.venvs/DS/lib/python3.11/site-packages/otter/cli.py", line 64, in assign_cli
    return assign(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/finotti/.venvs/DS/lib/python3.11/site-packages/otter/assign/__init__.py", line 157, in main
    run_tests(assignment, debug=debug)
  File "/home/finotti/.venvs/DS/lib/python3.11/site-packages/otter/assign/utils.py", line 230, in run_tests
    raise RuntimeError(f"Some autograder tests failed in the autograder notebook:\n" + \
RuntimeError: Some autograder tests failed in the autograder notebook:
    q1 results:
        q1 - 1 result:
            ❌ Test case failed
            Trying:
                t == 1
            Expecting:
                True
            **********************************************************************
            Line 1, in q1 0
            Failed example:
                t == 1
            Expected:
                True
            Got:
                False

Simple renaming the variable to something else, like tt makes it work.

I'm using Python 3.11.9 (from Debian Sid) and otter 5.5.0.

If relevant:

$ pip freeze
alabaster==0.7.16
annotated-types==0.7.0
anyio==4.4.0
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asttokens==2.4.1
astunparse==1.6.3
async-lru==2.0.4
attrs==23.2.0
Babel==2.15.0
beautifulsoup4==4.12.3
black==24.4.2
bleach==6.1.0
branca==0.7.2
certifi==2024.6.2
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
colorama==0.4.6
comm==0.2.2
contourpy==1.2.1
cycler==0.12.1
dacite==1.8.1
debugpy==1.8.2
decorator==5.1.1
defusedxml==0.7.1
dill==0.3.8
docutils==0.21.2
executing==2.0.1
fastjsonschema==2.20.0
fica==0.3.1
folium==0.17.0
fonttools==4.53.0
fqdn==1.5.1
gitdb==4.0.11
GitPython==3.1.43
h11==0.14.0
htmlmin==0.1.12
httpcore==1.0.5
httpx==0.27.0
idna==3.7
ImageHash==4.3.1
imagesize==1.4.1
ipykernel==6.29.4
ipylab==1.0.0
ipympl==0.9.4
ipython==8.26.0
ipython-genutils==0.2.0
ipywidgets==8.1.3
isoduration==20.11.0
jedi==0.19.1
Jinja2==3.1.4
joblib==1.4.2
json5==0.9.25
jsonpointer==3.0.0
jsonschema==4.22.0
jsonschema-specifications==2023.12.1
jupyter-events==0.10.0
jupyter-lsp==2.2.5
jupyter-server-mathjax==0.2.6
jupyter_client==8.6.2
jupyter_core==5.7.2
jupyter_server==2.14.1
jupyter_server_terminals==0.5.3
jupyterlab==4.2.4
jupyterlab-spellchecker==0.8.4
jupyterlab_code_formatter==3.0.0
jupyterlab_pygments==0.3.0
jupyterlab_server==2.27.2
jupyterlab_widgets==3.0.11
jupytext==1.16.4
kiwisolver==1.4.5
llvmlite==0.43.0
markdown-it-py==3.0.0
MarkupSafe==2.1.5
matplotlib==3.9.1
matplotlib-inline==0.1.7
mdit-py-plugins==0.4.1
mdurl==0.1.2
mistune==3.0.2
multimethod==1.11.2
mypy-extensions==1.0.0
nbclient==0.10.0
nbconvert==7.16.4
nbdime==4.0.1
nbformat==5.10.4
nest-asyncio==1.6.0
networkx==3.3
notebook_shim==0.2.4
numba==0.60.0
numpy==1.26.4
otter-grader==5.5.0
overrides==7.7.0
packaging==24.1
pandas==2.2.2
pandocfilters==1.5.1
parso==0.8.4
pathspec==0.12.1
patsy==0.5.6
pdfkit==1.0.0
pexpect==4.9.0
phik==0.12.4
pillow==10.3.0
platformdirs==4.2.2
prometheus_client==0.20.0
prompt_toolkit==3.0.47
psutil==6.0.0
ptyprocess==0.7.0
pure-eval==0.2.2
pyarrow==17.0.0
pycparser==2.22
pydantic==2.7.4
pydantic_core==2.18.4
Pygments==2.18.0
pyparsing==3.1.2
pypdf==4.3.1
python-dateutil==2.9.0.post0
python-json-logger==2.0.7
python-on-whales==0.71.0
pytz==2024.1
PyWavelets==1.6.0
PyYAML==6.0.1
pyzmq==26.0.3
referencing==0.35.1
requests==2.32.3
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rich==13.7.1
rpds-py==0.18.1
scipy==1.13.1
seaborn==0.13.2
Send2Trash==1.8.3
shellingham==1.5.4
six==1.16.0
smmap==5.0.1
sniffio==1.3.1
snowballstemmer==2.2.0
soupsieve==2.5
Sphinx==7.3.7
sphinxcontrib-applehelp==1.0.8
sphinxcontrib-devhelp==1.0.6
sphinxcontrib-htmlhelp==2.0.5
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.7
sphinxcontrib-serializinghtml==1.1.10
stack-data==0.6.3
statsmodels==0.14.2
terminado==0.18.1
tinycss2==1.3.0
tornado==6.4.1
tqdm==4.66.4
traitlets==5.14.3
typeguard==4.3.0
typer==0.12.3
types-python-dateutil==2.9.0.20240316
typing_extensions==4.12.2
tzdata==2024.1
uri-template==1.3.0
urllib3==2.2.2
visions==0.7.6
wcwidth==0.2.13
webcolors==24.6.0
webencodings==0.5.1
websocket-client==1.8.0
widgetsnbextension==4.0.11
wordcloud==1.9.3
wrapt==1.16.0
xlrd==2.0.1
xyzservices==2024.6.0
ydata-profiling==4.9.0

I can get around the issue (by renaming the variable), but I thought I should make the developers aware.

chrispyles commented 1 month ago

Interesting, it looks like a confluence of things is causing this bug. Because the only test case for the question is hidden, Otter by default doesn't include a check cell for the question after the solution cell, so all tests will get run at the end of the notebook. However, in order to run the tests, Otter's generated code reassigns the variable t:

from otter.execute import Checker
for t in ["./tests/q1.py"]:
    Checker.check_if_not_already_checked(t)

from otter.test_files import GradingResults
results = GradingResults(Checker.get_results())

import pickle
with open("/var/folders/c_/j_t422q90tq1fx3pc0r70jgw0000gn/T/tmp2b9m3vx7.pkl", "wb") as f:
    pickle.dump(results, f)

so t is no longer 1 when the test is run. It should be relatively simple to change the loop variable to something that won't cause collisions (i.e. a variable with a random name).