Autograder with Java kernel: How to implement assertions?

adibaba commented 6 years ago

Operating system

cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04 LTS"

`nbgrader --version`

Python version 3.6.5 | packaged by conda-forge | (default, Apr 6 2018, 13:39:56) [GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] nbgrader version 0.5.4

`jupyterhub --version` (if used with JupyterHub)

0.9.0rc1

`jupyter notebook --version`

5.5.0

Issue

I am using nbgrader on Jupyterhub with the scijava kernel. Code execution and importing packages works fine. The implementation of autograder tests with assert statements fail. A statement assert (1==1) is evaluated and nothing happens, which is fine. But a statement assert (1==2) produces an error inside the kernel, which is not recognized by nbgrader. Additionally, the autograder assigns full credit.

Is there a way to evaluate Java assertions? If not: How can errors be identified to not assign credits?

Expected behavior

Assertion is evaluated as false.
Autograder does not assign credits

Actual behavior

Assertion produces error (see below)
Autograder assigns full credit

[ERROR] null
Assertion failed: 

assert (1==2)
         |
         false

    at org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:404)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:650)
    at Script5.run(Script5.groovy:1)
    at org.scijava.plugins.scripting.groovy.GroovyScriptEngine.eval(GroovyScriptEngine.java:303)
    at org.scijava.plugins.scripting.groovy.GroovyScriptEngine.eval(GroovyScriptEngine.java:122)
    at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
    at org.scijava.script.ScriptModule.run(ScriptModule.java:160)
    at org.scijava.module.ModuleRunner.run(ModuleRunner.java:168)
    at org.scijava.jupyter.kernel.evaluator.Worker.run(Worker.java:108)
    at org.scijava.thread.DefaultThreadService$2.run(DefaultThreadService.java:221)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Steps to reproduce the behavior

Install Jupyterhub docker version (https://hub.docker.com/r/jupyterhub/jupyterhub/)
Install nbgrader (http://nbgrader.readthedocs.io/en/stable/user_guide/installation.html)
Install scijava-jupyter-kernel (https://github.com/scijava/scijava-jupyter-kernel)
If the kernel does not appear in the frontend, take a look at these solutions: https://github.com/scijava/scijava-jupyter-kernel/issues/79
Create a notebook with the assertion above

jhamrick commented 6 years ago

This sounds like a bug with the java kernel, would you mind filing an issue there instead? Specifically, the thing that needs to happen is for the java kernel to send an error message kernel reply, which will cause the notebook to create an "error" output: https://github.com/jupyter/nbformat/blob/master/nbformat/v4/nbbase.py#L55

nbgrader then looks for this error output. If it's assigning full credit, that means that there is no error output on the cell which makes me think it's a problem with the java kernel not sending the right type of response.

hkarl commented 6 years ago

Hello,

I was wondering if any progress has happened here? Will have the same issue as the OP soon...

thanks,

HK

adibaba commented 6 years ago

No updates from the project. Our workflow includes to use relational operators instead of assertions an to throw Exceptions for negative results. Have also a look at the kernel named IJava.

adibaba commented 6 years ago

Using the IJava kernel, there are two solutions:

Use kernel.json to set the ea flag
Use JUnit Assertions

The kernel developer provided a comprehensive answer: https://github.com/SpencerPark/IJava/issues/30#issuecomment-420713306

hkarl commented 6 years ago

Forget it, I was being stupid; staring at the wrong cells... it seems it might work.

hkarl commented 6 years ago

Forget it, I was being stupid; staring at the wrong cells... it seems it might work.

jhamrick commented 6 years ago

Closing this, as it seems there is a solution using the IJava kernel.

jupyter / nbgrader