gradescope / gradescope-utils

Python package for building Gradescope autograders
https://gradescope-utils.readthedocs.io/en/latest/
MIT License
34 stars 25 forks source link

Setting buffer to False in JSONTestRunner throws an error #23

Closed PrashRangarajan closed 1 year ago

PrashRangarajan commented 3 years ago

Describe the bug When using the JSONTestRunner class to obtain the test results in a JSON format, it throws an error when the input parameter buffer=False if any of the test cases fails.

To Reproduce We create a new testcase that fails.

import unittest
from gradescope_utils.autograder_utils.decorators import weight, visibility

class TestAutograder(unittest.TestCase):
    @weight(1)
    @visibility("visible")
    def test_auto(self):
        """
        A test that throws an error
        """
        self.assertTrue(False)

Running the above with the JSONTestRunner provided in gradescope_utils produces the following error:

TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str'

Expected behavior As mentioned in line 132 of the file json_test_runner.py, we Set buffer to True to include test output in JSON. So ideally, when it is False, there should be an empty output for the test.

Potential causes/fix When buffer is set to False, the getOutput method in JSONTestResult returns None. As a result, in the method buildResult, the variable output gets set to None as well. In line 77, if there is an error, the method tries to append the message Test Failed to the output, but since output is None, it is unable to execute the line output += "Test Failed: {0}\n".format(err[1]) and hence throws an error. To fix this, a check needs to be put on output (to see if it is not None) before appending the Test failed message onto it.

ibrahima commented 1 year ago

Hi @PrashRangarajan! Thanks for reporting this issue! So sorry that it took so long to get back to you. We'll look into fixing this issue and try and cut a new release soon.