Maybe a bug about the crash oracle?

[Kristoff-starling]

Issue

In run_code() (tf_library.py), the current code wraps exec(code) with "try" in order to catch unexpected exceptions:

try:
    exec(code)
    MARK_DONE_FLAG = True
except Exception as e:
    error = str(e)

However, the target code has been wrapped with a layer of "try" in generate_code(), so any exceptions during execution will be caught inside the target code and recorded in results dictionary. In this situation, error and MARK_DONE_FLAG will maintain the default value (None, True) and the check code in test_with_oracle() won't be able to detect the failure and invalid code will be put into the "success" directory:

results, error, MARK_DONE_FLAG = self.run_code(code)
if not MARK_DONE_FLAG:
    self.write_to_dir(join(self.output[oracle], "potential-bug"), api.api, code)
elif error == None:
    self.write_to_dir(join(self.output[oracle], "success"), api.api, code)
else:
    self.write_to_dir(join(self.output[oracle], "fail"), api.api, code)

Reproduction

+       code = "\
+try:\n\
+   unrunnable\n\
+except Exception as e:\n\
+   results['err'] = str(e)"
        try:
            exec(code)
            MARK_DONE_FLAG = True
        except Exception as e:
            error = str(e)

If we directly substitute the target code in run_code() with an invalid code shown above, as long as the code doesn't have syntax error, the execution will end normally and error won't catch the exception. In FreeFuzz's outputs, there also exists tests that offer invalid arguments to TF APIs and get away with the checker.

Solution

In tf.library.py, we can check the execution result by checking whether results['err'] exists instead of checking the value of error.

[dengyinlin]

Hi @Kristoff-starling , thank you very much for pointing out the bug and for kindly providing the solution! Indeed we should check results['err'] instead of error in run_code() (tf_library.py). The bug should be fixed after https://github.com/ise-uiuc/FreeFuzz/commit/703181a0c7c105d74d3aa4e497b83ffa32621315.

Just want to point out that this issue does not affect the crash oracle much. This is because usually when crash is triggered, TensorFlow fails to throw any Exception and the process just got terminated. In this case, there is no way to detect such crashes via catching exception. Instead, such crashes will be captured by examining the return code of the test process in https://github.com/ise-uiuc/FreeFuzz/blob/main/src/FreeFuzz.py#L62. On the other hand, results['err'] usually just catches some known exceptions that we do not consider as bugs or bug candidates.

[Kristoff-starling]

Hi @Kristoff-starling , thank you very much for pointing out the bug and for kindly providing the solution! Indeed we should check results['err'] instead of error in run_code() (tf_library.py). The bug should be fixed after 703181a.

Just want to point out that this issue does not affect the crash oracle much. This is because usually when crash is triggered, TensorFlow fails to throw any Exception and the process just got terminated. In this case, there is no way to detect such crashes via catching exception. Instead, such crashes will be captured by examining the return code of the test process in https://github.com/ise-uiuc/FreeFuzz/blob/main/src/FreeFuzz.py#L62. On the other hand, results['err'] usually just catches some known exceptions that we do not consider as bugs or bug candidates.

I think that I misunderstood the criteria for "crash" previously and by checking the return code of sub-processes serious crashes can indeed be detected. Thanks for your reply!