Running test failed for fuzzolic runnner

[Tricker-z]

Hi, fuzzolic devs

We have been utilizing fuzzolic for vulnerability detection recently. Unfortunately, we found that two of the internal tests failed. Here are the detailed steps and error messages.

$ docker run -ti --rm ercoppa/fuzzolic-runner-v1
$ make -C tests/ run

1. test_symbolic_index

...
        if perf_run:
            slowdown = emulated_time / native_time
            print("Slowdown: %s" % round(slowdown, 1))
            assert slowdown < 70

        if expected_inputs > 0:
            testcases = glob.glob(WORKDIR + "/tests/test_*.dat")
>           assert len(testcases) == expected_inputs
E           assert 0 == 1
E            +  where 0 = len([])

run.py:85: AssertionError

Running directory: /home/ubuntu/fuzzolic/tests/workdir/fuzzolic-00000
Using SMT solver
Run took 0.2 secs
[FUZZOLIC] no more testcase. Finishing.

2. test_switch

        if perf_run:
            slowdown = emulated_time / native_time
            print("Slowdown: %s" % round(slowdown, 1))
            assert slowdown < 70

        if expected_inputs > 0:
            testcases = glob.glob(WORKDIR + "/tests/test_*.dat")
>           assert len(testcases) == expected_inputs
E           AssertionError: assert 2 == 7
E            +  where 2 = len(['/home/ubuntu/fuzzolic/tests/workdir/tests/test_case_000_000.dat', '/home/ubuntu/fuzzolic/tests/workdir/tests/test_case_000_001.dat'])

run.py:85: AssertionError

Running directory: /home/ubuntu/fuzzolic/tests/workdir/fuzzolic-00000
Using SMT solver
[+] Keeping test_case_1_1.dat
[+] Keeping test_case_0_0.dat
Run took 0.2 secs

Running directory: /home/ubuntu/fuzzolic/tests/workdir/fuzzolic-00001
Using SMT solver
Run took 0.2 secs

Running directory: /home/ubuntu/fuzzolic/tests/workdir/fuzzolic-00002
Using SMT solver
Run took 0.2 secs
[FUZZOLIC] no more testcase. Finishing.

We would like to know whether these test fails affect the normal use of the program, because it seems that fuzzolic fails to handle some situations well.

Really appreciate your help!

[ercoppa]

We would like to know whether these test fails affect the normal use of the program

They should not affect the normal usage. The failing tests require to recompile fuzzolic with some experimental features. The current run.py skips the failing tests. I need to update the docker container.

because it seems that fuzzolic fails to handle some situations well.

Can you please provide some concrete examples? Unfortunately, there are cases that are not handled well by fuzzolic but without details is hard for me to point the finger toward a limitation or a possible bug.

[Tricker-z]

In fact, I'm currently learning fuzzolic and then just worrying that a failed test means the wrong feature. Since it does not affect the normal functionality, I expect fuzzolic to be better and will keep watching :)