search

kaist-cp / cs420

KAIST CS420: Compiler Design (2023 Spring)
418 stars 27 forks source link

[Question] Effective Debugging on Fuzzed Test Cases #337

Closed gmlwns2000 closed 2 years ago

gmlwns2000 commented 2 years ago

Eventually, I passed the test case test_examples_write_c.

I implement as accurately as I could, however, still I am not sure about my implementation of some c extensions such as __asm__, _FloatXx. So I did not expect I could pass the fuzz. So I tried to debug my code with fuzz.

However, I am struggling at debugging python3 tests/fuzz.py --print -n30, since fuzz.py generates about 4000 ~ 5000 lines with non-human-friendly codes.

How could I debug my code effectively?

I want to do the following jobs during debugging.

minseongg commented 2 years ago

You can use --reduce option to reduce the buggy input program as much as possible.

Suppose that you found a buggy input program (test_polished.c) by using fuzzer. Then you can reduce the program by python3 tests/fuzz.py --print --reduce. Reduced buggy input program will be saved to test_reduced.c.

For more details, please refer to this.

gmlwns2000 commented 2 years ago

However, I fix the problem. Fortunately, the problem was not about the C extension. @minseongg replied just few minutes ago, but I fixed my problem before I got response :sweat_smile:

How I worked

Let me explain how I debug the fuzz task.

  1. I fixed the seed of csmith which is a random C code generator that fuzz.py used. And I added easy flag to fuzz.py to ease the csmith generation option (generates fewer lines of code)
  2. Run python3 tests/fuzz.py --print -n100 --seed 42 then the tester will stop soon since my code was not complete. And I got wrong C code...
  3. Comment all lines in wrong C code, then un-comment part by part then run cargo run --bin fuzz --features=build-bin -- -p FILE
  4. Then I got the source chunk of bug, debug with that.
  5. Repeat until fuzz is passed.

However, --reduce is not working

Set seed as 42
Building KECC..
    Finished release [optimized] target(s) in 0.08s
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LC_CTYPE = "C.UTF-8",
        LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

C-Reduce cannot run because the interestingness test does not return
zero. Please ensure that it does so not only in the directory where
you are invoking C-Reduce, but also in an arbitrary temporary
directory containing only the files that are being reduced. In other
words, running these commands:

  DIR=`mktemp -d`
  cp /home/s20200535/library/kecc-ainl/tests/test_reduced.c $DIR
  cd $DIR
  /home/s20200535/library/kecc-ainl/tests/reduce-criteria.sh
  echo $?

should result in "0" being echoed to the terminal.

See "creduce --help" for more information.

Traceback (most recent call last):
  File "tests/fuzz.py", line 304, in <module>
    creduce(tests_dir, fuzz_arg)
  File "tests/fuzz.py", line 210, in creduce
    raise Exception("Reducing test_reduced.c by `{}` failed with exit code {}.".format(" ".join(args), proc.returncode))
Exception: Reducing test_reduced.c by `creduce --tidy ./reduce-criteria.sh test_reduced.c` failed with exit code 1.

I am not sure why this error is coming out. I am suspicious my changes on fuzz.py

minseongg commented 2 years ago

Can you show your tests/reduced-criteria.sh?

gmlwns2000 commented 2 years ago

Hi @minseongg , My reduced-criteria.sh is following.

#!/usr/bin/env bash

rm -f out*.txt

#ulimit -t 3000                                                                                                                                                                                             
#ulimit -v 2000000                                                                                                                                                                                          

if
  [ $FUZZ_ARG = '-i' ] &&\
  (! clang -pedantic -Wall -Werror=strict-prototypes -O1 -c test_reduced.c  > out.txt 2>&1 ||\
  grep 'main-return-type' out.txt ||\
  grep 'conversions than data arguments' out.txt ||\
  grep 'int-conversion' out.txt ||\
  grep 'incompatible redeclaration' out.txt ||\
  grep 'ordered comparison between pointer and zero' out.txt ||\
  grep 'ordered comparison between pointer and integer' out.txt ||\
  grep 'eliding middle term' out.txt ||\
  grep 'end of non-void function' out.txt ||\
  grep 'invalid in C99' out.txt ||\
  grep 'specifies type' out.txt ||\
  grep 'should return a value' out.txt ||\
  grep 'uninitialized' out.txt ||\
  grep 'incompatible pointer to' out.txt ||\
  grep 'incompatible integer to' out.txt ||\
  grep 'type specifier missing' out.txt ||\
  grep 'implicit-function-declaration' out.txt ||\
  grep 'infinite-recursion' out.txt ||\
  grep 'pointer-bool-conversion' out.txt ||\
  grep 'non-void function does not return a value' out.txt ||\
  grep 'too many arguments in call' out.txt ||\
  grep 'declaration does not declare anything' out.txt ||\
  grep 'not equal to a null pointer is always true' out.txt ||\
  grep 'empty struct is a GNU extension' out.txt ||\
  ! gcc -Wall -Wextra -O2 test_reduced.c > outa.txt 2>&1 ||\
  grep 'uninitialized' outa.txt ||\
  grep 'without a cast' outa.txt ||\
  grep 'control reaches end' outa.txt ||\
  grep 'return type defaults' outa.txt ||\
  grep 'cast from pointer to integer' outa.txt ||\
  grep 'useless type name in empty declaration' outa.txt ||\
  grep 'no semicolon at end' outa.txt ||\
  grep 'type defaults to' outa.txt ||\
  grep 'too few arguments for format' outa.txt ||\
  grep 'ordered comparison of pointer with integer' outa.txt ||\
  grep 'declaration does not declare anything' outa.txt ||\
  grep 'expects type' outa.txt ||\
  grep 'pointer from integer' outa.txt ||\
  grep 'incompatible implicit' outa.txt ||\
  grep 'excess elements in struct initializer' outa.txt ||\
  grep 'comparison between pointer and integer' outa.txt ||\
  ! gcc -O1 test_reduced.c > cc_out1.txt 2>&1 ||\
  ! gcc -O2 test_reduced.c > cc_out2.txt 2>&1 ||\
  ! cargo run --manifest-path $PROJECT_DIR/Cargo.toml --features=build-bin --release -- --parse test_reduced.c >/dev/null 2>&1)
then
  exit 1
fi

cargo run --manifest-path $PROJECT_DIR/Cargo.toml --features=build-bin --release --bin fuzz -- $FUZZ_ARG test_reduced.c 2>&1 | grep -q 'assertion failed'
gmlwns2000 commented 2 years ago

And my modified fuzz.py is here

minseongg commented 2 years ago

https://github.com/kaist-cp/cs420/issues/337#issuecomment-1060626423 This looks like your reduced-criteria-template.sh. Can you show your reduced-criteria.sh?

Also, it seems your modified fuzz.py has an indentation error in L272.

FYI: To reduce the input program properly, the last line of your reduced-criteria.sh should return zero on the tests directory.

# This is the last line of your `reduced-criteria-template.sh`. `$PROJECT_DIR` and `$FUZZ_ARG` will be resolved to the actual value in your `reduced-criteria.sh`.
cargo run --manifest-path $PROJECT_DIR/Cargo.toml --features=build-bin --release --bin fuzz -- $FUZZ_ARG test_reduced.c 2>&1 | grep -q 'assertion failed'
gmlwns2000 commented 2 years ago

Thanks, @minseongg , I fixed it. And here is reduced-criteria.sh. Sorry for the inconvenience.

#!/usr/bin/env bash

rm -f out*.txt

#ulimit -t 3000                                                                                                                                                                                             
#ulimit -v 2000000                                                                                                                                                                                          

if
  [ -p = '-i' ] &&\
  (! clang -pedantic -Wall -Werror=strict-prototypes -O1 -c test_reduced.c  > out.txt 2>&1 ||\
  grep 'main-return-type' out.txt ||\
  grep 'conversions than data arguments' out.txt ||\
  grep 'int-conversion' out.txt ||\
  grep 'incompatible redeclaration' out.txt ||\
  grep 'ordered comparison between pointer and zero' out.txt ||\
  grep 'ordered comparison between pointer and integer' out.txt ||\
  grep 'eliding middle term' out.txt ||\
  grep 'end of non-void function' out.txt ||\
  grep 'invalid in C99' out.txt ||\
  grep 'specifies type' out.txt ||\
  grep 'should return a value' out.txt ||\
  grep 'uninitialized' out.txt ||\
  grep 'incompatible pointer to' out.txt ||\
  grep 'incompatible integer to' out.txt ||\
  grep 'type specifier missing' out.txt ||\
  grep 'implicit-function-declaration' out.txt ||\
  grep 'infinite-recursion' out.txt ||\
  grep 'pointer-bool-conversion' out.txt ||\
  grep 'non-void function does not return a value' out.txt ||\
  grep 'too many arguments in call' out.txt ||\
  grep 'declaration does not declare anything' out.txt ||\
  grep 'not equal to a null pointer is always true' out.txt ||\
  grep 'empty struct is a GNU extension' out.txt ||\
  ! gcc -Wall -Wextra -O2 test_reduced.c > outa.txt 2>&1 ||\
  grep 'uninitialized' outa.txt ||\
  grep 'without a cast' outa.txt ||\
  grep 'control reaches end' outa.txt ||\
  grep 'return type defaults' outa.txt ||\
  grep 'cast from pointer to integer' outa.txt ||\
  grep 'useless type name in empty declaration' outa.txt ||\
  grep 'no semicolon at end' outa.txt ||\
  grep 'type defaults to' outa.txt ||\
  grep 'too few arguments for format' outa.txt ||\
  grep 'ordered comparison of pointer with integer' outa.txt ||\
  grep 'declaration does not declare anything' outa.txt ||\
  grep 'expects type' outa.txt ||\
  grep 'pointer from integer' outa.txt ||\
  grep 'incompatible implicit' outa.txt ||\
  grep 'excess elements in struct initializer' outa.txt ||\
  grep 'comparison between pointer and integer' outa.txt ||\
  ! gcc -O1 test_reduced.c > cc_out1.txt 2>&1 ||\
  ! gcc -O2 test_reduced.c > cc_out2.txt 2>&1 ||\
  ! cargo run --manifest-path /home/s20200535/library/kecc-ainl/Cargo.toml --features=build-bin --release -- --parse test_reduced.c >/dev/null 2>&1)
then
  exit 1
fi

cargo run --manifest-path /home/s20200535/library/kecc-ainl/Cargo.toml --features=build-bin --release --bin fuzz -- -p test_reduced.c 2>&1 | grep -q 'assertion failed'
minseongg commented 2 years ago

What is the return value of the last line? Please try the below commands on your tests directory.

cargo run --manifest-path /home/s20200535/library/kecc-ainl/Cargo.toml --features=build-bin --release --bin fuzz -- -p test_reduced.c 2>&1 | grep -q 'assertion failed'
echo $?
gmlwns2000 commented 2 years ago

The output is empty.

minseongg commented 2 years ago

It seems your output of the last line is empty. You can see the return value by using echo $?.

Then what is the output of the below command? Is it containing assertion failed?

cargo run --manifest-path /home/s20200535/library/kecc-ainl/Cargo.toml --features=build-bin --release --bin fuzz -- -p test_reduced.c 2>&1
gmlwns2000 commented 2 years ago

The result of echo $? is 1.

And the result of given commend is,

s20200535@kaist-cp-cs420:~/library/kecc-ainl/tests$ cargo run --manifest-path /home/s20200535/library/kecc-ainl/Cargo.toml --features=build-bin --release --bin fuzz -- -p test_reduced.c 2>&1
    Finished release [optimized] target(s) in 0.07s
     Running `/home/s20200535/library/kecc-ainl/target/release/fuzz -p test_reduced.c`

Seems fine for me...

minseongg commented 2 years ago

It seems your implementation does not have a bug for test_reduced.c. Reducing works only for the buggy input program. (For example, assertion failed should be in your output.)

So, if you want to use --reduce option properly, please find the buggy input program first by using python3 tests/fuzz.py --print (if the fuzzer finds a buggy input program, it terminates with an error) and then use python3 tests/fuzz.py --print --reduce. Then you will be able to debug your implementation with reduced buggy input program.

gmlwns2000 commented 2 years ago

Hi,

Oh, I understand about it now. I will try it on the next homework!

minseongg commented 2 years ago

We added user's manual for the fuzzer. (https://github.com/kaist-cp/kecc-public/blob/main/tests/README.md)

Also, I'll review your pull request soon. Thanks!