Closed gmlwns2000 closed 2 years ago
You can use --reduce
option to reduce the buggy input program as much as possible.
Suppose that you found a buggy input program (test_polished.c
) by using fuzzer. Then you can reduce the program by python3 tests/fuzz.py --print --reduce
. Reduced buggy input program will be saved to test_reduced.c
.
For more details, please refer to this.
However, I fix the problem. Fortunately, the problem was not about the C extension. @minseongg replied just few minutes ago, but I fixed my problem before I got response :sweat_smile:
Let me explain how I debug the fuzz task.
csmith
which is a random C code generator that fuzz.py
used. And I added easy
flag to fuzz.py
to ease the csmith
generation option (generates fewer lines of code)python3 tests/fuzz.py --print -n100 --seed 42
then the tester will stop soon since my code was not complete. And I got wrong C code...cargo run --bin fuzz --features=build-bin -- -p FILE
--reduce
is not workingSet seed as 42
Building KECC..
Finished release [optimized] target(s) in 0.08s
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "C.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
C-Reduce cannot run because the interestingness test does not return
zero. Please ensure that it does so not only in the directory where
you are invoking C-Reduce, but also in an arbitrary temporary
directory containing only the files that are being reduced. In other
words, running these commands:
DIR=`mktemp -d`
cp /home/s20200535/library/kecc-ainl/tests/test_reduced.c $DIR
cd $DIR
/home/s20200535/library/kecc-ainl/tests/reduce-criteria.sh
echo $?
should result in "0" being echoed to the terminal.
See "creduce --help" for more information.
Traceback (most recent call last):
File "tests/fuzz.py", line 304, in <module>
creduce(tests_dir, fuzz_arg)
File "tests/fuzz.py", line 210, in creduce
raise Exception("Reducing test_reduced.c by `{}` failed with exit code {}.".format(" ".join(args), proc.returncode))
Exception: Reducing test_reduced.c by `creduce --tidy ./reduce-criteria.sh test_reduced.c` failed with exit code 1.
I am not sure why this error is coming out. I am suspicious my changes on fuzz.py
Can you show your tests/reduced-criteria.sh
?
Hi @minseongg ,
My reduced-criteria.sh
is following.
#!/usr/bin/env bash
rm -f out*.txt
#ulimit -t 3000
#ulimit -v 2000000
if
[ $FUZZ_ARG = '-i' ] &&\
(! clang -pedantic -Wall -Werror=strict-prototypes -O1 -c test_reduced.c > out.txt 2>&1 ||\
grep 'main-return-type' out.txt ||\
grep 'conversions than data arguments' out.txt ||\
grep 'int-conversion' out.txt ||\
grep 'incompatible redeclaration' out.txt ||\
grep 'ordered comparison between pointer and zero' out.txt ||\
grep 'ordered comparison between pointer and integer' out.txt ||\
grep 'eliding middle term' out.txt ||\
grep 'end of non-void function' out.txt ||\
grep 'invalid in C99' out.txt ||\
grep 'specifies type' out.txt ||\
grep 'should return a value' out.txt ||\
grep 'uninitialized' out.txt ||\
grep 'incompatible pointer to' out.txt ||\
grep 'incompatible integer to' out.txt ||\
grep 'type specifier missing' out.txt ||\
grep 'implicit-function-declaration' out.txt ||\
grep 'infinite-recursion' out.txt ||\
grep 'pointer-bool-conversion' out.txt ||\
grep 'non-void function does not return a value' out.txt ||\
grep 'too many arguments in call' out.txt ||\
grep 'declaration does not declare anything' out.txt ||\
grep 'not equal to a null pointer is always true' out.txt ||\
grep 'empty struct is a GNU extension' out.txt ||\
! gcc -Wall -Wextra -O2 test_reduced.c > outa.txt 2>&1 ||\
grep 'uninitialized' outa.txt ||\
grep 'without a cast' outa.txt ||\
grep 'control reaches end' outa.txt ||\
grep 'return type defaults' outa.txt ||\
grep 'cast from pointer to integer' outa.txt ||\
grep 'useless type name in empty declaration' outa.txt ||\
grep 'no semicolon at end' outa.txt ||\
grep 'type defaults to' outa.txt ||\
grep 'too few arguments for format' outa.txt ||\
grep 'ordered comparison of pointer with integer' outa.txt ||\
grep 'declaration does not declare anything' outa.txt ||\
grep 'expects type' outa.txt ||\
grep 'pointer from integer' outa.txt ||\
grep 'incompatible implicit' outa.txt ||\
grep 'excess elements in struct initializer' outa.txt ||\
grep 'comparison between pointer and integer' outa.txt ||\
! gcc -O1 test_reduced.c > cc_out1.txt 2>&1 ||\
! gcc -O2 test_reduced.c > cc_out2.txt 2>&1 ||\
! cargo run --manifest-path $PROJECT_DIR/Cargo.toml --features=build-bin --release -- --parse test_reduced.c >/dev/null 2>&1)
then
exit 1
fi
cargo run --manifest-path $PROJECT_DIR/Cargo.toml --features=build-bin --release --bin fuzz -- $FUZZ_ARG test_reduced.c 2>&1 | grep -q 'assertion failed'
And my modified fuzz.py
is here
https://github.com/kaist-cp/cs420/issues/337#issuecomment-1060626423 This looks like your reduced-criteria-template.sh
. Can you show your reduced-criteria.sh
?
Also, it seems your modified fuzz.py
has an indentation error in L272.
FYI: To reduce the input program properly, the last line of your reduced-criteria.sh
should return zero on the tests
directory.
# This is the last line of your `reduced-criteria-template.sh`. `$PROJECT_DIR` and `$FUZZ_ARG` will be resolved to the actual value in your `reduced-criteria.sh`.
cargo run --manifest-path $PROJECT_DIR/Cargo.toml --features=build-bin --release --bin fuzz -- $FUZZ_ARG test_reduced.c 2>&1 | grep -q 'assertion failed'
Thanks, @minseongg , I fixed it.
And here is reduced-criteria.sh
. Sorry for the inconvenience.
#!/usr/bin/env bash
rm -f out*.txt
#ulimit -t 3000
#ulimit -v 2000000
if
[ -p = '-i' ] &&\
(! clang -pedantic -Wall -Werror=strict-prototypes -O1 -c test_reduced.c > out.txt 2>&1 ||\
grep 'main-return-type' out.txt ||\
grep 'conversions than data arguments' out.txt ||\
grep 'int-conversion' out.txt ||\
grep 'incompatible redeclaration' out.txt ||\
grep 'ordered comparison between pointer and zero' out.txt ||\
grep 'ordered comparison between pointer and integer' out.txt ||\
grep 'eliding middle term' out.txt ||\
grep 'end of non-void function' out.txt ||\
grep 'invalid in C99' out.txt ||\
grep 'specifies type' out.txt ||\
grep 'should return a value' out.txt ||\
grep 'uninitialized' out.txt ||\
grep 'incompatible pointer to' out.txt ||\
grep 'incompatible integer to' out.txt ||\
grep 'type specifier missing' out.txt ||\
grep 'implicit-function-declaration' out.txt ||\
grep 'infinite-recursion' out.txt ||\
grep 'pointer-bool-conversion' out.txt ||\
grep 'non-void function does not return a value' out.txt ||\
grep 'too many arguments in call' out.txt ||\
grep 'declaration does not declare anything' out.txt ||\
grep 'not equal to a null pointer is always true' out.txt ||\
grep 'empty struct is a GNU extension' out.txt ||\
! gcc -Wall -Wextra -O2 test_reduced.c > outa.txt 2>&1 ||\
grep 'uninitialized' outa.txt ||\
grep 'without a cast' outa.txt ||\
grep 'control reaches end' outa.txt ||\
grep 'return type defaults' outa.txt ||\
grep 'cast from pointer to integer' outa.txt ||\
grep 'useless type name in empty declaration' outa.txt ||\
grep 'no semicolon at end' outa.txt ||\
grep 'type defaults to' outa.txt ||\
grep 'too few arguments for format' outa.txt ||\
grep 'ordered comparison of pointer with integer' outa.txt ||\
grep 'declaration does not declare anything' outa.txt ||\
grep 'expects type' outa.txt ||\
grep 'pointer from integer' outa.txt ||\
grep 'incompatible implicit' outa.txt ||\
grep 'excess elements in struct initializer' outa.txt ||\
grep 'comparison between pointer and integer' outa.txt ||\
! gcc -O1 test_reduced.c > cc_out1.txt 2>&1 ||\
! gcc -O2 test_reduced.c > cc_out2.txt 2>&1 ||\
! cargo run --manifest-path /home/s20200535/library/kecc-ainl/Cargo.toml --features=build-bin --release -- --parse test_reduced.c >/dev/null 2>&1)
then
exit 1
fi
cargo run --manifest-path /home/s20200535/library/kecc-ainl/Cargo.toml --features=build-bin --release --bin fuzz -- -p test_reduced.c 2>&1 | grep -q 'assertion failed'
What is the return value of the last line? Please try the below commands on your tests
directory.
cargo run --manifest-path /home/s20200535/library/kecc-ainl/Cargo.toml --features=build-bin --release --bin fuzz -- -p test_reduced.c 2>&1 | grep -q 'assertion failed'
echo $?
The output is empty.
It seems your output of the last line is empty. You can see the return value by using echo $?
.
Then what is the output of the below command? Is it containing assertion failed
?
cargo run --manifest-path /home/s20200535/library/kecc-ainl/Cargo.toml --features=build-bin --release --bin fuzz -- -p test_reduced.c 2>&1
The result of echo $?
is 1
.
And the result of given commend is,
s20200535@kaist-cp-cs420:~/library/kecc-ainl/tests$ cargo run --manifest-path /home/s20200535/library/kecc-ainl/Cargo.toml --features=build-bin --release --bin fuzz -- -p test_reduced.c 2>&1
Finished release [optimized] target(s) in 0.07s
Running `/home/s20200535/library/kecc-ainl/target/release/fuzz -p test_reduced.c`
Seems fine for me...
It seems your implementation does not have a bug for test_reduced.c
. Reducing works only for the buggy input program. (For example, assertion failed
should be in your output.)
So, if you want to use --reduce
option properly, please find the buggy input program first by using python3 tests/fuzz.py --print
(if the fuzzer finds a buggy input program, it terminates with an error) and then use python3 tests/fuzz.py --print --reduce
. Then you will be able to debug your implementation with reduced buggy input program.
Hi,
Oh, I understand about it now. I will try it on the next homework!
We added user's manual for the fuzzer. (https://github.com/kaist-cp/kecc-public/blob/main/tests/README.md)
Also, I'll review your pull request soon. Thanks!
Eventually, I passed the test case
test_examples_write_c
.I implement as accurately as I could, however, still I am not sure about my implementation of some
c
extensions such as__asm__
,_FloatXx
. So I did not expect I could pass thefuzz
. So I tried to debug my code withfuzz
.However, I am struggling at debugging
python3 tests/fuzz.py --print -n30
, sincefuzz.py
generates about 4000 ~ 5000 lines with non-human-friendly codes.How could I debug my code effectively?
I want to do the following jobs during debugging.
C
statements.