search

google / fuzzbench

FuzzBench - Fuzzer benchmarking as a service.
https://google.github.io/fuzzbench/
Apache License 2.0
1.11k stars 269 forks source link

[SBFT24] Fox #1923

Open prashast opened 11 months ago

vrthra commented 11 months ago

Dear participants,

Thank you for the PR. This is a heads-up that your branch will need to be rebased on the SBFT branch once it is ready. We are still in progress of preparing the branch and will message you once we are done. This branch adds the functionality of adding mutation testing as a measurement criteria. We do not expect this to affect the fuzzer integration. Hence, we do not expect any conflicts, however, if there are any, we will help you resolve them.

phi-go commented 10 months ago

Hello, it seems the runner image does not quite build:

 > importing cache manifest from gcr.io/fuzzbench/runners/fox/freetype2_ftfuzzer-intermediate:
------
------
 > [2/2] RUN apt install -y unzip git gdb joe:
#4 0.217 
#4 0.217 WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
#4 0.217 
#4 0.257 Reading package lists...
#4 1.047 Building dependency tree...
#4 1.150 Reading state information...
#4 1.268 The following additional packages will be installed:
#4 1.268   file gdbserver git-man less libbabeltrace1 libbsd0 libc6-dbg libcbor0.6
#4 1.268   libcurl3-gnutls libdw1 libedit2 libelf1 liberror-perl libexpat1 libfido2-1
#4 1.268   libglib2.0-0 libglib2.0-data libicu66 libmagic-mgc libmagic1 libmpdec2
#4 1.268   libpython3.8 libpython3.8-minimal libpython3.8-stdlib libx11-6 libx11-data
#4 1.268   libxau6 libxcb1 libxdmcp6 libxext6 libxml2 libxmuu1 mime-support
#4 1.268   openssh-client shared-mime-info tzdata xauth xdg-user-dirs
#4 1.269 Suggested packages:
#4 1.269   gdb-doc gettext-base git-daemon-run | git-daemon-sysvinit git-doc git-el
#4 1.269   git-email git-gui gitk gitweb git-cvs git-mediawiki git-svn keychain
#4 1.269   libpam-ssh monkeysphere ssh-askpass zip
#4 1.336 The following NEW packages will be installed:
#4 1.336   file gdb gdbserver git git-man joe less libbabeltrace1 libbsd0 libc6-dbg
#4 1.336   libcbor0.6 libcurl3-gnutls libdw1 libedit2 libelf1 liberror-perl libexpat1
#4 1.336   libfido2-1 libglib2.0-0 libglib2.0-data libicu66 libmagic-mgc libmagic1
#4 1.336   libmpdec2 libpython3.8 libpython3.8-minimal libpython3.8-stdlib libx11-6
#4 1.337   libx11-data libxau6 libxcb1 libxdmcp6 libxext6 libxml2 libxmuu1 mime-support
#4 1.337   openssh-client shared-mime-info tzdata unzip xauth xdg-user-dirs
#4 1.486 0 upgraded, 42 newly installed, 0 to remove and 0 not upgraded.
#4 1.486 Need to get 47.7 MB of archives.
#4 1.486 After this operation, 166 MB of additional disk space will be used.
#4 1.486 Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libmagic-mgc amd64 1:5.38-4 [218 kB]
#4 1.581 Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libmagic1 amd64 1:5.38-4 [75.9 kB]
#4 1.585 Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 file amd64 1:5.38-4 [23.3 kB]
#4 1.586 Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 less amd64 551-1ubuntu0.1 [123 kB]
#4 1.599 Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libbsd0 amd64 0.10.0-1 [45.4 kB]
#4 1.601 Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libelf1 amd64 0.176-1.1ubuntu0.1 [44.2 kB]
#4 1.603 Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libexpat1 amd64 2.2.9-1ubuntu0.6 [74.6 kB]
#4 1.608 Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libglib2.0-0 amd64 2.64.6-1~ubuntu20.04.6 [1289 kB]
#4 1.651 Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libglib2.0-data all 2.64.6-1~ubuntu20.04.6 [6032 B]
#4 1.651 Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 tzdata all 2023c-0ubuntu0.20.04.2 [301 kB]
#4 1.657 Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libicu66 amd64 66.1-2ubuntu2.1 [8515 kB]
#4 1.811 Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 libmpdec2 amd64 2.4.2-3 [81.1 kB]
#4 1.878 Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-minimal amd64 3.8.10-0ubuntu1~20.04.9 [718 kB]
#4 1.943 Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 mime-support all 3.64ubuntu1 [30.6 kB]
#4 1.944 Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-stdlib amd64 3.8.10-0ubuntu1~20.04.9 [1674 kB]
#4 1.985 Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libxml2 amd64 2.9.10+dfsg-5ubuntu0.20.04.6 [640 kB]
#4 1.995 Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 shared-mime-info amd64 1.15-1 [430 kB]
#4 2.000 Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 xdg-user-dirs amd64 0.17-2ubuntu1 [48.3 kB]
#4 2.000 Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 libcbor0.6 amd64 0.6.0-0ubuntu1 [21.1 kB]
#4 2.001 Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 libedit2 amd64 3.1-20191231-1 [87.0 kB]
#4 2.002 Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 libfido2-1 amd64 1.3.1-1ubuntu2 [47.9 kB]
#4 2.002 Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 libxau6 amd64 1:1.0.9-0ubuntu1 [7488 B]
#4 2.038 Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 libxdmcp6 amd64 1:1.1.3-0ubuntu1 [10.6 kB]
#4 2.056 Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 libxcb1 amd64 1.14-2 [44.7 kB]
#4 2.083 Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libx11-data all 2:1.6.9-2ubuntu1.6 [114 kB]
#4 2.113 Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libx11-6 amd64 2:1.6.9-2ubuntu1.6 [577 kB]
#4 2.153 Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 libxext6 amd64 2:1.3.4-0ubuntu1 [29.1 kB]
#4 2.154 Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 libxmuu1 amd64 2:1.1.3-0ubuntu1 [9728 B]
#4 2.155 Ign:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 openssh-client amd64 1:8.2p1-4ubuntu0.9
#4 2.165 Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 xauth amd64 1:1.1-0ubuntu1 [25.0 kB]
#4 2.165 Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libdw1 amd64 0.176-1.1ubuntu0.1 [226 kB]
#4 2.165 Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 libbabeltrace1 amd64 1.5.8-1build1 [156 kB]
#4 2.169 Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8 amd64 3.8.10-0ubuntu1~20.04.9 [1625 kB]
#4 2.202 Err:29 http://security.ubuntu.com/ubuntu focal-updates/main amd64 openssh-client amd64 1:8.2p1-4ubuntu0.9
#4 2.202   404  Not Found [IP: 185.125.190.36 80]
#4 2.237 Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 gdb amd64 9.2-0ubuntu1~20.04.1 [3222 kB]
#4 2.418 Get:35 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 gdbserver amd64 9.2-0ubuntu1~20.04.1 [222 kB]
#4 2.421 Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libcurl3-gnutls amd64 7.68.0-1ubuntu2.21 [232 kB]
#4 2.424 Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 liberror-perl all 0.17029-1 [26.5 kB]
#4 2.424 Get:38 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 git-man all 1:2.25.1-1ubuntu3.11 [887 kB]
#4 2.436 Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 git amd64 1:2.25.1-1ubuntu3.11 [4605 kB]
#4 2.507 Get:40 http://archive.ubuntu.com/ubuntu focal/universe amd64 joe amd64 4.6-1build2 [509 kB]
#4 2.511 Get:41 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 unzip amd64 6.0-25ubuntu1.1 [168 kB]
#4 2.512 Get:42 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libc6-dbg amd64 2.31-0ubuntu9.14 [19.9 MB]
#4 2.729 Fetched 47.1 MB in 1s (34.3 MB/s)
#4 2.729 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/o/openssh/openssh-client_8.2p1-4ubuntu0.9_amd64.deb  404  Not Found [IP: 185.125.190.36 80]
#4 2.729 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
------
process "/bin/sh -c apt install -y unzip git gdb joe" did not complete successfully: exit code: 100
Reaping losing child 0x55f9903bee20 PID 1325 
make: *** [docker/generated.mk:231991: .fox-freetype2_ftfuzzer-intermediate-runner] Error 1
Removing child 0x55f9903bee20 PID 1325 from chain.
ERROR:root:Executed command: "make --debug=j -j build-fox-freetype2_ftfuzzer" returned: 2.
ERROR:root:Failed to build benchmark: freetype2_ftfuzzer, fuzzer: fox. Extras: 
    traceback: Traceback (most recent call last):
  File "/work/src/experiment/build/builder.py", line 191, in build_fuzzer_benchmark
    buildlib.build_fuzzer_benchmark(fuzzer, benchmark)
  File "/work/src/experiment/build/local_build.py", line 116, in build_fuzzer_benchmark
    make([image_name])
  File "/work/src/experiment/build/local_build.py", line 35, in make
    return new_process.execute(command,
  File "/work/src/common/new_process.py", line 125, in execute
    raise subprocess.CalledProcessError(retcode, command)
subprocess.CalledProcessError: Command '['make', '--debug=j', '-j', 'build-fox-freetype2_ftfuzzer']' returned non-zero exit status 2.
prashast commented 10 months ago

Thanks for testing! I wasn't able to reproduce this issue on my end, i.e., the runner image was able to retrieve the packages and run the fuzzer. Regardless, based on the error message you posted, I've pushed a change to the runner image by calling apt-get update (you can pull in the change). Let me know if this helps in solving the issue on your end.

phi-go commented 10 months ago

Thank you for the quick update. Sometimes the apt cache in the docker images can get outdated, likely that I have a older base image by now. I'll give it another go tomorrow.

phi-go commented 10 months ago

It builds fine now, even with my older image, thanks again.

phi-go commented 10 months ago

Dear participants,

as already mentioned by Rahul in the Google Groups for SBFT'24, the tool reports can only be two pages long, instead of the four as mentioned in our timeline (including the references). As being over this two page limit will likely result in a rejection of your report by ACM, we wanted to also provide this information here to make sure it reaches you.

With Best Regards, Fuzzing Competition Team.