Update LibAFL

[tokatoka]

and the rust compiler it uses

CC @andreafioraldi @addisoncrump

[addisoncrump]

So to fix that last one you can use a fuzzbench-specific profile like I do here:

• https://github.com/AFLplusplus/LibAFL/blob/main/libafl_libfuzzer/libafl_libfuzzer_runtime/Cargo.toml#L20-L24
• https://github.com/google/fuzzbench/blob/master/fuzzers/libafl_libfuzzer/builder.Dockerfile#L41

[tokatoka]

OK 👍

[addisoncrump]

#13 62.01 error[E0512]: cannot transmute between types of different sizes, or dependently-sized types
#13 62.01   --> /libafl/libafl/src/bolts/anymap.rs:59:5
#13 62.01    |
#13 62.01 59 |     assert_eq_size!(TypeId, u64);
#13 62.01    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#13 62.01    |
#13 62.01    = note: source type: `TypeId` (128 bits)
#13 62.01    = note: target type: `u64` (64 bits)
#13 62.01    = note: this error originates in the macro `assert_eq_size` (in Nightly builds, run with -Z macro-backtrace for more info)
#13 62.01 

We fixed this a while ago; old version of LibAFL being used?

[tokatoka]

I changed fuzzers/libafl. but for some reason the fuzzer being built is libafl_forkserver (Why?)

[addisoncrump]

Yeah that's weird. @jonathanmetzman @Alan32Liu there seems to be a CI bug here.

In the meantime, can we just update the commit for the other libafls?

[tokatoka]

Yes I deleted libafl_forkserver because it's just the same as aflrustrust and updated the commit

[tokatoka]

now good 😊

[DonggeLiu]

now good 😊

Not sure what has happened with the CI tests, but do they use the correct fuzzer now? I am happy to merge it if that is the case.

[tokatoka]

Not sure what has happened with the CI tests, but do they use the correct fuzzer now?

No I edited ['libafl'], but the CI runs all the tests for ['libafl', 'aflrustrust', 'libafl_forkserver']. CI looks fine because in the end I deleted 'libafl_forkserver' and patched 'aflfurst'

[addisoncrump]

now good 😊

Not sure what has happened with the CI tests, but do they use the correct fuzzer now? I am happy to merge it if that is the case.

The CI issue is likely still present, but @tokatoka deleted the older fuzzer in favour of the new.

[DonggeLiu]

I reckon that is because aflrustrust and libafl_forkserver depend on libafl.

This is an intended behaviour in FuzzBench, because it wants to make sure that changing the base fuzzer will not silently break other fuzzers that depend on it.

[tokatoka]

I see :ok_hand: You can merge this now.

[DonggeLiu]

I see 👌 You can merge this now.

Sure! But let's double-check with @andreafioraldi first :)

[DonggeLiu]

BTW, given this affects the master branch, would you think it makes sense to run an experiment to ensure the changes work as we expected before merging?

I guess you already know how to make a trivial modification to service/gcbrun_experiment.py and write your experiment request in this format : ) Thanks!

[tokatoka]

BTW, given this affects the master branch, would you think it makes sense to run an experiment to ensure the changes work as we expected before merging?

OK i'll push necessary changes now

[DonggeLiu]

OK i'll push necessary changes now

Thanks!

[tokatoka]

I added the old fuzzer using the old commit (slightly changed because it doesn't one of its dependency crate)

The command would be

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-25-libafl --fuzzers libafl libafl_27042023

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-25-libafl --fuzzers libafl libafl_27042023 aflrustrust

[DonggeLiu]

Added aflrustrust, as it is changed in this PR. Experiment 2023-09-25-libafl data and results will be available later at: The experiment data. The experiment report.

[DonggeLiu]

BTW, would you intend to completely remove libafl_forkserver in this PR? I noticed that its fuzzer.py was removed, but other files were not.

[tokatoka]

yes that's a mistake. i will remove everything

[tokatoka]

ah no that's not the case. I did deleted everything from libafl_forkserver

it seems it's just git that thought they were renamed to libafl_23042023/Dockerfile. Now that the experiment has started i'll delete the old fuzzer (later), and it should be fine

[tokatoka]

I think the is not run !?

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-26-libafl --fuzzers libafl libafl_27042023 aflrustrust

[DonggeLiu]

I think the is not run !?

Yeah, that's due to a gcloud trigger on GitHub, which I cannot fix. Basically, I should not comment before the experiment has successfully launched. I am pretty sure the new one works.

Experiment 2023-09-26-libafl data and results will be available later at: The experiment data. The experiment report.

[tokatoka]

Apparently libafl has gone worse within the past 5 month... we'll need to find out why

[tokatoka]

@Alan32Liu Indeed that was a bug in our llvm pass. we didn't notice it for 4 months you saved us

[DonggeLiu]

@Alan32Liu Indeed that was a bug in our llvm pass. we didn't notice it for 4 months you saved us

Very glad that we run an experiment before merging : )

[DonggeLiu]

Feel free to run more experiments here more often if that could save you time.

[tokatoka]

now you can merge this :pray:

[DonggeLiu]

now you can merge this 🙏

Would it make sense to run another experiment, just to triple-check?

[tokatoka]

okay :joy:

This is the command;

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-27-libafl --fuzzers libafl libafl_27042023 aflrustrust

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-27-libafl --fuzzers libafl libafl_27042023 aflrustrust

[tokatoka]

Now the experiment has run again. but the result has not improved.

and in the build log benchmark-libpcap_fuzz_both-fuzzer-libafl.txt

Step #2 - "libafl-libpcap_fuzz_both-builder-intermediate": 
Step #2 - "libafl-libpcap_fuzz_both-builder-intermediate": #7 [4/7] RUN git clone https://github.com/AFLplusplus/LibAFL /libafl
Step #2 - "libafl-libpcap_fuzz_both-builder-intermediate": #7 sha256:176f27286eece531de3dbf978cee7faecdf4771dd0fce583799967f009b4217a
Step #2 - "libafl-libpcap_fuzz_both-builder-intermediate": #7 pulling sha256:dcd3c5754ac77fd04a705522927db13aa3e5c571d655e7d0a2162db6db0eef18
Step #2 - "libafl-libpcap_fuzz_both-builder-intermediate": #7 pulling sha256:d4d55b1b8b30662f75a7938283a0c970af9b6b309b55508584a92b95a7ac929e
Step #2 - "libafl-libpcap_fuzz_both-builder-intermediate": #7 pulling sha256:d1ef8f825a26cbed27232333e2c285dba737bf5fa2cdec25c4db969b9a3ba462
Step #2 - "libafl-libpcap_fuzz_both-builder-intermediate": #7 pulling sha256:d1ef8f825a26cbed27232333e2c285dba737bf5fa2cdec25c4db969b9a3ba462 0.3s done
Step #2 - "libafl-libpcap_fuzz_both-builder-intermediate": #7 pulling sha256:dcd3c5754ac77fd04a705522927db13aa3e5c571d655e7d0a2162db6db0eef18 2.6s done
Step #2 - "libafl-libpcap_fuzz_both-builder-intermediate": #7 pulling sha256:d4d55b1b8b30662f75a7938283a0c970af9b6b309b55508584a92b95a7ac929e 5.0s done
Step #2 - "libafl-libpcap_fuzz_both-builder-intermediate": #7 CACHED
Step #2 - "libafl-libpcap_fuzz_both-builder-intermediate": 
Step #2 - "libafl-libpcap_fuzz_both-builder-intermediate": #8 [5/7] RUN cd /libafl && git checkout 324db072a2f0a25f4e891d8a008a2feff240fb79 || true
Step #2 - "libafl-libpcap_fuzz_both-builder-intermediate": #8 sha256:c5f24e432e51a041c7f098db02b62686e3b36711d2f93ba8a653991fa977f63d
Step #2 - "libafl-libpcap_fuzz_both-builder-intermediate": #8 0.294 fatal: reference is not a tree: 324db072a2f0a25f4e891d8a008a2feff240fb79
Step #2 - "libafl-libpcap_fuzz_both-builder-intermediate": #8 DONE 9.0s
Step #2 - "libafl-libpcap_fuzz_both-builder-intermediate": 

324db072a2f0a25f4e891d8a008a2feff240fb79 is the commit the includes the commit. but it says it can't checkout to the latest libafl because it is using "CACHED" old clone of libafl. and this guy doesn't contain my latest commit Is there a way to tell the docker to stop using cache?

[DonggeLiu]

324db072a2f0a25f4e891d8a008a2feff240fb79 is the commit the includes the commit. but it says it can't checkout to the latest libafl because it is using "CACHED" old clone of libafl. and this guy doesn't contain my latest commit

How about adding a git pull before git checkout?

[tokatoka]

ok

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-28-libafl --fuzzers libafl libafl_27042023 aflrustrust

[DonggeLiu]

Experiment 2023-09-28-libafl data and results will be available later at: The experiment data. The experiment report.

[jonathanmetzman]

ok

I think sometimes you need to modify the file to skip this caching. I'm not sure git pull will help here.

[tokatoka]

this pr is ready (for the 3rd time)

[DonggeLiu]

Experiment 2023-09-28-libafl data and results will be available later at: The experiment data. The experiment report.

This report shows [libafl_27042023](https://github.com/google/fuzzbench/blob/master/fuzzers/libafl_27042023) performs better than libafl. Is that because git pull does not work? Shall we re-run the exp after you update the file (which should avoid using cache)?

[tokatoka]

Yes indeed 2704 was better. especially with libjpeg We identified the cause and fixed some bugs in the latest commit of this PR. git pull worked for us.

Shall we re-run the exp after you update the file (which should avoid using cache)?

Yes please :pray:

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-05-libafl --fuzzers libafl libafl_27042023 aflrustrust

[DonggeLiu]

Experiment 2023-10-05-libafl data and results will be available later at: The experiment data. The experiment report.

[tokatoka]

this one looks good now we can merge it :pray:

[DonggeLiu]

this one looks good now we can merge it 🙏

Sure, could you please fix the format error in CI? Here is the log:

source .venv/bin/activate && python3 presubmit.py
--- /home/runner/work/fuzzbench/fuzzbench/fuzzers/libafl/fuzzer.py  (original)
+++ /home/runner/work/fuzzbench/fuzzbench/fuzzers/libafl/fuzzer.py  (reformatted)
@@ -66,6 +66,6 @@
         command += (['-x', dictionary_path])
     command += (['-o', output_corpus, '-i', input_corpus])
     fuzzer_env = os.environ.copy()
-    fuzzer_env['LD_PRELOAD']='/usr/lib/x[8](https://github.com/google/fuzzbench/actions/runs/6379929561/job/17313360212?pr=1902#step:8:9)6_64-linux-gnu/libjemalloc.so.2'
+    fuzzer_env['LD_PRELOAD'] = '/usr/lib/x86_64-linux-gnu/libjemalloc.so.2'
     print(command)
     subprocess.check_call(command, cwd=os.environ['OUT'], env=fuzzer_env)

Code is not formatted correctly, please run 'make format'
ERROR: format failed, see errors above.
Failed checks: format
Failed.
make: *** [Makefile:60: presubmit] Error 1
Error: Process completed with exit code 2.

[tokatoka]

Done 🙏

[DonggeLiu]

/gcbrun skip

[DonggeLiu]

Done 🙏

Thanks!