search

AFLplusplus / LibAFL

Advanced Fuzzing Library - Slot your Fuzzer together in Rust! Scales across cores and machines. For Windows, Android, MacOS, Linux, no_std, ...
Other
1.9k stars 292 forks source link

Improvements to testing framework #1799

Open s1341 opened 5 months ago

s1341 commented 5 months ago

Recently @mkravchik merged a PR aimed at adding e2e-testing capabilities to the library in order to allow for easy testing of e.g. frida ASAN.

After using his solution a bit, and making a few improvements (see changes in PR #1607), here are some ideas to improve upon it even more.

  1. Currently all the ASAN tests run as one 'cargo test' testcase. This has the following cons:
    • If a single test fails, the remainder of the tests do not run
    • There is no nice 'cargo test' UI showing pass/fails for each test.
    • It's impossible to run a single test from the list
  2. Currently to write a test, one needs to add a function to test_harness.cc, and then add the test name and expectation to the list in lib.rs. This is a bit clunky. Ideally we'd be able to both specify the C/C++ function to be tested and the expectations, easily from within the lib.rs and have machinery to produce a shared-object/dll/dylib for the test automatically.
  3. Currently we only support simple expectations. Specifically: was there an ASAN error, and what type was it. For true unit testing, we would want to be able to specify complex expectations, includng things like checking that the backtrace is correct, the registers captured are correct etc.
  4. Currently all test-cases are in a shared-object/dll. This is not strictly necessary. It might be signifcantly easier to use a static library and link the test-cases into the library.
  5. Currently we use the fuzz_one method to execute our test case. While this is sufficient for the ASAN tests, it will not be enough for e.g. cmplog testing. We should introduce a fuzz_until_solution method.
s1341 commented 5 months ago

Regarding 4, it may indeed be necessary to use a distinct shared-object for frida testing.

domenukk commented 5 months ago

I would say using cargo make test (or similar) is the correct solution for most of the issues described here. We can keep rust unittests simple and specify more complex integration tests in makefile.toml, why not?

s1341 commented 5 months ago

Having the tests tightly coupled to the code they are testing seems like a better stratergy to actually encourage tests being written... The simpler it is to write them, the better.

domenukk commented 5 months ago

I agree, however I'm not convinced blowing up cargo test to something super complex will make it simpler to write tests in the end