campaign automation

[il-steffen]

Draft PR for comments...let me know what you think :-)

Idea/purpose: More modular campaign running + debug approach. Instead of a single big runner, the new init_harness.py creates a campaign folder of kernel/fuzzer configs. We can run fuzz.sh there to do everything manual: build, fuzz, trace, smatch. The new campaign runner only has to chain all the steps together based on number of CPUs. But knowledge/config is encoded in low-level scripts, for example fuzz.sh build applies correct KCFLAGS to kernel build and prep_harness.py should be your basis for testing or improving a particular harness configuration.

Summary of changes

• lots of fixes to helper scripts to exit on errors, fix paths, print to stderr...
• codify the kernel build into new fuzz.sh build
• encapsulate initrd/htools/smatch audit steps as make prepare
• turn bkc/audit/Makefile and smatch/scripts/test_kernel.sh into single more flexible smatch_audit.sh
• extract the harness config part from run_experiments.py into init_harness.py
• new pipeline.py to automate workflow of setup, fuzzing, and processing results for a set of harnesses

Misc Issues

• [x] new fuzz.sh pipeline and fuzz.sh batch options implement a mini pipeline but probably can be removed?
• [x] the new pipeline.py runner works but is not really a huge improvement. Our problem seems simple but all the python workflow/template things I tested turned out to be quite annoying. And we don't want to implement everything in the pipeline since that would tie us to some framework and makes it hard to debug/test individual steps...
• [ ] it turns out that eu-addr2line sometimes returns non-0 exit code....I just muted it, but maybe not all the output is there..
• [ ] since smatcher is doing the smatch line matching, and can also consume input from the ghidra path, can we remove that smatch match feature from fast_matcher and rename the tool to something more obvious, like edges2lines?
• [ ] global resources created by make prepare should have separate location, like $ASSET_ROOT=$BKC_ROOT/assets/
• [ ] smatch audit should be done for current selected config and kernel tree. It would be good to have a check for this at least, or even better output directly to ASSET_ROOT/<version/config-hash>/. OTOH smatch audit always takes a couple minutes and is something we may often want to skip....e.g. pipeline.py --skip-smatch?
• [ ] all the fuzz.sh jobs can be turned into single parametrized task abstraction?
• [ ] not using much of parsl in the end, and the cloud/cluster abstractions are probably also half broken. perhaps we can switch this pipeline to multiprocess.pool() now?
• [ ] look at campaign stats and tune harness configuration, document expected results
• [x] hide some of the less relevant options in pipeline.py, maybe remove the multiple target/campaign folders feature

Blocking Issues

• [x] current pipeline keeps all the kernel builds...makes campaigns too big...
• [x] ghidra not auto-installed yet
• [x] should switch to or at least offer the smatch-matcher by Sebastian
• [x] add my attempts at auto-triage and summarize scripts (campaign_tools branch)
• [x] concurrent launch of kafl_fuzz.py is so fast that the instances don't detect each other and hog to same cpu sets
• [x] smatch match is broken because out-of-free build has different DWARF paths...
• [x] add a diagram + readme to document the new approach
• [x] make preprare broken for new env.sh?
• [x] add missing deps/build steps to ansible make deploy
  • [x] add step to create final smatcher report similar to stats + summary
  • [x] add busybox and elfutils dependencies. maybe as a new bkc role that also builds the tools there
  • [x] add parsl dependency...I think this (51f9ae9) is the wrong file+venv, better use bkc/kafl/requirements.txt
  • [x] build fast_matcher and smatcher as part of ansible/deploy, also pull in the rust deps