FI dependencies should probably be included in the base-builder:introspector image

[evverx]

The OSS-Fuzz toolchain doesn't support incremental builds in general so fuzz targets have to be rebuilt from scratch every time they change. Usually it leads to just recompiling projects and their fuzz targets but when FI is used its dependencies are built and installed every time https://github.com/google/oss-fuzz/blob/5260d875b2d55b128e52ada2ecec983ab0ce2667/infra/base-images/base-builder/compile#L211-L213

  apt-get install -y libjpeg-dev zlib1g-dev
  pip3 install --upgrade setuptools
  pip3 install cxxfilt pyyaml beautifulsoup4 lxml soupsieve matplotlib

and it takes too long. For example it's possible to build dbus-broker with three different sanitizers and also build coverage reports based on public OSS-Fuzz corpora while numpy is installed. I think it would be great if those dependencies were included in the image. Apart from saving a lot of time and making it easier to use FI somewhat interactively it would make it possible to just download the image and start experimenting with FI without having to build the custom clang or pulling the OSS-Fuzz toolchain: https://github.com/ossf/fuzz-introspector/pull/460#issuecomment-1214359418

[DavidKorczynski]

Not a complete solution yet, but until then:

I started to upload Docker images with latest fuzz introspector set up to https://hub.docker.com/u/davkor

The following diff in the images eases the matplotlib download/compilation burden: https://github.com/ossf/fuzz-introspector/blob/1338716b8a92003aa824ecfafcde45173482097f/oss_fuzz_integration/oss-fuzz-patches.diff#L34-L35

This script can be used to replace your OSS-Fuzz images:

#!/bin/bash -eu
# Copyright 2022 Fuzz Introspector Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
################################################################################

if [ -d "oss-fuzz" ]
then
  echo "OSS-Fuzz directory exists. Reusing existing one"
else
  echo "Cloning oss-fuzz"
  git clone https://github.com/google/oss-fuzz
  echo "Done"
fi

echo "Pulling dev images"
docker pull davkor/fuzz-introspector-base-image:latest
docker pull davkor/fuzz-introspector-base-clang:latest
docker pull davkor/fuzz-introspector-base-builder:latest
docker pull davkor/fuzz-introspector-base-builder-python:latest
docker pull davkor/fuzz-introspector-base-runner:latest

echo "Tagging dev images to OSS-Fuzz images"
docker tag davkor/fuzz-introspector-base-image:latest          gcr.io/oss-fuzz-base/base-image:latest
docker tag davkor/fuzz-introspector-base-clang:latest          gcr.io/oss-fuzz-base/base-clang:latest
docker tag davkor/fuzz-introspector-base-builder:latest        gcr.io/oss-fuzz-base/base-builder:latest
docker tag davkor/fuzz-introspector-base-builder-python:latest gcr.io/oss-fuzz-base/base-builder-python:latest
docker tag davkor/fuzz-introspector-base-runner:latest         gcr.io/oss-fuzz-base/base-runner:latest

echo "Cleaning up davkor tags as we only need them gcr tagged"
docker rmi davkor/fuzz-introspector-base-image
docker rmi davkor/fuzz-introspector-base-clang
docker rmi davkor/fuzz-introspector-base-builder
docker rmi davkor/fuzz-introspector-base-builder-python
docker rmi davkor/fuzz-introspector-base-runner

[evverx]

@DavidKorczynski thanks! I think it should speed up the process a bit but as far as I can see I can't use FI in davkor/fuzz-introspector-base-builder out of the box. I can compile code using clang -fsanitize=fuzzer -flto -fuse-ld=gold there but once I try to analyze the *.data and *.data.yaml files I get

Traceback (most recent call last):
  File "/fuzz-introspector/src/main.py", line 21, in <module>
    from fuzz_introspector import commands, constants
  File "/fuzz-introspector/src/fuzz_introspector/commands.py", line 17, in <module>
    import yaml
ModuleNotFoundError: No module named 'yaml'

More generally what I'm trying to say is that I think there should be an easy way to get both the custom clang and main.py that wouldn't require building anything from scratch or bringing the whole OSS-Fuzz workflow.

[evverx]

Since images like that would make it easier to somewhat decouple FI from OSS-Fuzz I think it would also help with https://github.com/ossf/fuzz-introspector/issues/24. The documentation could just point to those images and mention that for example https://github.com/ossf/fuzz-introspector/tree/main/tests/simple-example-0 can be built with a few commands generating coverage reports, building the target with -flto -fuse-ld=gold and running main.py to turn that into a FI report. Right now https://github.com/ossf/fuzz-introspector/blob/main/doc/LocalBuild.md says that to build that fuzz target it's necessary to build the custom clang and I'd say it's a steep learning curve :-)

[DavidKorczynski]

Closing this by way of https://github.com/google/oss-fuzz/pull/8912