search

google / oss-fuzz

OSS-Fuzz - continuous fuzzing for open source software.
https://google.github.io/oss-fuzz
Apache License 2.0
10.44k stars 2.21k forks source link

build_fuzzers: allow or default to adding additional docker volume mounts #8311

Open howardjohn opened 2 years ago

howardjohn commented 2 years ago

Currently for go fuzzers (at least), we build everything with no caching. This means downloading all modules again, building from scratch, etc.

It would be nice to mount build/module caches, or at least allow this as an option via environment variable, to speed up builds when using python infra/helper.py build_image

jonathanmetzman commented 2 years ago

build_image cannot use any volumes because it involves building docker images which are meant to be hermetic. The build has no access to any volumes. build_fuzzers may be another story. Instead I would recommend structuring your dockerfile so that long-running steps that do not change very often are done before steps that you will frequently change. This will take advantage of docker's builtin caching (though admittedly, that will go away if you pull our base-images which get rebuilt daily.

jonathanmetzman commented 2 years ago

Does this suffice?

howardjohn commented 2 years ago

Not really. I did mean build_fuzzer not build_image, I must have gotten them mixed up.

Without a mounted cache no matter what we do to the building logic it will be slow if we have to rebuild, and a rebuild is require anytime a Go file changes which is going to be ~every time we iterate.

It's also not great to change our dockerfile since we want the logic inside our repo as a script, not in the oss-fuzz repo. This puts it in our control and allows us to change it with our code

jonathanmetzman commented 2 years ago

Now that I think of it, I think you can mount a local repo: https://google.github.io/oss-fuzz/advanced-topics/reproducing/#reproduce-using-local-source-checkout

howardjohn commented 2 years ago

That allows mapping the source to a specific location; the request would be to allow adding arbitrary mounts in https://github.com/google/oss-fuzz/blob/2fa71e3c7f87497dc31a9b236175c2173756fc99/infra/helper.py#L700

In this specific case (or probably all Go fuzzers, really) this would end up looking like:

      command += [
          '-v',
          '$HOME/go/pkg/mod:/root/go/pkg/mod',
          '-v',
          '$HOME/.cache/go-build:/root/.cache/go-build',
      ]

This could be an --extra-mounts flag, an env var, or even built-in logic for the two listed above