Write more extensive fuzzer tests

[FirefoxMetzger]

ImageIO has a fuzzer (for fuzz-testing) which OSS-Fuzz is running to discover and report bugs and crashes. A caveat for this is that it uses atheris for fuzzing, which currently only works on Linux. Because of that local reproducibility of found bugs is currently limited to Linux systems. In either case, I think it would be worthwhile to make better use of this offer.

The purpose of this issue is to act as a reminder that we still have to write more extensive tests for the fuzzer.

[DaveLak]

Hey!

Is this still something you would be interested in seeing added to the project? If so, I am more than happy to contribute to Imageio's fuzzing setup to whatever degree you'd find helpful.

I was thinking:

Fuzz Tests

• With a few minor tweaks, I think the existing test could be improved rather significantly (for example, replacing the disk I/O with an in-memory alternative.)
• Many of the APIs in this project look like they'd make very interesting fuzz targets.

Documentation

To address the pain points of running fuzzers locally that you mentioned in the caveat in this issue's body:

• Adding a fuzzing specific README with instructions for running/reproducing locally would be super helpful.
• Adding a simple Dockerfile that comes with everything necessary for running the fuzzers could be a nice addition too.

You can find a concrete example of what I'm thinking in the GitPython project, where I recently contributed what described above: https://github.com/gitpython-developers/GitPython/tree/4e212c6e2cf7049835ef64809bec2ce30601e537/fuzzing

[FirefoxMetzger]

Yes, definitely still interested in having better fuzzing here :) If you want to work on a PR I'd be happy to review.

[DaveLak]

Great! I'll have something for review soon 🙂

[DaveLak]

Hey, sorry for the delay here. I was able to get the fuzzer working quite well locally; so well in fact that I wanted to reach out and ask how sensitive the maintainers would consider the type of bugs I've identified before posting anything publicly.

About My Findings

The bugs found so far all appear to be uncaught exceptions in some of the plugins housed in this repo.

They're mostly AttributeErrors, TypeErrors, and UnicadeDecodeErrors which I find are fairly typical for this sort of testing. There are also a few struct.errors, and at least one OverflowError (raised by the Python interpreter, not an overflow in a native C extension which this library does not have.) The latter two are less typical in my limited experience, but still do not seem much different than the first three mentioned as far as threat vectors go.

I'd imagine some of these could represent a DoS vector under some circumstances, while others simply point to a place where an explicitly raised exception could add an improved error message.

I've searched the issues on this repo and see many similar issues have been reported publicly without any security concerns, but out of an abundance of caution, I wanted to reach out and ask how the maintainers of this project would like to proceed before I disclose anything.

My Question for the Maintainers

How would you like me to proceeded regarding the exceptions I've identified? Some possible options include:

1. I can share access to a private fork of this repo with whomever the maintainers would like, and include my fuzzer code and an issue listing the exceptions & tracebacks for review/evaluation before deciding on next steps.
2. I can share the relevant details privately with a Tidelift security contact for review/evaluation before deciding on next steps.
3. I can open a public issue on this repo with the details, and a PR with the fuzzer improvements that identified them (note: there are several, so a single issue might then need to be broken down per plugin at least.)
4. Anything else the maintainers would prefer; I'm happy to do whatever is easiest for you all.

Thanks! I'm looking forward to your thoughts!