Fuzz testing

[kyz]

Google runs OSS-Fuzz project, where they spend a lot of CPU cycles fuzz-testing free software. They also offer bounties for getting projects to signing up (which makes me think of Ratbert's rat dance)

Laurent, would you consider requesting? Google may or may not accept.

Even if Google don't accept it, it would help us to have a fuzz test framework, i.e. develop a program like unadf, but without writing the extracted results to disk, and looks through all volumes on the device, and does traversal both with and without dircache if dircache is present. Then give it some disk images, and the fuzzer does the rest, mutating the input in random / directed ways to try and make the code go down different paths and crash. It watches the code execution flow and looks at what memory comparisons were made.

I'm not sure how we could fuzz-test writing files. I like to think that the only way that it would crash is by some sequence of operations, e.g. Open, Write, Seek, Read, Seek, Write..., or even have multiple files being written simultaneously. How would we turn a single input buffer into such a sequence, so the fuzzer can direct it? Perhaps some single-character string language, like "O" means open a new random-named file, "D" means create a directory, ... and other commands like enter a directory, go back to parent directory, switch to the next open file handle in a ring, write a block, read a block, seek forward a block, seek back a block, close a file handle, etc.

[lclevy]

Yes, why not. Should I request myself?

[kyz]

You should look into it.

I think anyone can follow the process, but the documentation seems to be written like it will always be an interested third party wanting to do the integration, and Google insists they ask the project maintainer to consent to being added, and then the interested third party will do all the things Google wants (e.g. put in the pull request and maintain what's on Google's side of the project).

I don't think there's anything stopping the project maintainer themselves doing the request, but I don't know what Google does in that case, I didn't read the documentation too carefully.

[t-w]

From: https://google.github.io/oss-fuzz/getting-started/accepting-new-projects/

Accepting New Projects

To be accepted to OSS-Fuzz, an open-source project must have a significant user base and/or be critical to the global IT infrastructure.

Do we know what ADFlib's user base is/might be (some estimate, projects that use it)?

In general, the idea of doing some fuzz testing is good, I am just not sure if at the stage the ADFlib is, it is not too early for that... Working on fuzz tests against an unstable API may be an overshot... But at some point - it should definitely be done!

I'd start from adding more standard things (static checks, more tests) and possibly clean-up the code and the API (so that there will be little more changes expected). Otherwise it will be duplicated work (fuzz tests most likely will have to be changed along with the API).

[lclevy]

I have read on internet that AWS S3 is based on adflib. Or maybe it is just rumors? 🤣 Of course adflib is not critical. It is retrocomputing or computer archeology...

[lclevy]

Your tests are good enough

[t-w]

Well, they only somewhat confirm that things work... For real security testing this is not enough, but the library needs to stabilize first, have tests that confirms that all the functionality works properly and so far that it only done in some part. There are still many things to do before doing more tests of robustness/security. If the things may change significantly, why invest a lot of time time in building such extensive tests (which will have to be changed as well)?

The idea to do such tests is good. But rather in some realistic way, ADFlib is not glibc or openssl... It does not mean that security is not important - ADFlib can be used to open files usually downloaded from the Internet. That always poses potential risk that should be possibly minimized.