Add Mayhem to Qualified Fuzz Source

[dbrumley]

Is your feature request related to a problem? Please describe. Currently you only describe oss-fuzz and onefuzz. Please add Mayhem from forallsecure.com, which has a free tier at mayhem.forallsecure.com. Currently there are over 1000 repos (more than google oss-fuzz or onefuzz) using Mayhem.

Describe the solution you'd like Add having a Mayhemfile to qualify for fuzzing

Describe alternatives you've considered N/A

Additional context github.com/mayhemheroes currently contains a fork of many oss projects that are now fuzzing. Some are running in Mayhem and oss-fuzz, but about half are only Mayhem.

OSS Projects Integrated and running fuzzing on every push with mayhem:

Total fuzzing users:

[azeemshaikh38]

Thanks for the report @dbrumley. Could you expand on how Scorecard can check if a project uses Mayhem?

[dbrumley]

The presence of a Mayhemfile for our code fuzzing solution.

We have two fuzzing solutions: one for code, and one for APIs. Alternatively one could check for the presence of a github action: https://github.com/ForAllSecure/mcode-action and our API fuzzing at https://github.com/ForAllSecure/mapi-action

Happy to look into alternatives as well.

[azeemshaikh38]

Checking for code fuzzing solution would be within the scope of Scorecard but probably not API fuzzing. @ossf/scorecard-maintainers fyi.

If the other maintainers agree to adding this, @dbrumley please send a PR to address this issue and one of us can review it and get it merged.

[laurentsimon]

@azeemshaikh38 I think API fuzzing works too. A great deal of projects are "dependency" projects that provide APIs. API fuzzing is actually more granular than top-level "main program" fuzzing, so we should encourage people to do it. It's basically like unit tests, but fuzzing.

@dbrumley Can you clarify the meaning of "code" vs "API" fuzzing? Do you mean "top-level program" and "API-level" fuzzing?

+1 to add support.

[azeemshaikh38]

IIUC, https://github.com/ForAllSecure/mapi-action is targeted towards something like REST APIs.

[dbrumley]

Mayhem for API takes an API spec, and then tries to get coverage of the spec. Coverage means that we can positively interact with the API (2xx) to make sure we're not just sending junk. Coverage is measured. In addition, I believe it checks the most number of CVEs in the OWASP API Top 10 (more than zap for sure).

As an example, m. api has found bugs in the K8 API that ended up being fixed.

Mayhem for code is a program fuzzing tool, and takes in a docker container and fuzzes an input source (file, network socket, domain socket, stdin, etc). It keeps track of (binary) code coverage. Inside it's using symbolic execution and coverage-guided fuzzing.

The difference is code fuzzing tools (Mayhem, AFL, LibFuzzer) tend to do poorly with web APIs. Mayhem for API addresses this by specifically targetting openapi specs.

Does that help? I'll put in a PR.

[laurentsimon]

SGTM.

[github-actions[bot]]

Stale issue message - this issue will be closed in 7 days

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity.

[github-actions[bot]]

This issue has been marked stale because it has been open for 60 days with no activity.