Have you ever run this against OWASP Benchmark?

[davewichers]

I'm the project lead for this project, which is at:

• https://owasp.org/www-project-benchmark/ - documentation
• https://github.com/OWASP/Benchmark - Code

This project is intended to measure the ability for application security testing (AST) tools to find vulnerabilities in applications. This happens to be a web app, although not pure REST. By that I mean that the responses aren't always simple XML or JSON. However, it does have an openapi spec generated for it which is available here: https://github.com/OWASP/Benchmark/blob/master/data/openapi.yaml

It would interesting to see if this tool can find any of the known vulnerabilities in this test suite. If it can, we can write a parser for the restler-fuzzer results file so we can automatically/repeatedly score the results of restler against OWASP Benchmark.

OWASP Benchmark supports automatic scorecard generation for numerous tools, as described here: https://owasp.org/www-project-benchmark/#div-scoring

If someone has the time to try it out, please let me know how it goes, and whether you think testing against Benchmark is useful for your project. You can contact me directly at dave.wichers@owasp.org.

[PatGod]

Hi Dave,

Thank you for your question/comment.

RESTler is not a web-app scanner so this specific benchmark does not look relevant to RESTler. Please see https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/FAQ.md#q-what-is-the-difference-between-restler-and-web-scanners

(Also, note that this openapi spec https://github.com/OWASP/Benchmark/blob/master/data/openapi.yaml does not describe a CRUD REST API, and therefore is again not relevant to RESTler.)

For examples of REST APIs and cloud/web services for which RESTler is applicable and relevant, please see the 4 research papers at the beginning on the main https://github.com/microsoft/restler-fuzzer. (Such services include open-source services like GitLab and various Microsoft services like Azure services.)

Thank you again for your feedback!

[marina-p]

Hi Dave,

Following up on what Patrice said - do you have any plans to add a benchmark targeting the OWASP API Security top 10 (https://owasp.org/www-project-api-security/)? Or, is there another project you know of that is already working on this?

Thanks,

Marina

[davewichers]

No plans currently. If someone wants to volunteer to help, that would be awesome! The project is (very slowly) working on 3 things currently. Benchmark 1.3 release (for Java still), and 2 new efforts just starting: new Benchmark for .NET, and a Benchmark for SCA tools (likely starting with Java as well). So we are starting to branch out to other Benchmark variants, but haven't had anyone volunteer to tackle an API Benchmark. It is a good idea though, and would be cool to have one.

[PatGod]

FYI, RESTler includes a simple example with a few seeded bugs: see https://github.com/microsoft/restler-fuzzer/tree/main/demo_server I would not call it a benchmark though. However, this example could easily be extended to more complex scenarios...

[mbiuki]

Taking a look at OWASP top 10 API Security, I believe is safe to declare almost all of those top 10 security vulnerabilities can be covered with RESTler's active checkers. See this paper: https://patricegodefroid.github.io/public_psfiles/icst2020.pdf.

For benchmarks, @davewichers, have you looked into FuzzBench?

• https://github.com/google/fuzzbench#sample-report
• The idea to pair up with fuzzbench rather than creating a new benchmark platform sounds compelling.