Categorizations, rankings & data veracity.

[gcj-web]

First of all, thank you for all the work and contributions your work group is making in securing server-side Web APIs.

I am currently conducting a systematic study of hundreds of Web API security issues. Part of this work consist in categorizing security bugs using the 2019 OWASP API Security Top 10. The 2023 categorization was not released at the time. I would have a few questions with respect to the OWASP API Security Top 10 categorizations:

1. For both the 2019 and 2023 Top 10, would you be able to confirm that security issues are supposed to be ranked according to their prevalence?
2. As mentioned in #77 by @PauloASilva, your work group used publicly available data to build the categorization’ rankings. I also used publicly available data, however, I end up with a significantly different ranking. Moreover, I also have some API-centric security issues that are more prevalent that some included in the Top 10. I would like to cross-check/look for threats to validity in both my data set and yours. I could not find any information wrt your data set online, is the data publicly disclosed somewhere?
3. Is the 2023 Top 10 categorization fixed? i.e., is it worth to start categorizing Web API security bugs according to the 2023 Top 10 as of now, or is it subject to change in the following weeks/months?

Thanks,

[PauloASilva]

Hi @t-cmu, Thanks for your kind words.

1. For both the 2019 and 2023 Top 10, would you be able to confirm that security issues are supposed to be ranked according to their prevalence?

No. The OWASP API Security Top 10 is an awareness document: you may find there risks that require/deserve awareness which would not pop-up from any dataset we could use. It also works the other way around: issues that will immediately pop-up from a dataset that were consciously left out from the API Security Top 10.

Although we have used the OWASP Risk Rating Methodology to perform risk analysis, the final order may have been changed according to risk and required awareness.

2. As mentioned in [#77](https://github.com/OWASP/API-Security/issues/77) by @PauloASilva, your work group used publicly available data to build the categorization’ rankings. I also used publicly available data, however, I end up with a significantly different ranking. Moreover, I also have some API-centric security issues that are more prevalent that some included in the Top 10. I would like to cross-check/look for threats to validity in both my data set and yours. I could not find any information wrt your data set online, is the data publicly disclosed somewhere?

We didn't publish it anywhere. If you're using publicly available data you're likely to have pulled data from the same sources (e.g. h1, bugcrowd, apisecurity.io, etc.). Note that for the 2023 edition we've considered only issue reported between 2019 and 2021 matching some basic criteria to filter API-related ones (e.g. includes keyword "API"). Then the results were manually reviewed.

When you say "hundreds of Web API security issues" I am not exactly sure about the magnitude of what you've got. Following this approach we've collected 371 issues but only 338 were considered valid and properly mapped to CWEs. This is clearly not enough considering data bias, nevertheless it was a good exercise to drive the discussion.

3. Is the 2023 Top 10 categorization fixed? i.e., is it worth to start categorizing Web API security bugs according to the 2023 Top 10 as of now, or is it subject to change in the following weeks/months?

No, it is still a release candidate. Hopefully we should be able to release the final version soon.

Again, keep in mind that the OWASP API Security Top 10 is an awareness document. It's about risks and not vulnerabilities or bugs. If you're looking to categorize the web API security bugs you're better mapping the issues to CWEs.

Cheers, Paulo A. Silva

[gcj-web]

Thank you for your reply. I am also using the CWE classification. I was looking forward to using the OWASP API Top 10 as well given how widespread its use is in industry. Given that the OWASP API Security Top 10 is an awareness document, would you then consider categorizing web API security issues using this Top 10 as a bad practice?

I am still a bit thrown off by the rankings. You mentioned the OWASP Risk Rating Methodology was used to perform risk analysis. However, in the 2019 Top 10, I have noticed some inconsistencies wrt the OWASP Risk Rating Methodology. For example, the excessive data exposure is ranked 3rd with {exploitability:3, prevalence:2, detectability:2, technical:2} whereas the security misconfiguration is ranked 7th with {exploitability:3, prevalence:3, detectability:3, technical:2}. I presume this is what you meant by "the final order may have been changed according to risk and required awareness"?

I'm in the same lines in terms order of magnitude: ~300 case studies accounting for a total of ~400 security bugs; I also manually reviewed each bug to ensure that every single one of them purely is web API-centric.

Thanks,

[PauloASilva]

Thank you for your reply. I am also using the CWE classification. I was looking forward to using the OWASP API Top 10 as well given how widespread its use is in industry. Given that the OWASP API Security Top 10 is an awareness document, would you then consider categorizing web API security issues using this Top 10 as a bad practice?

I can't say that "categorizing web API security issues using this Top 10 as a bad practice", it depends on your goal.

The way I see/do it. If you're:

1. performing a security assessment (pentest), for an API finding you should first provide the CWE (very specific) and then provide relevant references: the OWASP API Security Top 10 category the finding fits in will provide a more comprehensive view of the risk, but you should also provide relevant OWASP cheat sheets, OWASP testing guide references (if available), etc...
2. talking about API security to an audience (e.g. CISOs, DevSecOps, Developers, enthusiasts, ...) then presenting the API Security Top 10 is a very good option. I believe doing this will help you to tackle time and attention span constraints, maximizing the API security awareness.

I am still a bit thrown off by the rankings. You mentioned the OWASP Risk Rating Methodology was used to perform risk analysis. However, in the 2019 Top 10, I have noticed some inconsistencies wrt the OWASP Risk Rating Methodology. For example, the excessive data exposure is ranked 3rd with {exploitability:3, prevalence:2, detectability:2, technical:2} whereas the security misconfiguration is ranked 7th with {exploitability:3, prevalence:3, detectability:3, technical:2}. I presume this is what you meant by "the final order may have been changed according to risk and required awareness"?

Yes, you're right, that is exactly what I meant.

Cheers, Paulo A. Silva

[gcj-web]

I see. Thank you for your reply.