Risk-based instead of testability-based levels

[Sjord]

In Scope for 5.0 and beyond · Issue #1127 · OWASP/ASVS, @jmanico says:

I think the levels should be RISK-BASED and absolutely not TESTABILITY BASED

So, let's drop the requirement that conformance to L1 requirements can be determined by a hands-on test.

ASVS/0x03-Using-ASVS.md at master · OWASP/ASVS · GitHub says:

ASVS Level 1 is for low assurance levels, and is completely penetration testable.

ASVS/0x04-Assessment_and_Certification.md at master · OWASP/ASVS · GitHub says:

In version 4.0, we decided to make L1 completely penetration testable without access to source code, documentation, or developers.

[jmanico]

I agree with this 100%. CC @tghosth and @elarlang

[elarlang]

I don't like this current "must be blackbox pentestable" too much, but at least is easy and clear rule.

Whatever is the way we define levels, we really need to define them. It should be clear, why some requirements is level 1, another is level 2 and some is level 3. There will be gray areas for sure, but we need some guidelines.

[danielcuthbert]

So the logic behind making it pentestable is, well just that. We wanted to make it so that automated tools could adopt L1 and have scanning profiles etc.

the problem with making it just risk-based is that often risk-based is fluffy and handwavy and this standard is not 100% a GRC-based one. There has to be a balance here and this was at least our attempt at doing this. So how would you see this happening with the above in mind?

[cmlh]

ISO/IEC 27034-7:2018 specifies both an "Expected Level of Trust" and "Actual Level of Trust" which aligns to their various risk management standards.

[tghosth]

So @Sjord you are saying move away from testability?

[Sjord]

For me, the levels relate to the security requirements of your application. If you are building a nuclear missile launch system, you would comply to L3. If you are building a tic-tac-toe website, you would comply to L1. However, even a webapp with such low security requirements should do logging, have expiring sessions, and have other security measures that are difficult or impossible to pentest.

The attribute whether something is pentestable seems to be independent of security requirement levels.

[tghosth]

Ok, I have recently had conversations with two separate organizations using ASVS for verification. In both cases, it would have been easier for them to start with L1 to be assessed by pen testing and then find a way of verifying L2+ as a stretch goal or a 2nd phase.

In both cases, they were very clear that just being able to assess to L1, even as a stepping stone was not sufficient and that even their initial assessment methodology had to also L2, despite the added difficulty.

As such, I think I can agree that L1 being pen testable does not really help if no one thinks it is enough.

The 5.0 roadmap/plan reinforces this and I think it is clear that this will be the direction.

Any further comments from anyone?

[cmlh]

In both cases, they were very clear that just being able to assess to L1, even as a stepping stone was not sufficient and that even their initial assessment methodology had to also L2, despite the added difficulty.

What where their additional test cases considered under L2 @tghosth?

[elarlang]

In both cases, they were very clear that just being able to assess to L1, even as a stepping stone was not sufficient and that even their initial assessment methodology had to also L2, despite the added difficulty.

Can we just rephrase or translate it: "collect low-hanging fruits fast and early" to get at least some feedback. For me it seems that in this situation there is no actual feedback how we (should) group requirements to levels.

[tghosth]

What where their additional test cases considered under L2

Some L2 requirements are pen testable and the others they are having to think about what questions to ask @cmlh

Can we just rephrase or translate it: "collect low-hanging fruits fast and early" to get at least some feedback. For me it seems that in this situation there is no actual feedback how we (should) group requirements to levels.

Basically, we are going to have to assess a risk/prioritization level at some point but probably best to leave that for a later date :)

[tghosth]

I would be tempted to add the ASVS Draft label and leave this issue for now. It is something we will only really be able to do at the end.

[elarlang]

I would be tempted to add the ASVS Draft label and leave this issue for now. It is something we will only really be able to do at the end.

We need to finetune the text at the end, but we need to set correct levels based on some arguments for every requirement and it is not that rare when we have issue opened because current description and levels are not matching.

[appills]

For me, the levels relate to the security requirements of your application. If you are building a nuclear missile launch system, you would comply to L3. If you are building a tic-tac-toe website, you would comply to L1.

I wouldn't think a tic-tac-toe app would require authentication either (at least in a couch co-op setting) and this is what I would consider Level 1 - bare-bones functionality. As soon as sensitive data is introduced (e.g. username/password authentication), the app becomes Level 2.

At the very least, I'd advocate for introducing a Level 0 so that Level 1 can be required more often.

Example: Level 0: handles no data sensitive to itself, implements bare-bones security by accident (e.g. logging) Level 1: handles data sensitive to itself (it has implemented some form of authentication) Level 2: most apps Level 3: apps under regulations

[elarlang]

With risk - I think we need to start from question - why and for whom we define them / Who and how uses them?

My experience here is level 2 and level 3 from pentesting perspective and it matches the descriptions like @appills described them, so I don't know in practice (again from pen-testing perspective) do anyone order pen-test with level 0 or level 1? Basically "scan me"?

I can understand from development point of view those are needed, but not that clear as well. Authentication != senstive data - an application may have username-password authentication, but if it do not collect any PII, then I think it's the same non-sensitive application like application with non-auth.

So, if someone knows someone who uses ASVS risk levels somehow measured and validated way, we here are really interested to hear about the experience, please send them here to share their experience.

[alamers]

I have worked with a couple of clients that were working up to a security certification. There, we first asserted that all (custom developed) software should be at least level 1 (based on advs 4), and then based on a risk assessment increase levels. We did crystal box testing focused on level 1. Since most systems are connected in some way, we needed to establish a base line overall first.

So I can see some value in level 0 and 1, just to get started with a landscape. Especially if a level 0 contains stuff that there can’t be any reason not to implement it.

[elarlang]

Thank you for sharing @alamers

If we look back to previous versions, maybe the simplicity like in v3.0.1 works best. The main problem with risk definition for v4.0.3 is "completely penetration testable" end especially this part from the opened issue:

In version 4.0, we decided to make L1 completely penetration testable without access to source code, documentation, or developers.

Also, I think we need to keep max 3 levels, otherwise we go crazy with setting and discussing correct level for each requirement. There can be addition "level 0 - do whatever you want" which do not require any definition for the requirements.

Previous definitions: ASVS v4.0.3: The Application Security Verification Standard defines three security verification levels, with each level increasing in depth.

• ASVS Level 1 is for low assurance levels, and is completely penetration testable
• ASVS Level 2 is for applications that contain sensitive data, which requires protection and is the recommended level for most apps
• ASVS Level 3 is for the most critical applications - applications that perform high value transactions, contain sensitive medical data, or any application that requires the highest level of trust.

ASVS v3.0.1:

• ASVS Level 1 is meant for all software.
• ASVS Level 2 is for applications that contain sensitive data, which requires protection.
• ASVS Level 3 is for the most critical applications - applications that perform high value transactions, contain sensitive medical data, or any application that requires the highest level of trust.

ASVS v2.1:

• Level 0 (or Cursory) is an optional certification, indicating that the application has passed some type of verification.
• Level 1 Opportunistic - An application achieves Level 1 (or Opportunistic) certification if it adequately defends against application security vulnerabilities that are easy to discover.
• Level 2 Standard - An application achieves Level 2 (or Standard) verification if it also adequately defends against prevalent application security vulnerabilities whose existence poses moderate-to-serious risk.
• Level 3 Advanced - An application achieves Level 3 (or Advanced) certification if it also adequately defends against all advanced application security vulnerabilities, and also demonstrates principles of good security design.

[tghosth]

Ok. I thought about this a little this morning.

I agree with ignoring the pen testable aspect as I discussed here: https://github.com/OWASP/ASVS/issues/1495#issuecomment-1478533446

On the other hand, I am not sure I like the "risk based" concept we currently use. We say that L3 requirements are for the most highly sensitive applications but really what we usually mean is that L3 requirements are too complicated or widely scoped for most apps to implement.

I also don't think that the levels should describe the applications which will be certified but rather should be focused more specifically on the types of requirements within those levels.

As such, I thought about the following:

• Level 0 - Early value - These are basic requirements that any application should have and should bring some level of security without significant effort.
• Level 1 - Minimum Security - These are the minimum security requirements that an application should have in place to prevent common attacks.
• Level 2 - Standard security - This adds additional requirements to provide additional layers of defence and protect against a wider range of standard attacks.
• Level 3 - Advanced security - This adds requirements to provide defence against a wider scope of attacks or defences which are more complex to implement.

We would then leave it to app consumers or the market in general to decide which level they think a particular app should comply with.

What do people think?

[jmanico]

Josh,

I like your proposal, it’s well thought out.

However, I’d like to consider the direction MASVS is going which is only two levels. One for basics and another for solid security.

And we can’t expect everyone to do every requirement as ASVS is quickly turning into a catalog of all requirements to consider.

My 2 cents.

[tghosth]

Josh, I like your proposal, it’s well thought out. However, I’d like to consider the direction MASVS is going which is only two levels. One for basics and another for solid security. And we can’t expect everyone to do every requirement as ASVS is quickly turning into a catalog of all requirements to consider. My 2 cents.

@jmanico unfortunately I think that since we have so many more requirements, that is why we need more levels.

[elarlang]

@tghosth - I don't get what is difference between levels 0 and 1? Do you imagine/plan to set levels from 0 to 3 to every requirement?

On the other hand, I am not sure I like the "risk based" concept we currently use. We say that L3 requirements are for the most highly sensitive applications but really what we usually mean is that L3 requirements are too complicated or widely scoped for most apps to implement.

If we are not clear with that, we need to say that it is indication, not the rule.

[tghosth]

@tghosth - I don't get what is difference between levels 0 and 1?

Level 0 is designed as a gateway to the rest. It includes the obvious/easy/high value requirements but we cannot say it provides sufficient security to be a real level.

Level 1 is designed as the minimum level where we can say there is some actual level of security here.

Do you imagine/plan to set levels from 0 to 3 to every requirement?

Yes

If we are not clear with that, we need to say that it is indication, not the rule.

I agree, this is why we want to clarify the documentation :)

[cmlh]

If the [DNS] "implementation" the application depends on is excluded then we'll need to remove the definition of "pentestable" as its dependent infrastructure can no longer be verified as per #1788 et al.

[tghosth]

In an internal working group meeting, the idea of a Level 0 was not popular.

I personally am strongly against the 2 levels approach as I don't think it allows enough granularity and leads to a massive workload to implement simultaneously.

I have moved this to a discussion here: https://github.com/OWASP/ASVS/discussions/1839