Testability criteria for L1 - do we choose a hard metric or pen testability?

[vanderaj]

I'd like to add two columns, CWE and CVSSv3 score. That way, folks can see why we include it, and why it has the level we want.

CWE may or may not exist CVSSv3 should be from the CVSSv3 calculator, using the base score only.

Then once this is done, let's remap L1, L2, and L3

L1 ... CVSSv3 7.0-10.0, OWASP Top 10 2017 or OWASP Proactive Controls and all passives L2 ... CVSSv4 4.0-7.0. This means PCI DSS 3.2.1 == a Level 2 app L3 ... For apps that can destroy the world or kill you, or special circumstances.

[vanderaj]

For an example, please review check-in https://github.com/OWASP/ASVS/blob/7f436967617cae2965499f4b358d84839897068d/4.0/en/0x24-V19-Config.md

[tghosth]

So I think adding CWE is a great idea. I think CVSS is less subjective than just arbitrarily deciding L1, L2, L3 but I have a few thoughts/concerns

• Adding CVSS across the board will be a big effort when we are running out of time.
• I think CVSS is still a little subjective, especially we are talking about controls/requirements. If people disagree with L ratings, they can comment during the review period 😄
• I would rather that the bar for L1 was not too high to increase the accessibility of applying the standard. On the example you gave, you need to satisfy 10/12 (83%) for L1 and 11/12 (92%) for L2. This was why I opened #316 about helping organisations with sub-L1 apps to prioritise. On the other hand, CVSS could help with this by demonstrating the L1s with the higher risk.

[ghost]

This represents a fairly large shift, especially for L1 items; the assessment document points out that:

It is possible to perform a manual penetration test and verify a large number of L1 issues without requiring access to source code...

However, by shifting to defining L1 by CVSS, this will almost certainly result in a greater number of items that require access to source code to test - shifting L1 further away from assessment via authenticated dynamic (grey-box) testing.

As a result of a conversation on Twitter with @danielcuthbert, I was planning on submitting a pull-request in the very near future that would suggest shifting a few items from L1 to L2, so that L1 can be achieved with grey-box testing (or at least as close as reasonably possible). This would make it possible for companies that prefer grey-box testing to receive a test with full L1 coverage.

If this change in definition goes forward, it will almost certainly make it impossible to achieve any ASVS Level without access to systems and source code, making ASVS irrelevant for a form of testing that is extremely common.

This change would make the changes I planned on proposing irrelevant, as all items would need to be reevaluated and adjusted based on CVSS score - as such this effectively blocks my planned proposal. So at least personally, I would like to see this resolved to determine what the path forward is for L1 and grey-box testing.

[jmanico]

This is a fantastic idea. Go for it!

-- Jim Manico @Manicode Secure Coding Education +1 (808) 652-3805

On Jan 12, 2019, at 5:39 PM, Adam Caudill notifications@github.com wrote:

This represents a fairly large shift, especially for L1 items; the assessment document points out that:

It is possible to perform a manual penetration test and verify a large number of L1 issues without requiring access to source code...

However, by shifting to defining L1 by CVSS, this will almost certainly result in a greater number of items that require access to source code to test - shifting L1 further away from assessment via authenticated dynamic (grey-box) testing.

As a result of a conversation on Twitter with @danielcuthbert, I was planning on submitting a pull-request in the very near future that would suggest shifting a few items from L1 to L2, so that L1 can be achieved with grey-box testing (or at least as close as reasonably possible). This would make it possible for companies that prefer grey-box testing to receive a test with full L1 coverage.

If this change in definition goes forward, it will almost certainly make it impossible to achieve any ASVS Level without access to systems and source code, making ASVS irrelevant for a form of testing that is extremely common.

This change would make the changes I planned on proposing irrelevant, as all items would need to be reevaluated and adjusted based on CVSS score - as such this effectively blocks my planned proposal. So at least personally, I would like to see this resolved to determine what the path forward is for L1 and grey-box testing.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[tghosth]

As a result of a conversation on Twitter with @danielcuthbert, I was planning on submitting a pull-request in the very near future that would suggest shifting a few items from L1 to L2, so that L1 can be achieved with grey-box testing (or at least as close as reasonably possible). This would make it possible for companies that prefer grey-box testing to receive a test with full L1 coverage.

Note: That twitter discussion is here.

[tghosth]

My understanding is that L1 has previously been "the minimum standard for a secure application".

We are now proposing a shift to something like the following table where L1 is no longer what we consider to be "good enough" but rather that is now L2. I think that is fine as long as that is crystal clear in the introduction.

Level	Definition
1	These are bare minimum security requirements but exclude any requirements which cannot be tested grey-box.
2	These are the minimum required controls for an application to be considered secure including OWASP Top 10 Risks and Top 10 Proactive Controls.
3	These are additional important requirements which should be implemented for higher risk applications.

Does this sound correct?

[mgargiullo]

While I'm only one voice, I don't think reducing requirements of a Level 1 test is the right answer. From a security perspective, we're lowering the bar to grant the label...

My understanding was that this type of scenario is exactly what Level 0 was for. I'd much rather see Level 0 formally defined by this body to be a "bare minimum assessment", then shift all levels downward.

Honestly, I feel this denigrates the overall value of an ASVS assessment, but as I noted, I'm but one voice.

[danielcuthbert]

So just to clarify. L1 is still the minimum security standard for applications. Ideally, L1 should be achievable using mostly automated tools, but there are inevitably some aspects where automated tools can't perform this correctly.

L2 is where apps should aspire to be. This would raise the bar considerably.

I'm loathe to make L2 the standard, this sadly is still hard for many and would make the ASVS less attractive to use.

[tghosth]

Ideally, L1 should be achievable using mostly automated tools, but there are inevitably some aspects where automated tools can't perform this correctly.

Hey @danielcuthbert , my understanding was that @adamcaudill was proposing that's pretty much everything in L1 be testable grey-box/automatically. I think there are still various minimum controls that couldn't be tested in this way though.

[ghost]

Hey @danielcuthbert , my understanding was that @adamcaudill was proposing that's pretty much everything in L1 be testable grey-box/automatically.

To clarify this point a bit, my goal isn't about automation - but about being able to test it without access to source code. From the perspective of a 3rd-party security assessment, source code is not available in a fairly high percentage of cases. By making it possible to cover L1 without access to source code, it makes the standard far more useful for 3rd-party assessments.

With 3.0.1, the number of items that are L1 and require source code is extremely small - with 4.0, looks like it's going to be a bit higher, though I haven't completed my detailed review quite yet (family medical emergency has put me behind schedule on this).

With all of this said, my goal of course isn't to weaken the standard, but to make it more useful and easier to apply in the real world. If this can be done with minimal changes, it opens the door for it to be applied in many cases where it can't be used today due to access requirements.

[abhaybhargav]

I'd like to contribute to adding this to ASVS. Is there a specific way you would like this to be done? I am thinking that we add multiple CWEs to each control its associated with as part of another column in ASVS, that way, it will be easier for one control to be associated with fixes across multiple vulnerabilities, which is often the case

[vanderaj]

Let's divide the discussion on CWSS, and keep this ticket about testability. I'm going to rename the ticket.

I agree with Adam but with a nuance - in that L1 should be doable by pentests, but also we have a lot of L1's now, and that's tricky for short engagements or folks who need to do a cursory view. I am not suggesting it's automated only, as that will allow tool vendor marketing to claim ASVS L1 compliance for things that are 100% not automatically testable like access controls, business logic limits, and so on.

Should we have a L0 or Essential Level that is the outright replacement for the OWASP Top 10? I was sort of hoping by using a metric

[vanderaj]

We need to make a decision today, or we are going to miss the release of ASVS. I think CWSS scores will have to be a post 4.0 issue at this stage.

[tkisason]

Well, my opinion that a pentest without source code access is just skimming the surface, and it should be treated as such. Red teaming / threat emulation are a different ballpark.

Maybe we can create a "tag" or something that would mark certain verifications as required to be pentested or that they can be pentested? So just adding a new column?

[elarlang]

It's not only source code - it's also server configuration, logs, firewall configuration etc. And those for every component.

I can not see here black and white truth (L1 with source code or not). There are at least 50 shades of gray box.

I would live tags - this requirement needs source code or logs etc, but I guess it is not realistic mapping for ASVS v4.0 release.

[vanderaj]

Great feedback. I think in the future, if we have a data driven ASVS, such as the Daniel Santander Bank ASVS thingy, we might be able to maintain multiple mappings, but it's unrealistic to include that view on every page. It's already stuggling with the CWSS column (which I am going to strip).

I'm going to make as much of L1 pen-testable as possible, but if we also include OWASP Top 10, there are going to be L1 issues that require access to logs in some fashion. That's not pen-testable, but it's important to understand the OWASP Top 10 is less than minimum viable security, and it requires access to logs, so we need to include that in L1.

[vanderaj]

I'm going to close this one for now. Re-open if after the renumber / re-balance of L1 is not right, or log individual tickets about specific items.