Define site/app scanning

[laminfaty]

Is there a way to tell bdd-security which is using owasp-zap not to scan the whole application/website

ex: Scan only http://mysite.com/thisapponly instead of http://mysite.com/

[iriusrisk]

Could you not change the baseUrl in the config file to point to the other location?

If not, then you can modify the runScanner method in the AppScanningSteps.java to scan http://mysite.com/thisapponly instead of the Config.getBaseUrl()

[laminfaty]

Good stuff @continuumsecurity I noted that you add an ability to exclude some URLs. How about if someone want to exclude gazillions of URLs? I think the best approach is to be able to tell zap to spider and scan only an specific/unique URL. Let me know what you think.

[laminfaty]

In exclude_urls.table, can you give an example of URL exclusion?

[iriusrisk]

I've just added that functionality today and I can't get it to work 100% in ZAP. E.g. if I use the ZAP API directly through the browser and tell it to ignore .*user.* Then do a scan of the entire site, I can see that some of the alerts were found in a url like: http://localhost/user/list/something But in the Active Scan tab of ZAP, no urls with .*user.* are listed. So it looks like it does scan those URLs, but then lies about it. Will have to double check this and discuss with the zap team.

[laminfaty]

I have opened a similar issue few days ago. Maybe you can follow up and see what they are going to day.

http://code.google.com/p/zaproxy/issues/detail?id=1527&can=4

[iriusrisk]

I've not used the command line, but that is definitely possible in the API: http://zap/UI/ascan/action/scan/

[laminfaty]

Cool. So, what is the plan going forward?

[iriusrisk]

Next iteration will have the URLs to scan in a table, similar to how it's done with the exclude_urls.table and how it's done in the spider in the navigate_app.story No ETA on it though.

[laminfaty]

So, basically the URL to scan file will be dependent to the config file. Also in near future, we need to think about why to include some heavy fuzzing tests.

[iriusrisk]

The scanning scenarios will look like this:

Scenario: The application should not contain SQL injection vulnerabilities
Meta: @id scan_sql_injection
GivenStories: navigate_app.story
Given a scanner with all policies disabled
And the URL regular expressions listed in the file: tables/exclude_urls.table are excluded from the scanner
And the SQL-Injection policy is enabled
And the attack strength is set to High
And the alert threshold is set to Medium
When the scanner is run against the urls in: tables/include_urls.table
And false positives described in: tables/false_positives.table are removed
Then no Medium or higher risk vulnerabilities should be present

[iriusrisk]

ZAP doesn't expose its fuzzer in the API yet. Were you thinking of using another fuzzing tool?

[laminfaty]

@continuumsecurity The above scenario looks very good. I use Kali a lot and I think it has ton of fuzzing tools. I will try to pick a good fuzzing tool. Do you have any specific criteria for fuzzing tool?

[iriusrisk]

Something popular that other users are likely to use :)

[laminfaty]

Sounds good. When do you think these new functionalities will be done? Also did you take a look at the other issue that I open?

[laminfaty]

@continuumsecurity I just tested the exclude-urls and it was ok, however, it excludes everything. So I think you need to add the include-urls functionality.

[laminfaty]

@continuumsecurity This URL below is not part of the exclude-urls (regex), however it was filtered.

117813 [pool-1-thread-1] INFO org.zaproxy.zap.spider.SpiderController - URI: http://mywebsite/scanthisapp was filtered by a filter with reason: USER_RULES

[iriusrisk]

what regex are you using?

[laminfaty]

Here is my exclude-urls table as you see these are my excluded urls.

|regex | |.scanthisapp.| |.my-login.|

[laminfaty]

@continuumsecurity Any update or progress on the include-urls functionality to fixing owasp-zap scan.

[iriusrisk]

Earliest would be next week. The regex's in the table above don't look right. They should probably be:

    |regex                            |
    |.*scanthisapp.*             |
    |.*my-login.*                  |

[laminfaty]

@continuumsecurity My regex has pipe and asterisk. You mean to tell me that no asterisk or pipe. Because it is markdown that's why the asterisk is not showing.

[iriusrisk]

Ah, gotcha, fixed my comment as well if someone else stumbles on this issue.

[laminfaty]

So to clearify the regex will have pipe and asterisk. Coorect me if I am wrong. I think my regex is fine. The issue is zap filtering everything. Include-urls should fix this issue, I hope.

[iriusrisk]

Yes pipe and asterisk. I don't see why zap is filtering everything. Could you try doing the exact same thing but without the BDD-Sec framework. I.e.

1. configure browser to use zap
2. browse to target site and navigate around
3. browse to http://zap
   • this will take you to the API
   • navigate to the ascan exclude urls option
   • add your regex
     1. go back to ZAP and choose to scan the site.

[laminfaty]

Ok I will explore that.

[laminfaty]

@continuumsecurity After deep analysis of zap proxy scanner, I find out that using a regex (/.someurl.). Basically, a forward slash, dot, and asterisk in front around the excluded url on API works (404 error when scanning the excluded url). I am still testing it and I think we can get it work if we use exclude-urls and include-urls.

[laminfaty]

@continuumsecurity I hope you are doing well. I did some experimentations with Zap and find out that the best approach to get a specific URL scanned by Zap is to use: excludeFromScan (regex) form the api http://zap.UI/spider scan (url) from the api http://zap.UI/spider. This points to the URL that you wish to scan.

I tested it local and it works for me.

[iriusrisk]

Thanks @laminfaty, I haven't have time to add this functionality yet but I'm aware that it is still outstanding!

[laminfaty]

You are welcome @continuumsecurity. If you need assistance please let me know. I am very excited and looking forward for that new functionality.

[lfatty]

@continuumsecurity, Any update on the include_urls functionality?

[iriusrisk]

I'm upgrading to ZAP 2.4 and this feature will be included in the next release, ETA Friday 16th April.

[lfatty]

Thanks @continuumsecurity and I cannot wait. Also with ZAP, are you using active scan or passive scan plugins. The reason I ask is that on my jenkins console I am only seeing passive scan plugins.

[iriusrisk]

@lfatty with regards to the include_urls feature. I can't find a clear way to do this using ZAP. Are you able to replicate this functionality using the ZAP GUI? If so, could you explain how?

[lfatty]

@continuumsecurity, yes indeed I used ZAP GUI.

[iriusrisk]

Could you explain the steps you take?

[lfatty]

I did some experimentations with Zap and find out that the best approach to get a specific URL scanned by Zap is to use: excludeFromScan (regex) form the api http://zap.UI/spider scan (url) from the api http://zap.UI/spider. This points to the URL that you wish to scan.

I tested it local and it works for me.

[iriusrisk]

OK, but this is already how BDD-Security works. You can check the navigate_app.story file which does the spidering and it's the same workflow: first define the URLs to exclude then spider a set of URLs.

[lfatty]

So, I guess the issue maybe with ZAP. But take a look at the api http://zap.ui/spider and look at scan(url) function closely. Perhaps, something is missing.