search

intruxxer / zaproxy

Automatically exported from code.google.com/p/zaproxy
0 stars 0 forks source link

Forced Browsing should (allow to) display all the activity #559

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1. Run Forced Browsing on a site.
2. No activity is shown in the bottom pane (ForcedBrowse.jpg).

What is the expected output? What do you see instead?
Activity (request/response) shown in bottom pane (similar to Active Scan or 
Spider, ActiveScan.jpg).

What version of the product are you using? On what operating system?
Zap: 2.0.0
OS: Win 2003 R2 SP1
Java:
C:\>java -version
java version "1.7.0_17"
Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
Java HotSpot(TM) Client VM (build 23.7-b01, mixed mode, sharing)

Please provide any additional information below.

Original issue reported on code.google.com by kingtho...@gmail.com on 12 Mar 2013 at 2:09

Attachments:

GoogleCodeExporter commented 9 years ago 
What are the exact steps to reproduce the issue?

I tried with all the (forced browsing) options but I wasn't able to reproduce 
it.

Original comment by THC...@gmail.com on 13 Mar 2013 at 3:07

GoogleCodeExporter commented 9 years ago 
1) Launch ZAP (from desktop shortcut the installer created).
2) Set browser to use ZAP's local proxy.
3) In browser: Click on relevant bookmark for target site. Create temporary SSL 
exception (ZAP's cert).
4) Return to ZAP.
5) Right-click on target site, select "Attack : Forced Browse site" (as seen in 
the attached screen shot the small directorylist is set as default).
6) No activity appears in the bottom pane (as seen in the screenshot above). 
(For ForcedBrowsing concurrent threads are currently set to 2, though I also 
experienced this at 10). The progress bar does it's thing to 100%. Wireshark 
indicates that there is indeed traffic being sent.

Note: This is a side-by-side install (though I don't think it'll matter):
1.4.1: C:\Program Files (x86)\OWASP\Zed Attack Proxy
2.0.0: C:\Program Files (x86)\OWASP\Zed Attack Proxy2

Original comment by kingtho...@gmail.com on 13 Mar 2013 at 3:46

GoogleCodeExporter commented 9 years ago 
Are there any errors in the ZAP log file?

Original comment by psii...@gmail.com on 14 Mar 2013 at 8:26

GoogleCodeExporter commented 9 years ago 
C:\Program Files (x86)\OWASP\Zed Attack Proxy2\log

contains on dummy.txt which is empty.

If I launch ZAP using zap.bat I get the following output in the console when I 
run Forced Browsing (it's un-eventful):

----- snip -----

42635 [ZAP-PassiveScanner] INFO net.htmlparser.jericho  - Encountered possible S
tartTag at (r43,c17,p1422) whose content does not match a registered StartTagTyp
e
47050 [AWT-EventQueue-0] INFO org.zaproxy.zap.extension.bruteforce.BruteForce  -
 BruteForce : redacted.com:443/null threads: 2
Starting dir/file list based brute forcing
DirBuster Stopped
3287208 [Thread-35] INFO org.zaproxy.zap.extension.bruteforce.BruteForce  - Brut
eForce : redacted.com:443 finished

----- end -----

If I scroll back in the zap.bat command prompt window I do see that at start-up 
there was some sort of issue with the logger (might be a completely separate 
issue):

----- snip -----
2683 [main] INFO org.parosproxy.paros.core.scanner.PluginFactory  - loaded plugi
n External redirect
2683 [main] INFO org.parosproxy.paros.core.scanner.PluginFactory  - loaded plugi
n CRLF injection
2683 [main] INFO org.parosproxy.paros.core.scanner.PluginFactory  - loaded plugi
n Parameter tampering
java.io.IOException
        at org.owasp.jbrofuzz.system.Logger.checkOrCreateDirs(Logger.java:127)
        at org.owasp.jbrofuzz.system.Logger.appendToLogFile(Logger.java:187)
        at org.owasp.jbrofuzz.system.Logger.log(Logger.java:177)
        at org.owasp.jbrofuzz.core.Verifier.loadFile(Verifier.java:120)
        at org.owasp.jbrofuzz.core.Database.<init>(Database.java:81)
        at org.zaproxy.zap.extension.fuzz.ExtensionFuzz.<init>(Unknown Source)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Sou
rce)
        at java.lang.reflect.Constructor.newInstance(Unknown Source)
        at org.zaproxy.zap.control.AddOnLoader.getImplementors(Unknown Source)
        at org.zaproxy.zap.control.AddOnLoader.getImplementors(Unknown Source)
        at org.zaproxy.zap.control.ExtensionFactory.loadAllExtension(Unknown Sou
rce)
        at org.parosproxy.paros.control.Control.addExtension(Unknown Source)
        at org.parosproxy.paros.control.AbstractControl.loadExtension(Unknown So
urce)
        at org.parosproxy.paros.control.Control.init(Unknown Source)
        at org.parosproxy.paros.control.Control.initSingletonWithView(Unknown So
urce)
        at org.zaproxy.zap.ZAP.runGUI(Unknown Source)
        at org.zaproxy.zap.ZAP.run(Unknown Source)
        at org.zaproxy.zap.ZAP.main(Unknown Source)
java.io.IOException
        at org.owasp.jbrofuzz.system.Logger.checkOrCreateDirs(Logger.java:127)
        at org.owasp.jbrofuzz.system.Logger.appendToLogFile(Logger.java:187)
        at org.owasp.jbrofuzz.system.Logger.log(Logger.java:177)
        at org.owasp.jbrofuzz.core.Verifier.loadFile(Verifier.java:128)
        at org.owasp.jbrofuzz.core.Database.<init>(Database.java:81)
        at org.zaproxy.zap.extension.fuzz.ExtensionFuzz.<init>(Unknown Source)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Sou
rce)
        at java.lang.reflect.Constructor.newInstance(Unknown Source)
        at org.zaproxy.zap.control.AddOnLoader.getImplementors(Unknown Source)
        at org.zaproxy.zap.control.AddOnLoader.getImplementors(Unknown Source)
        at org.zaproxy.zap.control.ExtensionFactory.loadAllExtension(Unknown Sou
rce)
        at org.parosproxy.paros.control.Control.addExtension(Unknown Source)
        at org.parosproxy.paros.control.AbstractControl.loadExtension(Unknown So
urce)
        at org.parosproxy.paros.control.Control.init(Unknown Source)
        at org.parosproxy.paros.control.Control.initSingletonWithView(Unknown So
urce)
        at org.zaproxy.zap.ZAP.runGUI(Unknown Source)
        at org.zaproxy.zap.ZAP.run(Unknown Source)
        at org.zaproxy.zap.ZAP.main(Unknown Source)
java.io.IOException
        at org.owasp.jbrofuzz.system.Logger.checkOrCreateDirs(Logger.java:127)
        at org.owasp.jbrofuzz.system.Logger.appendToLogFile(Logger.java:187)
        at org.owasp.jbrofuzz.system.Logger.log(Logger.java:177)
        at org.owasp.jbrofuzz.core.Database.addZeroFuzzers(Database.java:123)
        at org.owasp.jbrofuzz.core.Database.<init>(Database.java:82)
        at org.zaproxy.zap.extension.fuzz.ExtensionFuzz.<init>(Unknown Source)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Sou
rce)
        at java.lang.reflect.Constructor.newInstance(Unknown Source)
        at org.zaproxy.zap.control.AddOnLoader.getImplementors(Unknown Source)
        at org.zaproxy.zap.control.AddOnLoader.getImplementors(Unknown Source)
        at org.zaproxy.zap.control.ExtensionFactory.loadAllExtension(Unknown Sou
rce)
        at org.parosproxy.paros.control.Control.addExtension(Unknown Source)
        at org.parosproxy.paros.control.AbstractControl.loadExtension(Unknown So
urce)
        at org.parosproxy.paros.control.Control.init(Unknown Source)
        at org.parosproxy.paros.control.Control.initSingletonWithView(Unknown So
urce)
        at org.zaproxy.zap.ZAP.runGUI(Unknown Source)
        at org.zaproxy.zap.ZAP.run(Unknown Source)
        at org.zaproxy.zap.ZAP.main(Unknown Source)
java.io.IOException
        at org.owasp.jbrofuzz.system.Logger.checkOrCreateDirs(Logger.java:127)
        at org.owasp.jbrofuzz.system.Logger.appendToLogFile(Logger.java:187)
        at org.owasp.jbrofuzz.system.Logger.log(Logger.java:177)
        at org.owasp.jbrofuzz.core.Database.<init>(Database.java:83)
        at org.zaproxy.zap.extension.fuzz.ExtensionFuzz.<init>(Unknown Source)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Sou
rce)
        at java.lang.reflect.Constructor.newInstance(Unknown Source)
        at org.zaproxy.zap.control.AddOnLoader.getImplementors(Unknown Source)
        at org.zaproxy.zap.control.AddOnLoader.getImplementors(Unknown Source)
        at org.zaproxy.zap.control.ExtensionFactory.loadAllExtension(Unknown Sou
rce)
        at org.parosproxy.paros.control.Control.addExtension(Unknown Source)
        at org.parosproxy.paros.control.AbstractControl.loadExtension(Unknown So
urce)
        at org.parosproxy.paros.control.Control.init(Unknown Source)
        at org.parosproxy.paros.control.Control.initSingletonWithView(Unknown So
urce)
        at org.zaproxy.zap.ZAP.runGUI(Unknown Source)
        at org.zaproxy.zap.ZAP.run(Unknown Source)
        at org.zaproxy.zap.ZAP.main(Unknown Source)
3822 [main] INFO org.zaproxy.zap.control.ExtensionFactory  - Load help files for
 extension 'ExtensionWebSocket' and merge with core help.
3853 [main] INFO org.zaproxy.zap.control.ExtensionFactory  - Load help files for
 extension 'ExtensionQuickStart' and merge with core help.
4181 [main] INFO org.parosproxy.paros.extension.filter.FilterFactory  - loaded f
ilter Change user agent to other browsers.
----- end -----

Original comment by kingtho...@gmail.com on 14 Mar 2013 at 2:02

GoogleCodeExporter commented 9 years ago 
The ZAP logs are (normally) created in the ZAP default directory [1] (unless a 
directory was specified [2]), the "log" directory is no longer used.

It seems that there's no forced browsing activity (at least it doesn't seem to 
be finding or able to fetch any page).
If you don't mind, could you use the attached jar and provide again the log?
Attached the zap jar built from branch 2.0 (r2972) (version 2.0.0) with the 
attached patch applied (enables DirBuster debug and prints the DirBuster 
options and configurations).
You still need to run the zap.bat file as the debug messages are printed to 
"standard" output stream.

The last snip is Issue 150.

[1] https://code.google.com/p/zaproxy/wiki/FAQconfig
[2] https://code.google.com/p/zaproxy/wiki/FAQcmdline

Original comment by THC...@gmail.com on 14 Mar 2013 at 4:22

Attachments:

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
I made a copy of directory-list-lowercase-2.3-small.txt with only 15 entries 
(directory-list-lowercase-2.3-extra-small.txt) then used the provided zap.jar. 
Attached is the output dirBust_debug.txt.

As you can see for this list all responses are 404 (which is really no 
surprise, really this is what we should expect). The basecase for 
https://redacted.com:443/ returns a 302 (which is normal). Yet I still see no 
activity in the bottom pane.

Further I checked:
C:\Users\<username>\OWASP ZAP\zap.log

----- snip -----

2013-03-14 13:58:01,373 INFO  Control - New Session
2013-03-14 13:58:16,989 INFO  jericho - StartTag at (r359,c1,p11886) missing 
required end tag - invalid nested start tag encountered before end tag
2013-03-14 13:58:16,989 WARN  jericho - Full sequential parse clearing all tags 
from cache. Consider calling Source.fullSequentialParse() manually immediately 
after construction of Source.
2013-03-14 13:58:17,067 INFO  jericho - Encountered possible StartTag at 
(r43,c17,p1417) whose content does not match a registered StartTagType
2013-03-14 13:58:17,441 WARN  ProxyThread - Timeout reading 
https://redacted.com/public/images/customization/Common/dd_sso_stg2_ap_general_u
i/logo_image_en.gif
2013-03-14 13:58:17,457 WARN  ProxyThread - Timeout reading 
https://redacted.com/public/images/my/header-transient.png
2013-03-14 13:58:22,651 INFO  BruteForce - BruteForce : redacted.com:443/null 
threads: 2
2013-03-14 13:58:28,798 INFO  BruteForce - BruteForce : redacted.com:443/null 
threads: 2
2013-03-14 13:58:59,249 INFO  BruteForce - BruteForce : redacted.com:443 
finished
2013-03-14 14:04:08,979 INFO  Control - Discard Session
2013-03-14 14:04:08,979 INFO  Scanner - scanner stopped
2013-03-14 14:04:09,307 INFO  ENGINE - dataFileCache commit start
2013-03-14 14:04:09,494 INFO  ENGINE - Database closed
2013-03-14 14:04:09,603 INFO  Control - OWASP ZAP 2.0.0 terminated.

----- end-----

jericho and proxy warning and info items seem unrelated to the dirbuster issue 
being experienced.

It seems that all HEAD requests for directories (even if I do a manual request 
for one that I know must exist) result in 404. Though this should be irrelevant 
to the activity being displayed (one should reasonably expect the majority of 
the dirbuster list requests to return 404).

Original comment by kingtho...@gmail.com on 14 Mar 2013 at 6:41

Attachments:

GoogleCodeExporter commented 9 years ago 
Thanks for taking time to look into it!

From the evidence provided I can say that the behaviour (not showing any 
activity in the bottom pane) is the expected. I was expecting at least one 
entry, the base case ("https://redacted.com:443/"), but since it returns a 
different status code than 200 and no redirections are followed it's normal 
that it's not shown (or crawled).
Only the successful forced browsed responses (status code other than 400 and 
404 or different than the base case) are shown in the bottom pane (the 
directories that would most likely be an issue).
It also shows the "normal" (and successful) requests (the normal requests are 
the base case and URLs found while crawling).

I'll raise an issue to allow to set the "follow redirects" option, this would 
allow to follow the redirect response to "https://redacted.com:443/" (it would 
then be shown in the bottom pane and crawled to find more directories to 
attack).

Yes, the previous log messages (INFO and WARN) are not related.

Does the directories that exist when requested with GET method return a 
different status code than with the HEAD request?
I'd expect the same status code for both HEAD and GET requests (although this 
is not required, as stated in RFC 2616 [3]).
If it returns a different status code there's an option to only use the GET 
method.

[3] https://tools.ietf.org/html/rfc2616#section-9.4

Original comment by THC...@gmail.com on 15 Mar 2013 at 5:17

GoogleCodeExporter commented 9 years ago 
Hi THC, thanks for the follow-up. I expected the lower pane to display all 
activity similar to other tools (active scan, fuzzer, etc). That way as a user:

1) I know something is actually going on (the tool is actually doing something 
beyond generating a progress bar) [yes I can check with wireshark but that's 
not the point].
2) I can see the progress in specific requests, and can see how the server is 
actually behaving.
3) At some later date users might be able to sort based on the response code 
column or other details {grin} 
[http://code.google.com/p/zaproxy/issues/detail?id=353] 

Perhaps a option to display only positives (200s based on normal behavior, and 
30x with follow redirects on) or all requests/responses could be implemented.

As you mentioned being able to control the "follow redirects" setting from 
within ZAP would be useful. It might also be good to expose the control for GET 
vs HEAD usage within the ZAP Forced Browsing options (mark HEAD as default and 
point out the potential bandwidth & time savings).

"Does the directories that exist when requested with GET method return a 
different status code than with the HEAD request?"

No, for example:
https://redacted.com/css exists as test.css is served from within it. However, 
either GETting or HEADing /css results in a 404 (same for a directory that 
doesn't exist). While GETting or HEADing /css/test.css returns a 200.

(I suspect this behavior is a result of tools like AppScan reporting "Hidden 
Directory Detected" i.e.: /css responded 403 Forbidden, then the 
devs/sysadmins/webadmins changed it so that all 403 reported 404...... but 
that's speculation.)

Original comment by kingtho...@gmail.com on 15 Mar 2013 at 1:08

GoogleCodeExporter commented 9 years ago 
Yes, that makes sense. Some things need to be changed in the DirBuster code to 
report all the requests done. Once all the requests are reported it will be 
easy to add the option to filter the requests.

OK, I'll add the "follow redirects" and "GET/HEAD" options. There's already an 
issue to include more DirBuster options (Issue 173), so I'll use that instead 
of creating a new one.

OK, everything is working as expected then.

Original comment by THC...@gmail.com on 26 Mar 2013 at 3:55

GoogleCodeExporter commented 9 years ago 
Has DirBuster been abandon? We seem to be using 0.12, which appears stable.

http://sourceforge.net/projects/dirbuster/files/DirBuster%20%28jar%20%2B%20lists
%29/1.0-RC1/

Doesn't show any progress since 2009.

Original comment by kingtho...@gmail.com on 13 Jul 2014 at 8:52

GoogleCodeExporter commented 9 years ago 
DirBuster has been forked and it's fixed/improved as needed.

Original comment by THC...@gmail.com on 13 Jul 2014 at 10:09