search

maurosoria / dirsearch

Web path scanner
11.81k stars 2.3k forks source link

Verbose mode? (feature request) #812

Closed ghost closed 2 years ago

ghost commented 3 years ago

Hello Folks! Lots of improvements had been made to dirsearch! Kudos for this Guys! I really enjoyed while playing with new version! (v0.4.1)

For me it seems only missing functionality is a: Verbosity. I think that would be really nice to see such output (via verbose mode enabled of course) while applying forced browsing using dirsearch: Especially status-code, response size may be usefull (for initial adjustements) Something like this:

python3 dirsearch.py -u "http://somedomain.tld/" --full-url -b -f -e php,txt,html,js,zip,log,rar -t 120 --proxy=http://127.0.0.1:3131 -x 403 -w boom.txt **--verbose=3**

Where we can control verbose level like --verbose=**_INTEGER_**

Output: insenz.tar [Status: 200, Size: 2854] nail-care.bz2 [Status: 200, Size: 2513] nail-care.tar.gz [Status: 200, Size: 5564] mech.zip [Status: 200, Size: 3984] dwts [Status: 200, Size: 6841] .........

Thanks in advance & Keep Up!

maurosoria commented 3 years ago

Hello @DarkRed777 ! How are you doing?

I always though that a debugging view would be helpful for development more than a "Feature".

What fields or elements do you whink are important for a verbose output?

ghost commented 3 years ago

Hello @DarkRed777 ! How are you doing?

I always though that a debugging view would be helpful for development more than a "Feature".

What fields or elements do you whink are important for a verbose output?

Hello Dear, Maurosoria ! Thank you very much for your reply! <3 Well, I agree with you regarding debugging purposes but it's also usefull while real pentests. IMHO adjustable verbosity gives a clear image of what's going on; returned status codes, current URL, response size should be enough i think. I have lots of cases when i needed to see status code and response size for each request during forced browsing. To be honest since dirsearch has no --verbose option the workaround for me is only to looking n(tail -f) at squid logs.

1619423367.160      2 127.0.0.1 TCP_MISS/301 469 GET http://somedomain.tld/cde.txt - HIER_DIRECT/XX.YYY.ZZ.PP text/html
1619423367.174      2 127.0.0.1 TCP_MISS/301 470 GET http://somedomain.tld/cde.html - HIER_DIRECT/XX.YYY.ZZ.PP text/html
1619423367.177      2 127.0.0.1 TCP_MISS/301 468 GET http://somedomain.tld/cde.js - HIER_DIRECT/XX.YYY.ZZ.PP text/html
1619423367.182      2 127.0.0.1 TCP_MISS/301 469 GET http://somedomain.tld/cde.zip - HIER_DIRECT/XX.YYY.ZZ.PP text/html
1619423367.189      2 127.0.0.1 TCP_MISS/301 469 GET http://somedomain.tld/cde.log - HIER_DIRECT/XX.YYY.ZZ.PP text/html
1619423367.195      2 127.0.0.1 TCP_MISS/301 469 GET http://somedomain.tld/cde.rar - HIER_DIRECT/XX.YYY.ZZ.PP text/html
1619423367.200      2 127.0.0.1 TCP_MISS/301 465 GET http://somedomain.tld/cde - HIER_DIRECT/XX.YYY.ZZ.PP text/html
1619423367.207      2 127.0.0.1 TCP_MISS/301 466 GET http://somedomain.tld/cde/ - HIER_DIRECT/XX.YYY.ZZ.PP text/html
1619423367.208      2 127.0.0.1 TCP_MISS/301 475 GET http://somedomain.tld/emailPage.php - HIER_DIRECT/XX.YYY.ZZ.PP text/html
1619423367.219      2 127.0.0.1 TCP_MISS/301 475 GET http://somedomain.tld/emailPage.txt - HIER_DIRECT/XX.YYY.ZZ.PP text/html

Anyways, Thank you very much Guys for such a great tool!

Cheers,

shelld3v commented 3 years ago

Right now, dirsearch is printing status code, human-readable size, URL and redirect. What do you expect?

ghost commented 3 years ago

Right now, dirsearch is printing status code, human-readable size, URL and redirect. What do you expect more?

Hello @shelld3v Well, i can't see them? (There is no response size and status code in this stage) (v (latest): v0.4.1) Please note that: This is from ongoing forced browsing:: that's why i'm asking such a usefull feature --verbose flag.

1) Screenshot: https://i.ibb.co/n31nKBp/1.png (https://ibb.co/CJ9MS6L)

Thanks in advance!

shelld3v commented 3 years ago
  1. Screenshot: https://i.ibb.co/n31nKBp/1.png (https://ibb.co/CJ9MS6L)

LOL, probably this is the first time you use dirsearch. The reason that you haven't seen anything is that dirsearch hasn't found any interesting endpoints yet. Only the results are printed with status codes and sizes

Welcome to dirsearch!

ghost commented 3 years ago
  1. Screenshot: https://i.ibb.co/n31nKBp/1.png (https://ibb.co/CJ9MS6L)

LOL, probably this is the first time you use dirsearch. The reason that you haven't seen anything is that dirsearch hasn't found any interesting endpoints yet. Only the results are printed with status codes and sizes

Welcome to dirsearch!

LMAO) Dear @shelld3v! dirsearch is mydaily tool) It seems you don't understand me why am i asking --verbose flag. I need --verbose flag only for ongoing forced browsing to print returned status codes, current URL, response size for each ongoing request (it doesn't matter dirsearch found || not found anything). Got it?

shelld3v commented 3 years ago

ongoing forced browsing

You meant the ... - Last request to: ...?

ghost commented 3 years ago

ongoing forced browsing

You meant the ... - Last request to: ...?

Yes! To make a clear please take a look to the following output: (this is from wfuzz) That's why i'm looking --verbose flag) Hope that makes sense.

$ python wfuzz.py --filter="(c!=404)" -c -z file,wordlist/general/megabeast.txt http://192.168.0.102/FUZZ

Target: http://192.168.0.102/FUZZ Total requests: 45463

================================================================== ID Response Lines Word Chars Request
==================================================================```

00009: C=301 7 L 20 W 235 Ch "admin" 11359: C=301 7 L 20 W 234 Ch "data" 21263: C=301 7 L 20 W 238 Ch "includes" 41255: C=301 7 L 20 W 234 Ch "test"

Total time: 36.16794 Processed Requests: 45463 Filtered Requests: 45459 Requests/sec.: 1256.997

shelld3v commented 3 years ago

ongoing forced browsing

You meant the ... - Last request to: ...?

Yes! To make a clear please take a look to the following output: (this is from wfuzz) That's why i'm looking --verbose flag) Hope that makes sense.

$ python wfuzz.py --filter="(c!=404)" -c -z file,wordlist/general/megabeast.txt http://192.168.0.102/FUZZ

  • Wfuzz 2.1.5 - The Web Bruteforcer *

Target: http://192.168.0.102/FUZZ Total requests: 45463

================================================================== ID Response Lines Word Chars Request ==================================================================```

00009: C=301 7 L 20 W 235 Ch "admin" 11359: C=301 7 L 20 W 234 Ch "data" 21263: C=301 7 L 20 W 238 Ch "includes" 41255: C=301 7 L 20 W 234 Ch "test"

Total time: 36.16794 Processed Requests: 45463 Filtered Requests: 45459 Requests/sec.: 1256.997

There are 3 reasons I can't agree with you:

The output now is pretty good, right?

  _|. _ _  _  _  _ _|_    v0.4.1
 (_||| _) (/_(_|| (_| )

Extensions: php | HTTP method: GET | Threads: 30 | Wordlist size: 8914

Error Log: /home/admin/dirsearch/logs/errors-21-04-26_16-43-38.log

Target: https://[redacted]

Output File: /home/admin/dirsearch/reports/[redacted]/_21-04-26_16-43-38.txt

[16:43:38] Starting:
[16:43:40] 500 -    3KB - /.jsp
[16:43:42] 500 -    3KB - /HelloHTML.jsp
CTRL+C detected: Pausing threads, please wait...
[e]xit / [c]ontinue: e

Canceled by the user
ghost commented 3 years ago

ongoing forced browsing

You meant the ... - Last request to: ...?

Yes! To make a clear please take a look to the following output: (this is from wfuzz) That's why i'm looking --verbose flag) Hope that makes sense. $ python wfuzz.py --filter="(c!=404)" -c -z file,wordlist/general/megabeast.txt http://192.168.0.102/FUZZ

  • Wfuzz 2.1.5 - The Web Bruteforcer *

Target: http://192.168.0.102/FUZZ Total requests: 45463

ID Response Lines Word Chars Request ==================================================================``` 00009: C=301 7 L 20 W 235 Ch "admin" 11359: C=301 7 L 20 W 234 Ch "data" 21263: C=301 7 L 20 W 238 Ch "includes" 41255: C=301 7 L 20 W 234 Ch "test" Total time: 36.16794 Processed Requests: 45463 Filtered Requests: 45459 Requests/sec.: 1256.997

There are 3 reasons I can't agree with you:

* The progress bar runs too fast, even we print in out, it's almost impossible for the users to read it

* Adding such information will make the progress bar too long, and bugs will come in play

* I can't find the use cases for this

The output now is pretty good, right?

  _|. _ _  _  _  _ _|_    v0.4.1
 (_||| _) (/_(_|| (_| )

Extensions: php | HTTP method: GET | Threads: 30 | Wordlist size: 8914

Error Log: /home/admin/dirsearch/logs/errors-21-04-26_16-43-38.log

Target: https://[redacted]

Output File: /home/admin/dirsearch/reports/[redacted]/_21-04-26_16-43-38.txt

[16:43:38] Starting:
[16:43:40] 500 -    3KB - /.jsp
[16:43:42] 500 -    3KB - /HelloHTML.jsp
CTRL+C detected: Pausing threads, please wait...
[e]xit / [c]ontinue: e

Canceled by the user

Well,

*"The progress bar runs too fast, even we print in out, it's almost impossible for the users to read it" Agree, but we can pause and scroll up. (Assuming in verbose mode they will be print-out from up to down? (without clobbering))

Adding such information will make the progress bar too long, and bugs will come in play Just printing them from up to down?

I can't find the use cases for this For initial adjustement (since we have --exclude-sizes, -minimal, --maximal etc flags). Then we can simply apply our rules. From other hand verbose mode can help us to get clear image of what's going on.

Thanks!

shelld3v commented 3 years ago

Agree, but we can pause and scroll up. (Assuming in verbose mode they will be print-out from up to down? (without clobbering))

How do you pause?

Just printing them from up to down?

I can't understand this. Can you give me an example of the output?

For initial adjustement (since we have --exclude-sizes, -minimal, --maximal etc flags)

The fact that dirsearch can do this for us. It performs calibration whenever we start a scan

ghost commented 3 years ago

1) How do you pause? CTRL+C ? Program may ask us: [e]xit / [c]ontinue [Change verbosity level]: 0

(If you familiar with "sqlmap" you may see it also supports changing verbosity level on-fly). So people can change verbosity level =0, 1, N... (onfly))

2) I can't understand this. Can you give me an example of the output? As an example here is default output from wfuzz. Everyting printed out to the terminal: (Similar output expected by dirsearch when --verbose flag set to something >= 1 (Just an example)) Nothing fancy here. 

$ python wfuzz.py -p 192.168.31.33:3131 --filter="(c!=403)" -c -z file,wordlist/general/megabeast.txt http://somesite.tld/FUZZ
********************************************************
* Wfuzz 2.1.5 - The Web Bruteforcer                      *
********************************************************

Target: http://somesite.tld/FUZZ
Total requests: 45463

==================================================================
ID  Response   Lines      Word         Chars          Request    
==================================================================

00000:  C=404      7 L        12 W      162 Ch    "_vti_script"
00001:  C=404      7 L        12 W      162 Ch    "abdominal"
00002:  C=404      7 L        12 W      162 Ch    "pr0n"
00003:  C=404      7 L        12 W      162 Ch    "porn"
00004:  C=404      7 L        12 W      162 Ch    "MessagingManager"
00005:  C=404      7 L        12 W      162 Ch    "IISAdmin"
00006:  C=404      7 L        12 W      162 Ch    "maps"
00007:  C=404      7 L        12 W      162 Ch    "MSADC"
00008:  C=404      7 L        12 W      162 Ch    "IISSamples"
00009:  C=404      7 L        12 W      162 Ch    "PBServer"
00010:  C=404      7 L        12 W      162 Ch    "BizTalkServerDocs"
00011:  C=404      7 L        12 W      162 Ch    "_vti_bin"
00012:  C=404      7 L        12 W      162 Ch    "Printers"
00013:  C=404      7 L        12 W      162 Ch    "Ababa"
00014:  C=404      7 L        12 W      162 Ch    "Abby"
00015:  C=404      7 L        12 W      162 Ch    "PBSData"
00016:  C=404      7 L        12 W      162 Ch    "Rpc"
00017:  C=404      7 L        12 W      162 Ch    "abbreviating"
00018:  C=404      7 L        12 W      162 Ch    "abbreviation"
00019:  C=404      7 L        12 W      162 Ch    "abdomen"
00020:  C=404      7 L        12 W      162 Ch    "abdomens"
00021:  C=404      7 L        12 W      162 Ch    "admin"
00022:  C=404      7 L        12 W      162 Ch    "Scripts"
00023:  C=404      7 L        12 W      162 Ch    "Backup"
00024:  C=404      7 L        12 W      162 Ch    "NR"
00025:  C=404      7 L        12 W      162 Ch    "warez"
00026:  C=404      7 L        12 W      162 Ch    "_private"
00027:  C=404      7 L        12 W      162 Ch    "htmldocs"
00028:  C=301      7 L        12 W      178 Ch    "images"
00029:  C=404      7 L        12 W      162 Ch    "_vti_log"
00030:  C=404      7 L        12 W      162 Ch    "content"
00031:  C=404      7 L        12 W      162 Ch    "IISHelp"
00032:  C=404      7 L        12 W      162 Ch    "Aarhus"
00033:  C=404      7 L        12 W      162 Ch    "abducts"
00034:  C=404      7 L        12 W      162 Ch    "abductions"
....................SNIP..............

3) The fact that dirsearch can do this for us. It performs calibration whenever we start a scan True, it does and works great. But --verbose can give us more flexibility instead of relying only dirsearch's auto-callibartion. Verbose mode can help troubleshoot problems, give some clues to us, to help make some adjustements / callibrations in timely fashion, increase program's flexibility etc.

Huge Thanks!

shelld3v commented 3 years ago

Oh, I misunderstood you! You want to print ALL results, all paths that have been tested, not just paths that exist on website (paths that give real responses, not 404 or stuff like that). Hmmm, seems useless, because:

For initial adjustement (since we have --exclude-sizes, -minimal, --maximal etc flags)

The fact that dirsearch has already done this for us. It performs calibration whenever we start a scan

shelld3v commented 3 years ago

calibration = adjusting filters to clean wildcard results

ghost commented 3 years ago

Oh, I misunderstood you! You want to print ALL results, all paths that have been tested, not just paths that exist on website (paths that give real responses, not 404 or stuff like that). Hmmm, seems useless, because:

For initial adjustement (since we have --exclude-sizes, -minimal, --maximal etc flags)

The fact that dirsearch has already done this for us. It performs calibration whenever we start a scan

Yep!)

Auto-callibration may suck. (Sorry) Use case: Suppose you dirbusting real website. Dirsearch's callibration etc are automatically applied. Ok Lovely Blue team notices forced browsing to the website and denies access to website || applies some ACL (via .htaccess). Now website returns 403 status code to your every request. Your dirbusting goes .... as nothing had happened. Until now you blindly dirbusting the website. Think about what if you dirbusting with very big dictionary? You'll waste your time. That's why i think we need verbose mode. (on-fly changeable verbose - that would be fantastic solution!) Believe me as web application penetration is my daily job i have seen lot's of such scenarios.

$ python3 dirsearch.py -u "http://SOMESITE.COM/" --full-url -b -f -e php,txt,html,js,zip,log,rar -t 120 --proxy=http://127.0.0.1:3131 -w /root/tools/dirsearchnew/dirsearch/shuf1.txt

  _|. _ _  _  _  _ _|_    v0.4.1
 (_||| _) (/_(_|| (_| )

Extensions: php, txt, html, js, zip, log, rar | HTTP method: GET | Threads: 120 | Wordlist size: 1093395

Error Log: /root/tools/dirsearch26042021/dirsearch/logs/errors-21-04-26_19-05-44.log

Target: http://SOMESITE.COM/

Output File: /root/tools/dirsearch26042021/dirsearch/reports/SOMESITE.COM/_21-04-26_19-05-44.txt

[19:05:44] Starting: 
0.40% | 263 req/s - Last request to: get_block.txt^Z

[1]+  Stopped                 python3 dirsearch.py -u "http://SOMESITE.COM/" --full-url -b -f -e php,txt,html,js,zip,log,rar -t 120 --proxy=http://127.0.0.1:3131 -w /root/tools/dirsearchnew/dirsearch/shuf1.txt

[root@localhost dirsearch]# tail -n 10 /var/log/squid/access.log
1619449561.510      0 127.0.0.1 TCP_MISS/403 476 GET http://SOMESITE.COM/mu-gb.txt - HIER_DIRECT/SOMESITE.COM text/html
1619449561.512      0 127.0.0.1 TCP_MISS/403 475 GET http://SOMESITE.COM/mu-gb.js - HIER_DIRECT/SOMESITE.COM text/html
1619449561.524      0 127.0.0.1 TCP_MISS/403 476 GET http://SOMESITE.COM/mu-gb.zip - HIER_DIRECT/SOMESITE.COM text/html
1619449561.525      1 127.0.0.1 TCP_MISS/403 476 GET http://SOMESITE.COM/mu-gb.log - HIER_DIRECT/SOMESITE.COM text/html
1619449561.533      1 127.0.0.1 TCP_MISS/403 476 GET http://SOMESITE.COM/mu-gb.rar - HIER_DIRECT/SOMESITE.COM text/html
1619449561.533      0 127.0.0.1 TCP_MISS/403 472 GET http://SOMESITE.COM/mu-gb - HIER_DIRECT/SOMESITE.COM text/html
1619449561.535      0 127.0.0.1 TCP_MISS/403 473 GET http://SOMESITE.COM/mu-gb/ - HIER_DIRECT/SOMESITE.COM text/html
1619449561.539      0 127.0.0.1 TCP_MISS/403 474 GET http://SOMESITE.COM/SQL.txt - HIER_DIRECT/SOMESITE.COM text/html
1619449561.541      0 127.0.0.1 TCP_MISS/403 474 GET http://SOMESITE.COM/SQL.php - HIER_DIRECT/SOMESITE.COM text/html
1619449561.543      0 127.0.0.1 TCP_MISS/403 475 GET http://SOMESITE.COM/SQL.html - HIER_DIRECT/SOMESITE.COM text/html

Cheers!

shelld3v commented 3 years ago

Auto-callibration may suck

I don't think so!

I don't think I'm going to implement this now, because of its rare use cases (really really rare). I will consider this in the future, but now we are working on so many enhancements that many users need.

shelld3v commented 2 years ago

Hi @DarkRed777, I decided to log all the requests (including uninteresting ones, as your suggestion) into a log file, in there you can find basic information of any request like status, method, length, ... This will be done after my PR gets merged, I hope you love it

ghost commented 2 years ago

Hi @DarkRed777, I decided to log all the requests (including uninteresting ones, as your suggestion) into a log file, in there you can find basic information of any request like status, method, length, ... This will be done after my PR gets merged, I hope you love it

Thank you very much and keep up bro! <3

Cheers!