Allow more than 1 screenshot per host

[mbrownnycnyc]

Hello,

This is a bit of a bug report and a bit (apparently) of a request.

I'd like to pass URL:s from a file into witnessme screenshot and have a screenshot of each URL be generated, saved and the related record be added to the database; however, it is failing to act as I expect or as I wish. :)

For example, I have run gobuster against vulnversity on tryhackme and have saved the output to a file as:

http://10.10.162.217:3333/.htpasswd
http://10.10.162.217:3333/.hta
http://10.10.162.217:3333/.htaccess
http://10.10.162.217:3333/css
http://10.10.162.217:3333/fonts
http://10.10.162.217:3333/images
http://10.10.162.217:3333/index.html
http://10.10.162.217:3333/internal
http://10.10.162.217:3333/js
http://10.10.162.217:3333/server-status

I then invoke witnessme as follows and rcv the following output:

kali@DESKTOP-67BAPDH:~$ cat tcp3333_dirb_common.txt | witnessme --threads 4 screenshot -
[witnessme.screenshot] INFO - Starting scan bcbae438-27be-43c4-990f-714c0eb30d98
[witnessme.headlessbrowser] INFO - Starting headless browser
[witnessme.headlessbrowser] INFO - Using 4 browser tab(s)/thread(s)
[witnessme.screenshot] INFO - Took screenshot of http://10.10.143.64:3333/.htpasswd
[witnessme.screenshot] INFO - Took screenshot of http://10.10.143.64:3333/.hta
[witnessme.screenshot] INFO - Took screenshot of http://10.10.143.64:3333/.htaccess
[witnessme.screenshot] INFO - Took screenshot of http://10.10.143.64:3333/css
[witnessme.headlessbrowser] INFO - Killing headless browser
[witnessme.headlessbrowser] INFO - Starting headless browser
[witnessme.headlessbrowser] INFO - Using 4 browser tab(s)/thread(s)
[witnessme.headlessbrowser] INFO - total: 10, done: 4, pending: 6
[witnessme.screenshot] INFO - Took screenshot of http://10.10.143.64:3333/fonts
[witnessme.screenshot] INFO - Took screenshot of http://10.10.143.64:3333/images
[witnessme.headlessbrowser] INFO - total: 10, done: 6, pending: 4
[witnessme.screenshot] INFO - Took screenshot of http://10.10.143.64:3333/internal
[witnessme.headlessbrowser] INFO - total: 10, done: 7, pending: 3
[witnessme.screenshot] INFO - Took screenshot of http://10.10.143.64:3333/index.html
[witnessme.headlessbrowser] INFO - Killing headless browser
[witnessme.headlessbrowser] INFO - Starting headless browser
[witnessme.headlessbrowser] INFO - Using 2 browser tab(s)/thread(s)
[witnessme.screenshot] INFO - Took screenshot of http://10.10.143.64:3333/js
[witnessme.screenshot] INFO - Took screenshot of http://10.10.143.64:3333/server-status
[witnessme.headlessbrowser] INFO - Killing headless browser
[witnessme.screenshot] INFO - Saved scan to scan_2020_10_17_073824/

The results are very interesting and are as follows:

kali@DESKTOP-67BAPDH:~$ wmdb scan_2020_10_17_073824/
[!] Press tab for autocompletion and available commands
WMDB ≫ servers
+----+------------------------------------+---------------+------------------------+----------------------+
| Id | URL                                | Title         | Server                 | Matched Signature(s) |
+----+------------------------------------+---------------+------------------------+----------------------+
| 1  | http://10.10.143.64:3333/.htpasswd | 403 Forbidden | Apache/2.4.18 (Ubuntu) | None                 |
+----+------------------------------------+---------------+------------------------+----------------------+
WMDB ≫ hosts
+----+--------------+----------+---------------------+----------------------+
| Id | IP           | Hostname | Discovered Services | Matched Signature(s) |
+----+--------------+----------+---------------------+----------------------+
| 1  | 10.10.143.64 |          | 1                   |                      |
+----+--------------+----------+---------------------+----------------------+
WMDB ≫ generate_report
[witnessme.signatures] DEBUG - Loaded 296 signature(s)
[witnessme] DEBUG - Starting signature scan...
[witnessme] DEBUG - Signature scan completed, identified 0 service(s) in 00m00s
[witnessme.reporting] INFO - Generating HTML report, please wait...
[witnessme.reporting] INFO - Done

In the generated report, the URL is noted as it is in id 1 of wmdb servers output (http://10.10.143.64:3333/.htpasswd). However, the screenshot is that of http://10.10.143.64:3333/js; this screenshot is the second to last screenshot noted in witnessme stdout. This is reproducible on demand, and I believe it must be related to the thread logic.

Environmental notes: 1) I am running this on kali in WSL2, which, as far as I'm aware, is irrelevant and should not cause oddities, as WSL2 is running as a VM within a "lite" version of hyperv. 2) specifically, I am targeting vulnversity on tryhackme.

If I attempt this with --threads 1, then things appear to be predictable... the noted string is that of the first item (that contained in wmdb servers output). and the last screenshot produced (as noted in the witnessme stdout.

So, I have only have one ask: This is all sort of weird... but not that weird since you clearly didn't code to support multiple screenshots of URLs from the same host(?). Can you extend support to allow for multiple URLs from the same host?

Thanks

[byt3bl33d3r]

@mbrownnycnyc Sorry for the late reply, I understand the issue now.

The reason why the database doesn't have all of the screenshots is because it was designed to have only 1 screenshot of each unique host (This is done via SQL UNIQUE constraints here). When I was writing this, this really was the primary use case, i really didn't want more than 1 screenshot of a single host.

That being said, I can see the value in what you're using this for, however I'm not sure if I'd want to implement it under the screenshot command. I'll have to think about how to implement this functionality.

Thanks