bug: Error creating EyeWitness screenshots with a non-standard URL

[psyray]

Is there an existing issue for this?

• [X] I have searched the existing issues

Current Behavior

Submitted by @solo10010

The task of creating screenshots stops completely if EyeWitness encounters a strange URL

python3 /usr/src/github/EyeWitness/Python/EyeWitness.py -f /usr/src/scan_results/**********.com_9/endpoints_alive.txt -d /usr/src/scan_results/* *********.com_9/screenshots --no-prompt --timeout 10 --threads 40

Scan logs.

TERM environment variable not set.
################################################## ###############################
#EyeWitness#
################################################## ###############################
# Red Siege Information Security - https://www.redsiege.com #
################################################## ###############################

ERROR: You potentially provided an mal-formed URL!
ERROR: URL is - https://account.servicenow.com/sign-in?client_id=0oapi9ote73XCWSst0x7&redirect_uri=https://account.servicenow.com/auth&response_type=code&state=/profile/home?locale=en-us&scope=openid&source_id= account&locale=en-us

Expected Behavior

I expected that if errors occurred, the collection of screenshots would continue and not be interrupted completely.

Steps To Reproduce

1. Install the latest version of rengine
2. Set up any scan with collection of subdomains and creation of screenshots for the servicenow.com domain
3. Monitor scanning tasks, screenshot creation has been stopped on the account.servicenow.com subdomain

Environment

- reNgine: 2.0.2
- OS: Ubuntu 22.04.1 LTS.
- Python: 3.10.4
- Docker engine: 24.0.7
- Docker Compose: v2.23.0
- Browser: Chrome 120.0.6099.224.

Anything else?

No response

[yarysp]

Reproduced, working on it

[yarysp]

Bug came from EyeWitness itself, explication here https://github.com/RedSiege/EyeWitness/pull/605

I've tested the patch, it works fine. I notice a huge load on the VM while screenshot to a lot of targets.

Problem here is that the code of EyeWitness need to be patched, it's quite hard to maintain. https://github.com/RedSiege/EyeWitness/pull/605/files

So we could filter the known bad url before, here using the semi column, and launch the task

I will try that way

[Talanor]

Posted a new issue to see if we must circumvent this on our end : https://github.com/RedSiege/EyeWitness/issues/656

[Relkci]

Is the space in your example URL intended?

ERROR: URL is - https://account.servicenow.com/sign-in?client_id=0oapi9ote73XCWSst0x7&redirect_uri=https://account.servicenow.com/auth&response_type=code&state=/profile/home?locale=en-us&scope=openid&source_id= account&locale=en-us

After the original MR, you would have had traceback because EyeWitness has no logic to handle invalid spaces (and I'm not entirely convinced there is a reason to build the logic since the traceback is sufficiently indicative of the error).

That is to say, https://github.com/RedSiege/EyeWitness/pull/605/files might prevent an incorrect malformed URL error, but will not handle the error resulting from the space in your example.

Anyway, https://github.com/RedSiege/EyeWitness/pull/659

To be clear though, this doesn't address that Eyewitness will stop completely on malformed URLs. Might be worth a new issue to address that explicitly-- to allow the rejection of a malformed URL, rather than completely stopping.

[psyray]

Is the space in your example URL intended?

ERROR: URL is - https://account.servicenow.com/sign-in?client_id=0oapi9ote73XCWSst0x7&redirect_uri=https://account.servicenow.com/auth&response_type=code&state=/profile/home?locale=en-us&scope=openid&source_id= account&locale=en-us

No, not intended. It's a copy-paste from the original issue on the legacy repository of reNgine During my test, I found that the problem came from the ://, spaces were not a problem, or maybe I've fixed it while testing but didn't update the issue to fix the url space.

To be clear though, this doesn't address that Eyewitness will stop completely on malformed URLs. Might be worth a new issue to address that explicitly-- to allow the rejection of a malformed URL, rather than completely stopping.

👍 Thanks for your clarification

[Relkci]

Merged https://github.com/RedSiege/EyeWitness/pull/659

You can probably close #15

Cheers