bug: Detection of alive hosts

psyray commented 9 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

@yogeshojha @AnonymousWP

According to this issue #1056 and some investigation on my side, I think we have a problem with the detection of alive hosts.

I explain

This piece of code is used to check if an endpoint is alive https://github.com/yogeshojha/rengine/blob/52b4baa4a63586ba959b8e4a56a1ac79452493fd/web/reNgine/common_func.py#L338-L340

The main problem with this check is that it is used as the base check to launch scan like

So the method get_http_urls is mandatory to launch scan of the above type.

The main problem comes from the is_alive method of the Endpoint class in the startScan model https://github.com/yogeshojha/rengine/blob/52b4baa4a63586ba959b8e4a56a1ac79452493fd/web/startScan/models.py#L374-L375

As you can see, if, in those conditions :

an URL returns 404 or http status code above 500
get_only_default_urls option is set to true
No default URL has been set (the default URL are set only in the target scan, not subdomain scan) https://github.com/yogeshojha/rengine/blob/52b4baa4a63586ba959b8e4a56a1ac79452493fd/web/reNgine/tasks.py#L146-L152

No base url is returned, so no scan is launched.

It's problematic because dir_file_fuzz could be launched even if the base endpoint returned 404, and it's the same thing for fetch_url and vulnerability_scan

So we need to rework this part to always send to some tools the base URL, and also correctly set the

Expected Behavior

From the moment we have a subdomain, that have an IP, and give some HTTP response, we must run :

dir_file_fuzz
vulnerability_scan
fetch_url

Steps To Reproduce

Try to launch a scan on a website that have the base URL responding HTTP status code >= 500 or 404

Environment

- reNgine: 2.0.2
- OS: debian
- Python: 3.10

Workaround

As a workaround to launch scan :

Set enable_http_crawl to false in the launched scanEngine could help launching scan with those HTTP status returned cases
```
'enable_http_crawl': false,
```
Also, using the admin endpoints and setting is_default to Yes on the base endpoint could solve the issue

Modifying those 2 values could be needed

github-actions[bot] commented 9 months ago

👋 Hi @psyray, Issues is only for reporting a bug/feature request. Please read documentation before raising an issue https://rengine.wiki For very limited support, questions, and discussions, please join reNgine Discord channel: https://discord.gg/azv6fzhNCE Please include all the requested and relevant information when opening a bug report. Improper reports will be closed without any response.

Talanor commented 5 months ago

@psyray Do you have any update or ETA on this?

yogeshojha / rengine