bug: Detection of alive hosts

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

According to this issue #7 and some investigation on my side, I think we have a problem with the detection of alive hosts.

I explain

This piece of code is used to check if an endpoint is alive https://github.com/Security-Tools-Alliance/rengine-ng/blob/52b4baa4a63586ba959b8e4a56a1ac79452493fd/web/reNgine/common_func.py#L338-L340

The main problem with this check is that it is used as the base check to launch scan like

So the method get_http_urls is mandatory to launch scan of the above type.

The main problem comes from the is_alive method of the Endpoint class in the startScan model https://github.com/Security-Tools-Alliance/rengine-ng/blob/52b4baa4a63586ba959b8e4a56a1ac79452493fd/web/startScan/models.py#L374-L375

As you can see, if, in those conditions :

an URL returns 404 or http status code above 500
get_only_default_urls option is set to true
No default URL has been set (the default URL are set only in the target scan, not subdomain scan) https://github.com/Security-Tools-Alliance/rengine-ng/blob/52b4baa4a63586ba959b8e4a56a1ac79452493fd/web/reNgine/tasks.py#L146-L152

No base url is returned, so no scan is launched.

It's problematic because dir_file_fuzz could be launched even if the base endpoint returned 404, and it's the same thing for fetch_url and vulnerability_scan

So we need to rework this part to always send to some tools the base URL, and also correctly set the default URL for root endpoint

Expected Behavior

From the moment we have a subdomain, that have an IP, and give some HTTP response, we must run :

dir_file_fuzz
vulnerability_scan
fetch_url

Steps To Reproduce

Try to launch a scan on a website that have the base URL responding HTTP status code >= 500 or 404

Environment

- reNgine: 2.0.2
- OS: debian
- Python: 3.10

Anything else?

No response

Security-Tools-Alliance / rengine-ng