[BUG] Fix target being skipped due to max-host-error

[tarunKoyalwar]

Is there an existing issue for this?

• [X] I have searched the existing issues.

Current Behavior

• due to a recent updates related to max-host-error and target skipping whenever we run nuclei with concurrency more than the default one nuclei actively skips that targets

• The current implementation checks for 30 (fixed) permanent errors in a given time period ( 1 minute ) and if any targets satisfy this criteria then it is actively skipped,

• when nuclei is run with default values there is no max-host-error or skipping but this can be seen as soon as concurrency is increased. so instead of hardcoding it to 30 we should set its value to the value of concurrency flag and timeperiod can also be set to 1.5 x -timeout flag value

Expected Behavior

no skipping of targets when its intentional / expected to have some errors

Steps To Reproduce

$ nuclei -u telsa.com -stats -c 200                                                       

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.3.1

        projectdiscovery.io

[WRN] Found 1 templates with runtime error (use -validate flag for further examination)
[INF] Current nuclei version: v3.3.1 (latest)
[INF] Current nuclei-templates version: v9.9.3 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 56
[INF] Templates loaded for current scan: 8429
[INF] Executing 8429 signed templates from projectdiscovery/nuclei-templates
[INF] Targets loaded for current scan: 1
[INF] Running httpx on input host
[INF] Found 1 URL from httpx
[INF] Templates clustered: 1587 (Reduced 1499 Requests)
[0:00:05] | Templates: 8429 | Hosts: 1 | RPS: 240 | Matched: 0 | Errors: 1159 | Requests: 1204/13859 (8%)
[INF] Using Interactsh Server: oast.online
[0:00:10] | Templates: 8429 | Hosts: 1 | RPS: 132 | Matched: 0 | Errors: 1161 | Requests: 1329/13859 (9%)
[0:00:15] | Templates: 8429 | Hosts: 1 | RPS: 116 | Matched: 0 | Errors: 1163 | Requests: 1741/13859 (12%)
[0:00:20] | Templates: 8429 | Hosts: 1 | RPS: 111 | Matched: 0 | Errors: 1173 | Requests: 2230/13859 (16%)
[0:00:25] | Templates: 8429 | Hosts: 1 | RPS: 122 | Matched: 0 | Errors: 1175 | Requests: 3055/13859 (22%)
[azure-domain-tenant] [http] [info] https://login.microsoftonline.com:443/telsa.com/v2.0/.well-known/openid-configuration ["35fc9766-ae04-4cb0-889f-359d312f8c35"]
[0:00:30] | Templates: 8429 | Hosts: 1 | RPS: 132 | Matched: 1 | Errors: 1175 | Requests: 3985/13859 (28%)
[waf-detect:apachegeneric] [http] [info] https://telsa.com
[0:00:35] | Templates: 8429 | Hosts: 1 | RPS: 126 | Matched: 2 | Errors: 1175 | Requests: 4435/13859 (32%)
[0:00:40] | Templates: 8429 | Hosts: 1 | RPS: 129 | Matched: 2 | Errors: 1177 | Requests: 5170/13859 (37%)
[0:00:45] | Templates: 8429 | Hosts: 1 | RPS: 135 | Matched: 2 | Errors: 1576 | Requests: 6109/13859 (44%)
[0:00:50] | Templates: 8429 | Hosts: 1 | RPS: 137 | Matched: 2 | Errors: 1803 | Requests: 6888/13859 (49%)
[tls-version] [ssl] [info] telsa.com:443 ["tls12"]
[INF] Skipped telsa.com:80 from target list as found unresponsive 30 times
[0:00:55] | Templates: 8429 | Hosts: 1 | RPS: 131 | Matched: 3 | Errors: 1814 | Requests: 7233/13859 (52%)
[0:01:00] | Templates: 8429 | Hosts: 1 | RPS: 125 | Matched: 3 | Errors: 1945 | Requests: 7533/13859 (54%)
[0:01:05] | Templates: 8429 | Hosts: 1 | RPS: 117 | Matched: 3 | Errors: 1945 | Requests: 7657/13859 (55%)
[0:01:10] | Templates: 8429 | Hosts: 1 | RPS: 111 | Matched: 3 | Errors: 1946 | Requests: 7800/13859 (56%)
[0:01:15] | Templates: 8429 | Hosts: 1 | RPS: 105 | Matched: 3 | Errors: 1946 | Requests: 7946/13859 (57%)
[0:01:20] | Templates: 8429 | Hosts: 1 | RPS: 101 | Matched: 3 | Errors: 1946 | Requests: 8082/13859 (58%)
[0:01:25] | Templates: 8429 | Hosts: 1 | RPS: 96 | Matched: 3 | Errors: 1946 | Requests: 8191/13859 (59%)
[0:01:30] | Templates: 8429 | Hosts: 1 | RPS: 92 | Matched: 3 | Errors: 1946 | Requests: 8321/13859 (60%)
[0:01:35] | Templates: 8429 | Hosts: 1 | RPS: 88 | Matched: 3 | Errors: 1946 | Requests: 8385/13859 (60%)
[0:01:40] | Templates: 8429 | Hosts: 1 | RPS: 83 | Matched: 3 | Errors: 1946 | Requests: 8385/13859 (60%)
[0:01:43] | Templates: 8429 | Hosts: 1 | RPS: 81 | Matched: 3 | Errors: 1946 | Requests: 8385/13859 (60%)

Relevant log output

No response

Environment

- OS: All
- Nuclei: >v3.2.x 
- Go: 1.22

Anything else?

• https://github.com/projectdiscovery/fastdialer/pull/301

[Sh4d0wHunt3rX]

Hey, thanks for this, I also wanted to add that if I select specific directory for templates, let's say "http" , then there won't be any errors.

[tarunKoyalwar]

thanks for feedback @amiremami , @dwisiswant0 we should also make sure that host skipping happens on address ( i.e host+port) and not on just hostname/ip ( from above snapshot it looks like when running tcp/js templates errors of those ports ( 22 etc) are being counted towards that of ip/host )

we did add support for this a while ago ^ but something might have changed in fastdialer logic or here in nuclei

[xhzeem]

The current implementation of the max-host-error mechanism needs to be reconsidered. During heavy scanning, it's common to experience up to 30 dropped requests, but skipping the target simply because 30 requests fail while sending 13,000 requests doesn’t make sense. The target should only be skipped if none of the requests are succeeding, or if 30 consecutive requests fail without a response. Otherwise, it leads to prematurely skipping targets before they are fully tested.

I've been using Nuclei for a while and noticed that I wasn’t getting meaningful results until I debugged this issue. I discovered that no target was being fully scanned due to this flawed error-handling logic. I suggest a different approach to handling target skipping, or at least give more control over this.

---

Update: I'm using -no-mhe or -mhe 300 until a better fix is implemented (To be able to differ between a non-responsding target from a target that fails sometimes)