Scan Planner

[Mzack9999]

Please describe your feature request:

As of now nuclei uses two naive strategies for scanning:

• Template-Spray: iterate all hosts vs same template
• Host-Spray: iterate all templates vs same host

A third modality called auto was left out for future improvements, at current time it fall back to template spray. The current task is about implementing a smart planner, similar to the one of postgres to elaborate an execution plan:

Cost Function

Identify a generic cost function that keeps into account fixed costs with dynamic costs and various boost factors. Listing a few potential metrics:

• Protocol Type as base cost (JS pooled/non-pooled implies will have a base cost higher than http with connection reuse)
• TLS (<TLS1.2 adds additional CPU + network activity than TLS1.3)
• Number of Threads (Will imply a spike in Memory/CPU)
• Number of estimated iterations (equivalent to the current named max-requests, as request should be more tied to the protocol level implementation, for example 1 "request" in js might imply multiple network requests within the engine and it's currently equivalent in terms of cost to a "request" in network templates)
• Uses interactsh
• Etc

As a start the cost function can be expressed in it's simplest form as $C(x) = \sum F + \sum V(x)$ with x being the template, F the sum of based fixed costs (protocol type? etc) and V(X) the sum of dynamic costs as Y*X with Y being an expression that turns into a numeric number the impact of the involved metric (eg for threads number_of_threads * unitary_memory_cost_impact)

Runtime planner

Once the cost function has been defined and the parallelism limits are set by the user, the planner should organize the execution into chunks that have a cost floating between wanted_avg_cost +/- tolerance

[tarunKoyalwar]

due to recent observations with Marshal/Unmarshal Overhead of storing data in hmap we can also add support for batch / chunked loading of targets (ex: 10k chunk size) thereby eliminating need for structured storage using hmap

Note: while this is negligible in host-spray mode it is significant in template-spray mode

with recent refactor of input/targets ( consolidation of list and other formats like openapi,jsonl etc) using appropriate interfaces and implementations . chunked loading / processing can be properly abstracted without a major refactor (see: https://github.com/projectdiscovery/nuclei/pull/4477)