Open Mzack9999 opened 6 months ago
due to recent observations with Marshal/Unmarshal Overhead of storing data in hmap we can also add support for batch / chunked loading of targets (ex: 10k chunk size) thereby eliminating need for structured storage using hmap
Note: while this is negligible in host-spray mode it is significant in template-spray mode
with recent refactor of input/targets ( consolidation of list and other formats like openapi,jsonl etc) using appropriate interfaces and implementations . chunked loading / processing can be properly abstracted without a major refactor (see: https://github.com/projectdiscovery/nuclei/pull/4477)
Please describe your feature request:
As of now nuclei uses two naive strategies for scanning:
A third modality called
auto
was left out for future improvements, at current time it fall back to template spray. The current task is about implementing a smart planner, similar to the one of postgres to elaborate an execution plan:Cost Function
Identify a generic cost function that keeps into account fixed costs with dynamic costs and various boost factors. Listing a few potential metrics:
max-requests
, as request should be more tied to the protocol level implementation, for example 1 "request" in js might imply multiple network requests within the engine and it's currently equivalent in terms of cost to a "request" in network templates)As a start the cost function can be expressed in it's simplest form as $
C(x) = \sum F + \sum V(x)
$ withx
being the template,F
the sum of based fixed costs (protocol type? etc) andV(X)
the sum of dynamic costs asY*X
with Y being an expression that turns into a numeric number the impact of the involved metric (eg for threadsnumber_of_threads * unitary_memory_cost_impact
)Runtime planner
Once the cost function has been defined and the parallelism limits are set by the user, the planner should organize the execution into chunks that have a cost floating between
wanted_avg_cost +/- tolerance