Nuclei SDK passive mode does not work

tomdottom commented 3 months ago

Nuclei version:

v.3.2.9

Current Behavior:

The nuclei SDK is not currently able to configure an engine and/or load targets to run passive scans on save HTTP responses.

The current EnablePassive configuration option does not appear to be sufficient to make an engine run passive scans.

Expected Behavior:

The SDK API and documentation should provide a way to run a passive scan against stored HTTP responses.

Steps To Reproduce:

Confirm that passive nuclei passive mode works from the cli


$ httpx-toolkit -sr -u https://juice-shop.herokuapp.com

__    __  __       _  __
/ /_  / /_/ /_____ | |/ /
/ __ \/ __/ __/ __ \|   /
/ / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
         /_/

    projectdiscovery.io

[INF] Current httpx version v1.6.4 (outdated) https://juice-shop.herokuapp.com

$ nuclei -passive -target output/response/juice-shop.herokuapp.com/

                 __     _

__ ___/ / (_) / \/ / / / / / \/ / / / / / // / // / / / // //_,/\/_/__// v3.2.9

    projectdiscovery.io

[INF] Current nuclei version: v3.2.9 (latest) [INF] Current nuclei-templates version: v9.9.1 (latest) [WRN] Scan results upload to cloud is disabled. [INF] New templates added in latest release: 75 [INF] Templates loaded for current scan: 1171 [INF] Executing 1171 signed templates from projectdiscovery/nuclei-templates [INF] Targets loaded for current scan: 1 [addeventlistener-detect] [] [info] output/response/juice-shop.herokuapp.com/47a267551a6bb823709fdfb9cf2092d8e282e7ba.txt [x-recruiting-header] [] [info] output/response/juice-shop.herokuapp.com/47a267551a6bb823709fdfb9cf2092d8e282e7ba.txt [cowboy-detect] [] [info] output/response/juice-shop.herokuapp.com/47a267551a6bb823709fdfb9cf2092d8e282e7ba.txt [fingerprinthub-web-fingerprints:qm-system] [] [info] output/response/juice-shop.herokuapp.com/47a267551a6bb823709fdfb9cf2092d8e282e7ba.txt [owasp-juice-shop-detect] [] [info] output/response/juice-shop.herokuapp.com/47a267551a6bb823709fdfb9cf2092d8e282e7ba.txt


2. Try and run the same scan using the SDK

```golang
package main

import (
    "fmt"

    nuclei "github.com/projectdiscovery/nuclei/v3/lib"
    output "github.com/projectdiscovery/nuclei/v3/pkg/output"
)

func handleResultEvent(event *output.ResultEvent) {
    fmt.Printf("%#v/n", event)
}

func main() {
    fmt.Println("Starting new engine")

    ne, err := nuclei.NewNucleiEngine(
        nuclei.EnablePassiveMode(),
    )
    if err != nil {
        panic(err)
    }
    defer ne.Close()

    // This should probably be done in EnablePassiveMode
    ne.Options().DisableHTTPProbe = true

    ne.LoadTargetsWithHttpData(
        "./output/response/localhost_3000/d167406719b748a6d5d052538558eed53735a389.txt",
        "",
    )
    err = ne.ExecuteWithCallback(handleResultEvent)
    if err != nil {
        panic(err)
    }
}

$ go run main.go 
Starting new engine
panic: [:RUNTIME] No targets available

goroutine 1 [running]:
main.main()
    /tmp/tmp.ruOdyy29rH/main.go:35 +0x153
exit status 2

Anything else:

It should be noted the LoadTargetsWithHttpData defaults to a ListInputProvider when provided a .txt. file, and there is currently no such thing as a TxtResponseInputProvider (which may be a missing object). I was not able to prevent the ListInputProvider from spltting the http response data by lines.
I also attempted to set the executor ProtocolType = OfflineHTTPProtocol (both on executer and engine options), but this did not work. It may be that this has to be set before the engine is created.

dogancanbakir commented 3 months ago

Thanks for letting us know! Please use LoadTargets func instead. LoadTargetsWithHttpData func currently supports burp xml,openapi,swagger,proxify json formats, and there is no default when you use it -it always checks/asserts the value provided.

Modified example

package main

import (
    "context"
    "fmt"

    nuclei "github.com/projectdiscovery/nuclei/v3/lib"
    output "github.com/projectdiscovery/nuclei/v3/pkg/output"
)

func handleResultEvent(event *output.ResultEvent) {
    fmt.Printf("%#v/n", event)
}

func main() {
    ne, err := nuclei.NewNucleiEngineCtx(
        context.Background(),
        nuclei.EnablePassiveMode(),
    )
    if err != nil {
        panic(err)
    }
    defer ne.Close()

    ne.LoadTargets(
        []string{"PATH_TO_DIR/output/response/hackerone.com/23014af90b981cfdf1f0f1cedae02baf06b5d3c6.txt"},
        false,
    )

    err = ne.ExecuteWithCallback(handleResultEvent)
    if err != nil {
        panic(err)
    }
}

I've raised a PR for the following part:

// This should probably be done in EnablePassiveMode
    ne.Options().DisableHTTPProbe = true

tomdottom commented 3 months ago

That example appears to work :+1:

Is there any way to just pass the response string to the Nuclei engine?

dogancanbakir commented 3 months ago

These are the only options available currently. If you want, you can submit a feature request.

projectdiscovery / nuclei