[Bug] Httpx integration fails on templates using raw http

[xavier-gerondeau-stoik]

Nuclei version:

v2.9.5

Current Behavior:

I'm using the nuclei go lib to run a scan through GO code on an hostname. Httpx is automatically used to probe the host for web services in order to run HTTP based templates. However, certain templates, specifically those utilizing the http raw methods, fail to work. For instance, CVE-2010-1081 functions correctly, but CVE-2021-24947 does not. The error message is:

{"level":"WRN","msg":"[CVE-2021-24947] Could not execute request for testdomain.com: [raw:RUNTIME] failed to create request with url ://testdomain.com/wp-login.php got [:RUNTIME] failed to parse url <- parse \"://testdomain.com/wp-login.php\": missing protocol scheme <- [:RUNTIME] failed to parse url <- parse \"://testdomain.com/wp-login.php\": missing protocol scheme","timestamp":"2023-05-31T16:46:39+0000"}

It seems that the protocol scheme is not appended properly to the queried URL on templates using raw http queries.

Expected Behavior:

When running nuclei on an hostname, I expect httpx to be used to identified all web services in order to run HTTP based templates.

Steps To Reproduce:

The code below is an almost exact replica of the example here https://github.com/projectdiscovery/nuclei/blob/main/v2/examples/simple.go with small modifications on the used templates and gologger configuration:

func main() {
    gologger.DefaultLogger.SetMaxLevel(levels.LevelVerbose)
    gologger.DefaultLogger.SetFormatter(&formatter.JSON{})
    cache := hosterrorscache.New(30, hosterrorscache.DefaultMaxHostsCount, nil)
    defer cache.Close()

    mockProgress := &testutils.MockProgressClient{}
    reportingClient, _ := reporting.New(&reporting.Options{}, "")
    defer reportingClient.Close()

    outputWriter := testutils.NewMockOutputWriter()
    outputWriter.WriteCallback = func(event *output.ResultEvent) {
        fmt.Printf("Got Result: %v\n", event)
    }

    defaultOpts := types.DefaultOptions()
    protocolstate.Init(defaultOpts)
    protocolinit.Init(defaultOpts)

    defaultOpts.Templates = goflags.StringSlice{"http/cves"}

    interactOpts := interactsh.DefaultOptions(outputWriter, reportingClient, mockProgress)
    interactClient, err := interactsh.New(interactOpts)
    if err != nil {
        log.Fatalf("Could not create interact client: %s\n", err)
    }
    defer interactClient.Close()

    home, _ := os.UserHomeDir()
    catalog := disk.NewCatalog(path.Join(home, "nuclei-templates"))
    executerOpts := protocols.ExecuterOptions{
        Output:          outputWriter,
        Options:         defaultOpts,
        Progress:        mockProgress,
        Catalog:         catalog,
        IssuesClient:    reportingClient,
        RateLimiter:     ratelimit.New(context.Background(), 150, time.Second),
        Interactsh:      interactClient,
        HostErrorsCache: cache,
        Colorizer:       aurora.NewAurora(true),
        ResumeCfg:       types.NewResumeCfg(),
    }
    engine := core.New(defaultOpts)
    engine.SetExecuterOptions(executerOpts)

    workflowLoader, err := parsers.NewLoader(&executerOpts)
    if err != nil {
        log.Fatalf("Could not create workflow loader: %s\n", err)
    }
    executerOpts.WorkflowLoader = workflowLoader

    store, err := loader.New(loader.NewConfig(defaultOpts, catalog, executerOpts))
    if err != nil {
        log.Fatalf("Could not create loader client: %s\n", err)
    }
    store.Load()

    inputArgs := []*contextargs.MetaInput{{Input: "docs.harckerone.com"}}

    input := &inputs.SimpleInputProvider{Inputs: inputArgs}
    _ = engine.Execute(store.Templates(), input)
    engine.WorkPool().Wait() // Wait for the scan to finish
}

When run, you should see outputs similar to:

{"level":"WRN","msg":"[CVE-2021-44848] Could not execute request for docs.hackerone.com: [raw:RUNTIME] failed to create request with url ://docs.hackerone.com/changePassword?username=administrator got [:RUNTIME] failed to parse url <- parse \"://docs.hackerone.com/changePassword\": missing protocol scheme <- [:RUNTIME] failed to parse url <- parse \"://docs.hackerone.com/changePassword\": missing protocol scheme","timestamp":"2023-06-01T07:37:39+0000"}
{"level":"WRN","msg":"[CVE-2021-46005] Could not execute request for docs.hackerone.com: [raw:RUNTIME] failed to create request with url ://docs.hackerone.com/admin/ got [:RUNTIME] failed to parse url <- parse \"://docs.hackerone.com/admin/\": missing protocol scheme <- [:RUNTIME] failed to parse url <- parse \"://docs.hackerone.com/admin/\": missing protocol scheme","timestamp":"2023-06-01T07:37:39+0000"}
{"level":"VER","msg":"[CVE-2018-14574] Sent HTTP request to https://docs.hackerone.com//www.interact.sh","timestamp":"2023-06-01T07:37:39+0000"}
{"level":"WRN","msg":"[CVE-2022-2467] Could not execute request for docs.hackerone.com: [raw:RUNTIME] failed to create request with url ://docs.hackerone.com/login.php got [:RUNTIME] failed to parse url <- parse \"://docs.hackerone.com/login.php\": missing protocol scheme <- [:RUNTIME] failed to parse url <- parse \"://docs.hackerone.com/login.php\": missing protocol scheme","timestamp":"2023-06-01T07:37:39+0000"}
{"level":"VER","msg":"[CVE-2013-5979] Sent HTTP request to https://docs.hackerone.com/index.php?_=1355714673828&ajax=true&p=../../../../../../../../../../../../../../../../etc/passwd%00index&q=About","timestamp":"2023-06-01T07:37:39+0000"}
{"level":"WRN","msg":"[CVE-2020-23972] Could not execute request for docs.hackerone.com: [raw:RUNTIME] failed to create request with url ://docs.hackerone.com/index.php?controller=editlieux&option=com_gmapfp&task=upload_image&tmpl=component got [:RUNTIME] failed to parse url <- parse \"://docs.hackerone.com/index.php\": missing protocol scheme <- [:RUNTIME] failed to parse url <- parse \"://docs.hackerone.com/index.php\": missing protocol scheme","timestamp":"2023-06-01T07:37:39+0000"}

Anything else:

While this issue is similar to https://github.com/projectdiscovery/nuclei/issues/454, this is not the same issue. It seems that at the time, httpx was not used to probe hostname for web services (which is now the case), the current issue is about httpx integration with web based templates.

[ehsandeep]

@xavier-gerondeau-stoik this is now fixed with latest release - https://github.com/projectdiscovery/nuclei/releases/tag/v2.9.10

[gnuletik]

Hi @ehsandeep,

Thanks for the fix!

It seems that the issue remains on v2.9.10. After updating the code to call SetWithProbe, nuclei logs the following warnings:

{"level":"WRN","msg":"[CVE-2021-24947] Could not execute request for google.com: [raw:RUNTIME] failed to create request with url ://google.com/wp-login.php got [:RUNTIME] failed to parse url <- parse \"://google.com/wp-login.php\": missing protocol scheme <- [:RUNTIME] failed to parse url <- parse \"://google.com/wp-login.php\": missing protocol scheme","timestamp":"2023-08-09T11:55:19+0000"}
{"level":"WRN","msg":"[CVE-2021-24947] Could not make http request for https://google.com: unresolved variables found: username,password","timestamp":"2023-08-09T11:55:19+0000"}

Do you know why ?

Thanks!

[gnuletik]

Also, it seems that adding the following

+       input := &inputs.SimpleInputProvider{Inputs: inputArgs}
+
+       httpxOptions := httpx.DefaultOptions
+       httpxOptions.Timeout = 5 * time.Second
+       httpxClient, err := httpx.New(&httpxOptions)
+       if err != nil {
+               return fmt.Errorf("httpx.New: %w", err)
+       }
+       input.SetWithProbe(opts.target, httpxClient)

Like you added on the updated example in the PR.

It leads to a duplicate call of outputWriter.WriteCallback with the same finding.