projectdiscovery / httpx

httpx is a fast and multi-purpose HTTP toolkit that allows running multiple probes using the retryablehttp library.
https://docs.projectdiscovery.io/tools/httpx
MIT License
7.53k stars 822 forks source link

ASN Option not working even when PDCP API Key Auth is specified in Httpx #1884

Closed mr-pmillz closed 1 month ago

mr-pmillz commented 1 month ago

httpx version:

v1.6.8

Current Behavior:

Even with PdcpAuth set with the PDCP_API_KEY and/or having the env var set, when using the ASN option, a bunch of the following missing asnmap-api key errors get thrown:

[asnmap-api] missing or invalid api key (get free api key & configure it from https://cloud.projectdiscovery.io/?ref=api_key)
[asnmap-api] missing or invalid api key (get free api key & configure it from https://cloud.projectdiscovery.io/?ref=api_key)
[asnmap-api] missing or invalid api key (get free api key & configure it from https://cloud.projectdiscovery.io/?ref=api_key)

Expected Behavior:

I expect to only have to specify the PdcpAuth: pdcpAuth or by setting the PDCP_API_KEY env var once to not see a bunch of asnmap errors and be able to get the ASN data as well. How to fix this?

Steps To Reproduce:

I'm running the latest httpx version v1.6.8 as a go library via the following code:

func WriteLines(lines []string, path string) error {
    file, err := os.Create(path)
    if err != nil {
        return err
    }
    defer file.Close()

    w := bufio.NewWriter(file)
    for _, line := range lines {
        if len(line) > 0 {
            _, _ = fmt.Fprintln(w, line)
        }
    }
    return w.Flush()
}

// runHTTPX runs httpx against a slice of hosts or urls ...
func runHTTPX(targetHosts []string, outputDir, userAgent, pdcpAuth string) (string, error) {
    httpxOutputDir := fmt.Sprintf("%s/httpx", outputDir)
    if err := os.MkdirAll(httpxOutputDir, 0750); err != nil {
        return "", err
    }
    // write targets to file
    targetHostsFile := fmt.Sprintf("%s/all-recon-scope-targets.txt", httpxOutputDir)
    if err := WriteLines(targetHosts, targetHostsFile); err != nil {
        return "", err
    }

    // set PDCP_API env vars
    if err := os.Setenv("PDCP_API_KEY", pdcpAuth); err != nil {
        return "", err
    }
    // default api server
    if err := os.Setenv("PDCP_API_SERVER", "https://api.projectdiscovery.io"); err != nil {
        return "", err
    }
    // default asnmap server url
    if err := os.Setenv("SERVER_URL", "https://asn.projectdiscovery.io/"); err != nil {
        return "", err
    }

    httpxCSVOutputFilePath := fmt.Sprintf("%s/httpx-output.csv", httpxOutputDir)
    httpxOutputFileName := fmt.Sprintf("%s/httpx-output", httpxOutputDir)
    options := runner.Options{
        Methods:             "GET",
        InputFile:           targetHostsFile,
        CustomHeaders:       []string{fmt.Sprintf("User-Agent: %s", userAgent)},
        Output:              httpxOutputFileName,
        OutputAll:           true,
        Threads:             25,
        Jarm:                true,
        Asn:                 true,
        FollowRedirects:     true,
        FollowHostRedirects: true,
        PdcpAuth:            pdcpAuth, // This is theoretically required now in order to obtain ASN's via ASNmap. PdcpAuth is a DynamicVar.
        AssetUpload:         false,
        Location:            true,
        TechDetect:          true,
        CSVOutput:           true,
        JSONOutput:          true,
        OutputCDN:           "true", // OutputCDN changed from a BoolVar to a DynamicVar of type string. Defaults to // "true"
        OutputIP:            true,
        ExtractTitle:        true,
        StatusCode:          true,
        Debug:               false,
        DebugResponse:       false,
        DebugRequests:       false,
        NoColor:             false,
        ContentLength:       true,
        StoreChain:          false,
        StoreResponse:       false,
        OutputServerHeader:  true,
        OutputContentType:   true,
        OutputMethod:        true,
        OutputResponseTime:  true,
        OutputCName:         true,
        OutputLinesCount:    true,
        OutputWordsCount:    true,
        Timeout:             60,
    }
    if err := options.ValidateOptions(); err != nil {
        return "", err
    }

    httpxRunner, err := runner.New(&options)
    if err != nil {
        return "", err
    }
    defer httpxRunner.Close()
    httpxRunner.RunEnumeration()

    return httpxCSVOutputFilePath, nil
}

Anything else:

if I comment out ASN in the runner.Options{} struct, then all is well and I don't see the asnmap-:api error a bunch:

// Asn:                 true,
mr-pmillz commented 1 month ago

What's interesting is that the -asn flag works on the cli, but when run as a library, for some reason the env and/or the PdcpAuth are not respecting the api key for asnmap. Am not sure if this is related to goland test environment or what.

Oh wait, wow, nevermind. This is a false positive. I needed to set the env variables in Goland's debugger config. Interesting that GoLand does not respect env vars set via os.Setenv() in the debugger / test environments.

Closing issue. Sorry about that. Cheers 🍻