projectdiscovery / httpx

httpx is a fast and multi-purpose HTTP toolkit that allows running multiple probes using the retryablehttp library.
https://docs.projectdiscovery.io/tools/httpx
MIT License
7.53k stars 822 forks source link

.host Property Occasionally Missing From Version 1.6.2 Onwards 2 #1815

Closed acamboe closed 1 month ago

acamboe commented 2 months ago

httpx version:

Working: v1.6.1 Broken: v1.6.2

Current Behavior:

The .host property is occasionally missing from the JSON output. The issue is the same problem described in #1753, but the underlying cause may be different. Although I'm not aware of all the causes where the issue arises, it can be commonly recreated by specify hosts with a port.

Expected Behavior:

The .host property should always be returned to identify the IP address queried.

Steps To Reproduce:

$ httpx -u 'http://google.co.uk:80,http://ebay.com:80' -j -silent | jq -c '{input,host}'
{"input":"http://google.co.uk:80","host":null}
{"input":"http://ebay.com:80","host":null}

But if you specify the same domain twice, removing the port from one:

$ httpx -u 'http://google.co.uk:80,http://google.co.uk' -j -silent | jq -c '{input,host}'
{"input":"http://google.co.uk","host":"172.217.16.227"}
{"input":"http://google.co.uk:80","host":"172.217.16.227"}

And mixed:

$ httpx -u 'http://google.co.uk:80,ebay.com' -j -silent | jq -c '{input,host}'
{"input":"ebay.com","host":"23.48.165.132"}
{"input":"http://google.co.uk:80","host":null}

Here's a unfiltered output:

$  httpx -u "http://google.co.uk:80" -j

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/

        projectdiscovery.io

[INF] Current httpx version v1.6.5 (latest)
{"timestamp":"2024-07-08T22:09:49.294748+01:00","port":"80","url":"http://google.co.uk:80","input":"http://google.co.uk:80","location":"http://www.google.co.uk/","title":"301 Moved","scheme":"http","webserver":"gws","content_type":"text/html","method":"GET","path":"/","time":"37.977542ms","a":["172.217.169.35"],"aaaa":["2a00:1450:4009:822::2003"],"tech":["Google Web Server"],"words":9,"lines":6,"status_code":301,"content_length":221,"failed":false,"knowledgebase":{"PageType":"error","pHash":0},"resolvers":["1.1.1.1:53","1.0.0.1:53"]}