search

projectdiscovery / httpx

httpx is a fast and multi-purpose HTTP toolkit that allows running multiple probes using the retryablehttp library.
https://docs.projectdiscovery.io/tools/httpx
MIT License
7.53k stars 822 forks source link

Timeout response when using a VPN (only few hosts) #1840

Closed AlessandroZ closed 1 month ago

AlessandroZ commented 2 months ago

Hello,

Sorry to open a new issue again, however, I had this problem various time without being able to debug it. Maybe it's a well known issue.

When I use a VPN, sometimes I have a timeout response on certain host. 

> httpx -u scanme.sh -j -irh -debug -v

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/

        projectdiscovery.io

[INF] Current httpx version v1.6.6 (latest)
[INF] Dumped HTTP request for https://scanme.sh

GET / HTTP/1.1
Host: scanme.sh
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36 Edg/96.0.1054.34
Accept-Charset: utf-8
Accept-Encoding: gzip

[INF] Dumped HTTP request for http://scanme.sh

GET / HTTP/1.1
Host: scanme.sh
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36
Accept-Charset: utf-8
Accept-Encoding: gzip

[DBG] Failed 'http://scanme.sh': GET http://scanme.sh giving up after 1 attempts: Get "http://scanme.sh": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Whereas on the same host, using the same VPN, httprobe give me the good response.

> echo "scanme.sh" | httprobe 
https://scanme.sh
http://scanme.sh

When I connect to my vps and use httpx on the same domain, this time, it works. 

> httpx -u scanme.sh -silent
https://scanme.sh

I'm sorry to not beeing able to give you more information, I can close this issue if you cannot reproduce it. It does not happen all the time but I faced this issue multiple times (it's not a recent problem). I checked it doing diff between httprobe and httpx results. I cannot explain why this happens on certain domains only and all the time httprobe does not miss it, so the way it does request should be different.

Have a nice day.

dogancanbakir commented 2 months ago

Using the -timeout flag, could you retry it with a different timeout value? The default is 10s

AlessandroZ commented 2 months ago

When I faced this problem, I tried changing these parameter but without success. The best way should be to build manually to check exactly where the code fails but right now, I don't have enough time to do it. Sorry.

So same error as you could see:

> httpx -u scanme.sh -j -irh -debug -v -timeout 50

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/

        projectdiscovery.io

[INF] Current httpx version v1.6.6 (latest)
[INF] Dumped HTTP request for https://scanme.sh

GET / HTTP/1.1
Host: scanme.sh
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0
Accept-Charset: utf-8
Accept-Encoding: gzip

[INF] Dumped HTTP request for http://scanme.sh

GET / HTTP/1.1
Host: scanme.sh
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:70.0) Gecko/20100101 Firefox/70.0
Accept-Charset: utf-8
Accept-Encoding: gzip

[DBG] Failed 'http://scanme.sh': GET http://scanme.sh giving up after 1 attempts: Get "http://scanme.sh": [address=scanme.sh:80] context deadline exceeded
dogancanbakir commented 1 month ago

Closing this: I was able to repro the issue with the VPN gateways, which I suspect to be a bit problematic or sluggish. Therefore, it appears that we are left with no immediate solution.