search

projectdiscovery / httpx

httpx is a fast and multi-purpose HTTP toolkit that allows running multiple probes using the retryablehttp library.
https://docs.projectdiscovery.io/tools/httpx
MIT License
7.1k stars 785 forks source link

Proxy in k8s: short DNS name not resolved properly #1375

Open ocervell opened 9 months ago

ocervell commented 9 months ago

I am running a Privoxy proxy and a client using httpx within kubernetes, trying to have httpx use that proxy when making requests.

Full DNS name Running with the full k8s DNS name tor-privoxy.default.svc.cluster.local works:

$ httpx -silent -u https://www.google.com -proxy http://tor-privoxy.default.svc.cluster.local:8118
https://www.google.com [Google Web Server,HSTS,HTTP/3]

Service reduced DNS name Running with the reduced k8s DNS name tor-privoxy as proxy doesn't seem to work (no results):

$ httpx -silent -u https://www.google.com -json -proxy http://tor-privoxy:8118

Seems like httpx's DNS resolution doesn't work all the way but I'm not getting any error from the output. Name resolution using nslookup does work:

$ nslookup tor-privoxy
Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   tor-privoxy.default.svc.cluster.local
Address: 10.110.18.226

Service IP Running with the proxy's IP works fine as well:

$ httpx -silent -u https://www.google.com -json -proxy http://10.110.18.226:8118
https://www.google.com [HTTP/3]

The resolv.conf on k8s is like:

cat /etc/resolv.conf 
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local mshome.net
options ndots:5

I wasn't sure if the resolvers flag (-r) worked for proxy DNS, but I've tried passing it -r 10.96.0.10 to no effect: are we bypassing default system DNS resolution ?

Mzack9999 commented 3 weeks ago

@ocervell Apologies for the very late reply. Internally we use fastdialer and by default we parse the content of the hosts file along with retrying with system resolvers on failure. Are you still able to reproduce the issue?