rockdaboot / wget2

The successor of GNU Wget. Contributions preferred at https://gitlab.com/gnuwget/wget2. But accepted here as well 😍
GNU Lesser General Public License v3.0
542 stars 74 forks source link

wget2 stopped working with proxychains-ng (wget/curl still work) #296

Open koedx opened 11 months ago

koedx commented 11 months ago

I am trying to understand why wget2 fails to connect when being run through proxychains-ng. Prior versions of wget2 (& current versions of curl & wget) work as expected.

Running proxychains4 wget2 -d --no-dns-cache --no-local-db --no-check-certificate --progress=none -O - "http://google.com" produces:

[proxychains] config file found: /data/data/com.termux/files/home/.proxychains/proxychains.conf
[proxychains] preloading /data/data/com.termux/files/usr/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.16
17.040123.681 name=dns-cache value=--no-local-db invert=1
17.040123.682 name=local-db value=--no-check-certificate invert=1
17.040123.682 name=check-certificate value=--progress=none invert=1
17.040123.682 name=progress value=none invert=0
17.040123.682 name=dns-cache value=--no-local-db invert=1
17.040123.682 name=local-db value=--no-check-certificate invert=1
17.040123.682 name=check-certificate value=--progress=none invert=1
17.040123.682 name=progress value=none invert=0
17.040123.682 Local URI encoding = 'UTF-8'
17.040123.682 Input URI encoding = 'UTF-8'
17.040123.682 set_exit_status(0)
17.040123.682 *url =
17.040123.682 *3 http://google.com                               17.040123.682 host_add_job: job fname (null)
17.040123.682 host_add_job: 0xae490a88 http://google.com
17.040123.682 host_add_job: qsize 1 host-qsize=1
17.040123.682 queue_size: qsize=1
17.040123.682 queue_size: qsize=1
17.040123.682 queue_size: qsize=1
17.040123.683 [0] action=1 pending=0 host=0x0
17.040123.683 dequeue job http://google.com
17.040123.683 resolving google.com:80...
17.040123.684 has 224.0.0.1:80
17.040123.684 closing connection
Failed to connect: General error
17.040123.684 host_increase_failure: google.com failures=1
17.040123.684 [0] action=3 pending=1 host=0xae4a0168
17.040123.684 released job http://google.com
17.040123.684 [0] action=1 pending=0 host=0x0
17.040123.684 host google.com is paused 1000ms
17.040123.684 main: wake up
17.040123.684 queue_size: qsize=1
17.040124.684 [0] action=1 pending=0 host=0x0
17.040124.684 dequeue job http://google.com
17.040124.684 resolving google.com:80...
17.040124.686 has 224.0.0.1:80
17.040124.686 closing connection
Failed to connect: General error

wget2 acquires the "dummy" ip from proxychains-ng (the 224.x.x x address) as expected (the "has" output above). At this point things seem to go wrong with wget2 (wget or curl connect to that ip & successfully download the requested page; wget2 always fails... I am using google in the posted example, but the issue exists no matter what url I attempt to connect to).

When I use wget2 w/o proxychains, after the provided host url is resolved to an IP ("has" x.x.x.x), wget2 connects to said ip (& outputs "try" x.x.x.x to indicate this).

Why isn't wget2 trying to connect to the provided ip when invoked thru proxychains-ng??

Any guidance in troubleshooting this issue would be much appreciated 🙏

rockdaboot commented 10 months ago

Just a wild guess, try it with --no-tcp-fastopen. Because wget2 tries to connect to that IP and sees this General error. On the other hand side, TFO didn't change in a long time.

Btw, what version of wget2 are you using?

rockdaboot commented 10 months ago

Maybe you can help me to reproduce. Currently, proxychain4 on Debian testing doesn't seem to work. I tried a bunch of proxies, adding with http ip port to /etc/proxychains4.conf.

All I ever get is a timeout

[proxychains] Strict chain  ...  <ip>:<port>  ...  timeout
koedx commented 10 months ago

wget2 -V:

GNU Wget2 2.1.0 - multithreaded metalink/file/website downloader

+digest +https +ssl/gnutls +ipv6 +iri +large-file -nls -ntlm -opie -psl -hsts +iconv +idn2 +zlib -lzma +brotlidec +zstd -bzip2 -lzip +http2 +gpgme

I tried running w/ --no-tcp-fast-open & the general error persists.

Regarding proxychains4 issues you mention, the "original" proxychains4 (although still in the repos) is unmaintained. The maintained version/package should be installed via apt-get proxychains-ng (v. 4.16, the GitHub repo for that package is linked below). Is this the proxychains4 you are running? (if not, installing the -ng package should address the timeout issues you experienced). I can confirm that the -ng package proxifies current versions of curl & wget w/o issue (for me).

https://github.com/rofl0r/proxychains-ng