Googler crashes when the --tld value is not set explicitly

[Battery233]

My environment 💻

googler: v 4.3.2 (latest) py: v.3.9.5 MacOS 11.4 & Zsh locale: All UTF8 for googler Also, proxy is enabled in my environment: export http_proxy=http://127.0.0.1:12639 export https_proxy=http://127.0.0.1:12639

Bug report 🐞

First, I execute ➜ ~ googler -d --tld=hk test:

It works fine ☑️ log📃⬇️ [DEBUG] googler version 4.3.2 [DEBUG] Python version 3.9.5 [DEBUG] Platform: macOS-11.4-x86_64-i386-64bit [DEBUG] Connecting to proxy server 127.0.0.1:12639 [DEBUG] Tunnelling to host www.google.com.hk [DEBUG] Opened socket to 127.0.0.1:12639 [DEBUG] new_connection completed in 0.106s [DEBUG] Fetching URL /search?ie=UTF-8&oe=UTF-8&q=test&sei=hMypT5SJTLC9hXt1fgve4w [DEBUG] Cookie: 1P_JAR=2021-05-27-04 [DEBUG] fetch_page completed in 1.856s [DEBUG] Response body written to '/var/folders/v4/6p2nxvvn2g5f6gzb__pfq3dh0000gn/T/googler-response-ol0upico.html'. [DEBUG] parse completed in 0.032s

Then I execute ➜ googler -d test:

It crashed 💥 It seems that two redirecting are involved in this case and the second one failed.

Since it is redirecting from google.com.hk to google.com.hk, the statement at https://github.com/jarun/googler/blob/71c89fdee6ac96f18b354c2ec255a6e68d5cdfc1/googler#L2269 is not executed.

This crash is reproducible only when the local proxy is enabled.

log📃⬇️ [DEBUG] googler version 4.3.2 [DEBUG] Python version 3.9.5 [DEBUG] Platform: macOS-11.4-x86_64-i386-64bit [DEBUG] Connecting to proxy server 127.0.0.1:12639 [DEBUG] Tunnelling to host www.google.com [DEBUG] Opened socket to 127.0.0.1:12639 [DEBUG] new_connection completed in 0.059s [DEBUG] Fetching URL /search?ie=UTF-8&oe=UTF-8&q=test&sei=SolO8XrVTM6iz2j83yZPPA [DEBUG] Cookie: 1P_JAR=2021-05-27-03 [DEBUG] Redirecting to URL https://www.google.com.hk/url?sa=p&hl=zh-CN&pref=hkredirect&pval=yes&q=https://www.google.com.hk/search%3Fie%3DUTF-8%26oe%3DUTF-8%26q%3Dtest%26sei%3DSolO8XrVTM6iz2j83yZPPA&ust=1622088028285085&usg=AOvVaw1wuZNm0AD53XvofdwAjoXR [DEBUG] Connecting to proxy server 127.0.0.1:12639 [DEBUG] Tunnelling to host www.google.com.hk [DEBUG] Opened socket to 127.0.0.1:12639 [DEBUG] new_connection completed in 0.088s [DEBUG] Fetching URL /url?sa=p&hl=zh-CN&pref=hkredirect&pval=yes&q=https://www.google.com.hk/search%3Fie%3DUTF-8%26oe%3DUTF-8%26q%3Dtest%26sei%3DSolO8XrVTM6iz2j83yZPPA&ust=1622088028285085&usg=AOvVaw1wuZNm0AD53XvofdwAjoXR [DEBUG] Redirecting to URL https://www.google.com.hk/search?ie=UTF-8&oe=UTF-8&q=test&sei=SolO8XrVTM6iz2j83yZPPA [DEBUG] Fetching URL /search?ie=UTF-8&oe=UTF-8&q=test&sei=SolO8XrVTM6iz2j83yZPPA [DEBUG] Got exception: Request-sent. Traceback (most recent call last): File "/usr/local/bin/googler", line 2273, in _redirect self._raw_get(relurl) File "/usr/local/bin/googler", line 2303, in _raw_get self._resp = self._conn.getresponse() File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1335, in getresponse raise ResponseNotReady(self.__state) http.client.ResponseNotReady: Request-sent

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/bin/googler", line 3819, in main() File "/usr/local/bin/googler", line 3808, in main repl.cmdloop() File "/usr/local/bin/googler", line 3168, in cmdloop self.fetch_and_display() File "/usr/local/bin/googler", line 2726, in enforced_method method(self, *args, kwargs) File "/usr/local/bin/googler", line 2974, in fetch_and_display self.fetch() File "/usr/local/bin/googler", line 2726, in enforced_method method(self, *args, *kwargs) File "/usr/local/bin/googler", line 2839, in fetch page = self._conn.fetch_page(self._google_url.relative()) File "/usr/local/bin/googler", line 1569, in wrapped ret = func(args, kwargs) File "/usr/local/bin/googler", line 2231, in fetch_page self._redirect(redirection_url) File "/usr/local/bin/googler", line 2276, in _redirect raise GoogleConnectionError("Failed to get '%s'." % url) main.GoogleConnectionError: Failed to get 'https://www.google.com.hk/search?ie=UTF-8&oe=UTF-8&q=test&sei=SolO8XrVTM6iz2j83yZPPA'.

FIX:

@ PR #403

[jarun]

Please try to fix it and raise a PR. I am unable to repro from my geo.

[jarun]

Just noticed your PR and merged. Thank you!