Suddenly stopped working

kurtericmiller commented 4 years ago

Greetings, Can ping duckduckgo.com (no replies but contactable) Can use duckduckgo.com from chrome

some debug output: PING google.com (172.217.20.110) 56(84) bytes of data. 64 bytes from ams17s01-in-f14.1e100.net (172.217.20.110): icmp_seq=1 ttl=117 time=39.3 ms 64 bytes from ams17s01-in-f14.1e100.net (172.217.20.110): icmp_seq=2 ttl=117 time=38.9 ms 64 bytes from ams17s01-in-f14.1e100.net (172.217.20.110): icmp_seq=3 ttl=117 time=39.2 ms 64 bytes from ams17s01-in-f14.1e100.net (172.217.20.110): icmp_seq=4 ttl=117 time=38.8 ms

--- google.com ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3005ms rtt min/avg/max/mdev = 38.834/39.043/39.297/0.202 ms [kmiller@KEM-B9]~:ping -c4 duckduckgo.com PING duckduckgo.com (40.114.177.156) 56(84) bytes of data.

--- duckduckgo.com ping statistics --- 4 packets transmitted, 0 received, 100% packet loss, time 3061ms

[kmiller@KEM-B9]~:ddgr -d ubuntu 20.04 [DEBUG] ddgr version 1.8.1 Python version 3.8.2 [DEBUG] q:ubuntu 20.04, region:us-en, page:0, curindex:1, safe:1 [DEBUG] nextParams: [DEBUG] proxy:{'https': None} [DEBUG] ua:ddgr/1.8.1 (textmode; Linux x86_64; 1024x768) [DEBUG] Response body written to '/tmp/ddgr-response-tixs8svt'. [DEBUG] Prev nextParams: [DEBUG] Next nextParams: No results. ddgr (? for help)

cat /tmp/ddgr-response-tixs8svt <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

DuckDuckGo

About DuckDuckGo

jarun commented 4 years ago

There is some change in the service. i can fetch the first page with the following patch:

diff --git a/ddgr b/ddgr
index 39510e6..647718e 100755
--- a/ddgr
+++ b/ddgr
@@ -705,7 +705,7 @@ class DdgConnection:
     """

     def __init__(self, proxy=None, ua=''):
-        self._u = 'https://duckduckgo.com/html'
+        self._u = 'https://html.duckduckgo.com/html'

         self._proxies = {
             'https': proxy if proxy is not None else (os.getenv('https_proxy')

However, fetching the next page still fails and I am not sure why. Please see if you can debug and figure this out. I can't spend time on this.

jarun commented 4 years ago

Other things that would fail are bangs, tab-completion, ~instant results~.

jetzerb commented 4 years ago

Yes, it must be due to a change in DDG's API. I get the same "No Results" message from the version packaged for Ubuntu 20.04

$ dpkg -l ddgr
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version                   Architecture Description
+++-==============-=========================-============-=================================
ii  ddgr           1.7+git20190928.bccdc92-2 all          DuckDuckGo from the terminal

$ ddgr -d Ubuntu 20.04
[DEBUG] ddgr version 1.7 Python version 3.8.2
[DEBUG] q:Ubuntu 20.04, region:us-en, page:0, curindex:1, safe:1
[DEBUG] nextParams:
[DEBUG] proxy:{'https': None}
[DEBUG] ua:ddgr/1.7 (textmode; Linux x86_64; 1024x768)
[DEBUG] Response body written to '/tmp/ddgr-response-sxfn32q0'.
[DEBUG] Prev nextParams: 
[DEBUG] Next nextParams: 
No results.
ddgr (? for help)

NOTE: When I run the latest code that includes the update referenced above, I have no problem navigating to the next or previous page, and bangs seem to work fine.

aalapd commented 4 years ago

Same problem here. $ ddgr hello No results. ddgr (? for help)

brandong954 commented 4 years ago

Same here

jarun commented 4 years ago

Guys, I know it's there. Unfortunately, I do not have the time to debug and fix every aspect of it. Someone has to pitch in and fix fetching next pages, bangs, auto-completion on tab.

jetzerb commented 4 years ago

@jarun: Could you expand on the troubles you're experiencing? I just tried a couple things now, using a freshly-cloned ddgr repo:

Next/Prev pages Sometimes work, but the responses coming back from DDG seem strange. The number of results varies. I imagine this is why n/p seems to work sometimes but not others:

$ rm /tmp/ddgr-response-*
$ for i in $(seq 10); do ddgr --np --debug microsoft; done >/dev/null 2>&1
$ grep -c 'result__a' /tmp/ddgr-response-*
/tmp/ddgr-response-0k1wqttu:19
/tmp/ddgr-response-39i1ed80:21
/tmp/ddgr-response-6461a4gc:21
/tmp/ddgr-response-b99mdhdr:21
/tmp/ddgr-response-ca181t92:20
/tmp/ddgr-response-dnwhc41t:21
/tmp/ddgr-response-eubap2vm:21
/tmp/ddgr-response-irh29qd7:21
/tmp/ddgr-response-uth25bvw:30
/tmp/ddgr-response-zgyjbck9:21

Bangs are working fine for me, both on the commandline (ddgr '!w Galilean moons') and at the ddgr interactive menu.

auto-completion is also working for me on Ubuntu 20.04 running Bash:

$ ddgr --
--colors             --help               --num                --unsafe
--debug              --json               --proxy              --url-handler
--ducky              --nocolor            --reg                --version
--expand             --noprompt           --show-browser-logs  
--gb                 --noua               --site               
--gui-browser        --np                 --time               
$ ddgr --d
--debug  --ducky  
$ ddgr --ducky

(that last command auto-completed after I entered --du and hit tab)

jarun commented 4 years ago

Thanks for checking it out. In that case I think the only issue is next and prev. You can compare the data sent from firefox (for next) in developer mode with what ddgr sends. Please see if that gives a hint.

jarun commented 4 years ago

Sorry, I didn't mean option completion by auto-completion. in ddgr you could also complete possible search terms earlier. That is not working.

Bangs are also not working. Try the 4th example: https://github.com/jarun/ddgr#examples

jetzerb commented 4 years ago

I wasn't aware ddgr had search-term auto-completion. How did that work? Just type (a portion of) a word and hit tab? Where? On the commandline? At the interactive prompt?

Regarding bangs: I'm running inside a Linux Docker container without a GUI browser so I can't test your 4th example exactly, but both ddgr \!w hello world and ddgr '!w hello world' work for me. Lynx opens and displays the "Hello World!" page:

#
"Hello, World!" program - Wikipedia (p1 of 7)
   #Edit this page Wikipedia (en) Wikipedia Atom feed

"Hello, World!" program

   From Wikipedia, the free encyclopedia
     (Redirected from Hello world)
   Jump to navigation Jump to search
   "Hello World" redirects here. For other uses, see Hello World (disambiguation).
   Traditional beginners' computer program
. . . snip . . .

jarun commented 4 years ago

You are right! Bangs are working. I am on zsh and missed the \.

Just type (a portion of) a word and hit tab? Where? On the commandline? At the interactive prompt?

That's correct. At he commandline. And I see it is working in bash.

~$ ddgr he
headphones           hello                hello\ neighbor      hertz                
hearts               hello\ fresh         hemorrhagic\ stroke  hertz\ car\ sales

In fact I missed both these cases as I switched to zsh nowadays.

So we are back to only next and prev are not working.

jetzerb commented 4 years ago

Interesting...I tried tab completion at the commandline earlier and it didn't seem like it was working. Tried again now and discovered that I have to hit tab twice before it offers suggestions.

jarun commented 4 years ago

That's correct.

jarun commented 4 years ago

@cmonfortep noticed you actively contribute to the DDG android app project. Can you/refer someone who can help us with fetching the next and previous pages?

cmonfortep commented 4 years ago

Hi @jarun. I will try to ping someone from the backend team and see if they can comment here. 👍

jarun commented 4 years ago

Thank you so much!

muraii commented 4 years ago

For what it's worth, I'm using Ubuntu 18.04 which is capped at ddgr v1.2.something. The __init__ function was already using the html.duckduckgo.com/html URI but I was still getting the "No results" behavior. Since I'm not updating to 20.04 any time soon, I instead decided to keep a package-managed version of ddgr but not use Ubuntu's packaging.

Enter pipx. Now I've installed ddgr v.1.8.1, edited the __init__ to point to html.duckduckgo.com/html and everything works, even next and prev. Well, at least on search term "farts".

Is there any kind of debug information I can provide regarding getting next and prev working generally?

jarun commented 4 years ago

everything works, even next and prev

Not really. You are only viewing the results of the first fetch (first 30). Do next 3 times and then you'll see the next 30 results fetch fail.

Is there any kind of debug information I can provide regarding getting next and prev working generally?

I think fetching next page fails because there is some change required in the data we send to fetch the next page. The server is expecting something else. You can debug by comparing with the data Firefox sends (you cna see it in Web Developer -> Network).

Note: I am not in a position to work on this now. There's enough info on how/what to debug in this thread. Someone has to pick it up.

muraii commented 4 years ago

Quite right. Thanks for the clarification.

On Mon, Jul 13, 2020 at 5:24 PM Jana notifications@github.com wrote:

everything works, even next and prev

Not really. You are only viewing the results of the first fetch (first 30). Do next 3 times and then you'll see the next 30 results fetch fail.

Is there any kind of debug information I can provide regarding getting next and prev working generally?

I think fetching next page fails because there is some change in the data we send to fetch the next page. The server is expecting something else. You can debug by comparing with the data Firefox sends (you cna see it in Web Developer -> Network).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jarun/ddgr/issues/115#issuecomment-657871286, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADCTGIBGPQENWXNMC6YYODR3OJT3ANCNFSM4OMUF7HA .

jarun commented 4 years ago

Fixed at #117. :+1:

jn0 commented 4 years ago

hm-hm... 1.8.1 from snap on ubuntu 18.04 still fails:

jno@shuttle:~$ ddgr -d ubuntu 20.04
ERROR: ld.so: object 'libgtk3-nocsd.so.0' from LD_PRELOAD cannot be preloaded (failed to map segment from shared object): ignored.
[DEBUG] ddgr version 1.8.1 Python version 3.5.2
[DEBUG] q:ubuntu 20.04, region:us-en, page:0, curindex:1, safe:1
[DEBUG] nextParams:
[DEBUG] proxy:{'https': None}
[DEBUG] ua:ddgr/1.8.1 (textmode; Linux x86_64; 1024x768)
[DEBUG] Response body written to '/tmp/ddgr-response-1xytet86'.
[DEBUG] Prev nextParams: 
[DEBUG] Next nextParams: 
No results.
ddgr (? for help) 
ddgr (? for help) 
jno@shuttle:~$

(ignore the ERROR - it always here)

And I couldn't find that /tmp/ddgr-response-1xytet86 (even before leaving the program)

And, yes, it has still usr/local/bin/ddgr: self._u = 'https://duckduckgo.com/html' while declares 1.8.1… :(

jarun commented 4 years ago

v1.8.1 will not work. Wait for the next release or use master. The fix is in master.

cmonfortep commented 4 years ago

@jarun sorry I didn't respond you back about this. I reported the issue internally but I've not been actively following it (I'm enjoying some days off 😅). Glad to see this was solved.

jarun commented 4 years ago

No problem! Thank you!

jarun commented 4 years ago

Guys, I have released ddgr v1.9 tonight: https://github.com/jarun/ddgr/releases/tag/v1.9

Everything works as before!

Many thanks to @RTek-dev for vital PR #117. :+1:

jarun / ddgr

Suddenly stopped working #115