search

jarun / ddgr

:duck: DuckDuckGo from the terminal
GNU General Public License v3.0
2.94k stars 139 forks source link

Results are different from real (web) DuckDuckGo #124

Closed NightMachinery closed 3 years ago

NightMachinery commented 3 years ago 
$ ddgr 'lorde' --site='https://open.spotify.com/artist/' -n 5
 1.  Lorde on Spotify
     https://open.spotify.com/artist/23XObzIZxcYGLsnhg99BYN
     Lorde, Category: Artist, Albums: Teofagia, Top Tracks: Capoeira Mandou Te Chamar, Anseios (Florbela Espanca), Teofagia,
     Do Pé Ao Cume, A Solidão Listen to Lorde in full in the Spotify app.

 2.  Listen to Lorde Sanctus on Spotify. Artist · 2.3K monthly listeners.
     https://open.spotify.com/artist/6shY6D0A4NFmB8m15oV479
     Lorde Sanctus is an emcee/songwriter/producer from the coast of California (805) he is the leader of the Connect the
     Coast movement, who's mission is to represent everything lit from the central coast of...

 3.  Lorde Empress on Spotify
     https://open.spotify.com/artist/73pqss5ocI6ykYUMkl5Juz
     Listen to Lorde Empress now. Listen to Lorde Empress in full in the Spotify app.

 4.  Lorde Sanctus on Spotify
     https://open.spotify.com/artist/4uokTIPJMgLqvYmYYXY8JK
     Listen to Lorde Sanctus now. Listen to Lorde Sanctus in full in the Spotify app.

 5.  Manessa Lorde on Spotify
     https://open.spotify.com/artist/1ttuuMIICD4Fe8Ux7ms9tm
     Manessa Lorde, Category: Artist, Singles: Take Control - EP, Take Control, Top Tracks: You Are Great, Take Control, I
     Got My Healing, I Will Soar Listen to Manessa Lorde in full in the Spotify app.

ddgr (? for help)

Compare with: image

Notice how the real, painfully obvious result https://open.spotify.com/artist/163tK9Wjr9P9DmM0AVK7lm does not appear at all in ddgr.

PS: I have also tried ddgr 'lorde site:https://open.spotify.com/artist/' -n 5, ddgr --reg 'us-en' --unsafe .... They are the same.

NightMachinery commented 3 years ago

I can confirm that this happens even on normal web browsing via Chrome if I use the non-JS version (https://html.duckduckgo.com/html?q=lorde%20site%3Ahttps%3A%2F%2Fopen.spotify.com%2Fartist%2F). I suggest we get the HTML using puppeteer (I have a script for doing this at https://github.com/NightMachinary/.shells/blob/master/scripts/javascript/curlfull.js). Alternatively, ddgr can have an option to read the HTML from stdin, and just parse it.

NightMachinery commented 3 years ago

The differing results, for reference:

❯ eval-memoi full-html2 'https://html.duckduckgo.com/html?q=lorde%20site%3Ahttps%3A%2F%2Fopen.spotify.com%2Fartist%2F' | pup 'div.result .result__a attr{href}'
https://open.spotify.com/artist/23XObzIZxcYGLsnhg99BYN
https://open.spotify.com/artist/6shY6D0A4NFmB8m15oV479
https://open.spotify.com/artist/73pqss5ocI6ykYUMkl5Juz
https://open.spotify.com/artist/4uokTIPJMgLqvYmYYXY8JK
https://open.spotify.com/artist/50bA3xDkGbcIMNwPCFY0Yp
https://open.spotify.com/artist/1ttuuMIICD4Fe8Ux7ms9tm
https://open.spotify.com/artist/2eU4XsCquIPSI7OmgFudTF
https://open.spotify.com/artist/6xELsxazXWfWxLF19hXMf1
https://open.spotify.com/artist/6AXqyWYALVEFxxb48BScbh
https://open.spotify.com/artist/5ujOqE1nhhiQSEdU6Vz9vE
https://open.spotify.com/artist/3XvGZcd9G2IMFzAbDOAqns
https://open.spotify.com/artist/7kuiAPIVC7ue4UTSsmMkee
https://open.spotify.com/artist/3uoSPCFT4rIbBtGa6hmPlY
https://open.spotify.com/artist/0tvPrkKyiMTEKTH78Rb658
https://open.spotify.com/artist/2V5YUNq5OzVJUWXMhRKObw
❯ eval-memoi withchrome full-html2 'https://duckduckgo.com/?q=lorde+site%3Ahttps%3A%2F%2Fopen.spotify.com%2Fartist%2F&t=h_&ia=web' | pup 'div.result .result__a attr{href}' | rg -v 'https://duckduckgo.com'
https://open.spotify.com/artist/163tK9Wjr9P9DmM0AVK7lm
https://open.spotify.com/artist/163tK9Wjr9P9DmM0AVK7lm/about
https://open.spotify.com/artist/23XObzIZxcYGLsnhg99BYN
https://open.spotify.com/artist/6YvX19KWC0PsHrJyiazt2M
https://open.spotify.com/artist/35ZLwj2Lenf9f1HTajFryM
https://open.spotify.com/artist/6shY6D0A4NFmB8m15oV479
https://open.spotify.com/artist/6BypGPi4Q4ujl5reKdksNx
https://open.spotify.com/artist/3WYkOUwlwJRTXZqhqkeERJ
https://open.spotify.com/artist/4tL9AA0HtF3BfxMhbtFnJR
https://open.spotify.com/artist/3hb9hQLDvGNeZY0T6zhnve

Their diff -u js nonjs:

-https://open.spotify.com/artist/163tK9Wjr9P9DmM0AVK7lm
-https://open.spotify.com/artist/163tK9Wjr9P9DmM0AVK7lm/about
 https://open.spotify.com/artist/23XObzIZxcYGLsnhg99BYN
-https://open.spotify.com/artist/6YvX19KWC0PsHrJyiazt2M
-https://open.spotify.com/artist/35ZLwj2Lenf9f1HTajFryM
 https://open.spotify.com/artist/6shY6D0A4NFmB8m15oV479
-https://open.spotify.com/artist/6BypGPi4Q4ujl5reKdksNx
-https://open.spotify.com/artist/3WYkOUwlwJRTXZqhqkeERJ
-https://open.spotify.com/artist/4tL9AA0HtF3BfxMhbtFnJR
-https://open.spotify.com/artist/3hb9hQLDvGNeZY0T6zhnve
+https://open.spotify.com/artist/73pqss5ocI6ykYUMkl5Juz
+https://open.spotify.com/artist/4uokTIPJMgLqvYmYYXY8JK
+https://open.spotify.com/artist/50bA3xDkGbcIMNwPCFY0Yp
+https://open.spotify.com/artist/1ttuuMIICD4Fe8Ux7ms9tm
+https://open.spotify.com/artist/2eU4XsCquIPSI7OmgFudTF
+https://open.spotify.com/artist/6xELsxazXWfWxLF19hXMf1
+https://open.spotify.com/artist/6AXqyWYALVEFxxb48BScbh
+https://open.spotify.com/artist/5ujOqE1nhhiQSEdU6Vz9vE
+https://open.spotify.com/artist/3XvGZcd9G2IMFzAbDOAqns
+https://open.spotify.com/artist/7kuiAPIVC7ue4UTSsmMkee
+https://open.spotify.com/artist/3uoSPCFT4rIbBtGa6hmPlY
+https://open.spotify.com/artist/0tvPrkKyiMTEKTH78Rb658
+https://open.spotify.com/artist/2V5YUNq5OzVJUWXMhRKObw
jarun commented 3 years ago

ddgr parses the html ddg results.

And I see results match:

 1.  Lorde on Spotify
     https://open.spotify.com/artist/23XObzIZxcYGLsnhg99BYN
     Lorde, Category: Artist, Albums: Teofagia, Top Tracks: Capoeira Mandou Te Chamar, Anseios (Florbela
     Espanca), Teofagia, Do Pé Ao Cume, A Solidão Listen to Lorde in full in the Spotify app.

 2.  Listen to Lorde Sanctus on Spotify. Artist · 2.3K monthly listeners.
     https://open.spotify.com/artist/6shY6D0A4NFmB8m15oV479
     Lorde Sanctus is an emcee/songwriter/producer from the coast of California (805) he is the leader of
     the Connect the Coast movement, who's mission is to represent everything lit from the central coast
     of...

 3.  Lorde Empress on Spotify
     https://open.spotify.com/artist/73pqss5ocI6ykYUMkl5Juz
     Listen to Lorde Empress now. Listen to Lorde Empress in full in the Spotify app.

 4.  Lorde Sanctus on Spotify
     https://open.spotify.com/artist/4uokTIPJMgLqvYmYYXY8JK
     Listen to Lorde Sanctus now. Listen to Lorde Sanctus in full in the Spotify app.

 5.  Lorde Sanctus on Spotify
     https://open.spotify.com/artist/50bA3xDkGbcIMNwPCFY0Yp
     Listen to Lorde Sanctus now. Listen to Lorde Sanctus in full in the Spotify app.

 6.  Lordes on Spotify
     https://open.spotify.com/artist/6xELsxazXWfWxLF19hXMf1
     Lordes, Category: Artist, Top Tracks: Someone Like You, Monthly Listeners: 46, Where People Listen:
     Dublin, Wellington, Thornbury, Drachten, Uccle. Listen to Lordes in full in the Spotify app.

 7.  Lordes on Spotify
     https://open.spotify.com/artist/6AXqyWYALVEFxxb48BScbh
     Lordes, Category: Artist, Top Tracks: Someone Like You, Monthly Listeners: 7, Where People Listen:
     Oslo, Edmonton, Nepean, Malate, St Louis. Listen to Lordes in full in the Spotify app.

 8.  Lorde $anctu$ on Spotify
     https://open.spotify.com/artist/2eU4XsCquIPSI7OmgFudTF
     Listen to Lorde $anctu$ now. Listen to Lorde $anctu$ in full in the Spotify app.

 9.  Quest Lorde on Spotify
     https://open.spotify.com/artist/7kuiAPIVC7ue4UTSsmMkee
     Quest Lorde. Play on Spotify. 0. Where people listen. Listen to Quest Lorde now.

 10. Lordes on Spotify
     https://open.spotify.com/artist/3XvGZcd9G2IMFzAbDOAqns
     Listen to Lordes now. Listen to Lordes in full in the Spotify app.
NightMachinery commented 3 years ago

@jarun I built my own parser. Feel free to close this issue, though adding it as a known issue to the readme will probably save others some headache.

jarun commented 3 years ago

@NightMachinary starting with the subject-line to the last comment your comments show unwarranted rudeness. Developers of random open source projects don't owe you anything. Please be polite when you approach them.