search

jarun / googler

:mag: Google from the terminal
GNU General Public License v3.0
6.1k stars 528 forks source link

Issues with results if user-agent is removed #114

Closed jarun closed 8 years ago

jarun commented 8 years ago
  1. Google News doesn't work
  2. File mime type doesn't show
  3. A regular hello search results:
$ ./googler hello

1 HELLO! Online: celebrity & royal news, magazine, babies, weddings ...
/url?q=http://www.hellomagazine.com/&sa=U&ved=0ahUKEwjQw7PF3Z_NAhUQSI8KHfVNDTMQFghoMAw&usg=AFQjCNHe-OhTvjOvC5smGVTvd1mxsfl2FA
HELLO! Online brings you the latest celebrity & royal news from the UK & around  the world, magazine exclusives,
celeb babies, weddings, pregnancies and ...

2 hello network
/url?q=https://hello.com/&sa=U&ved=0ahUKEwjQw7PF3Z_NAhUQSI8KHfVNDTMQFghuMA0&usg=AFQjCNHu3ySM5MQ_ySXDktfr8Ax4fFxy4w
hello connects you with people and content around your passions. Show the  world who you are, express what you
love, and create meaningful connections.

3 Hello (Adele song) - Wikipedia, the free encyclopedia
/url?q=https://en.wikipedia.org/wiki/Hello_(Adele_song)&sa=U&ved=0ahUKEwjQw7PF3Z_NAhUQSI8KHfVNDTMQFgh0MA4&usg=AFQjCNHJRKExeuRwwdFmAKkJ6dH7qv_TeQ
"Hello" is a song by English singer Adele. It was released on 23 October 2015 by  XL Recordings as the lead
single from her third studio album, 25 (2015). Adele ...

4 Hello - Wikipedia, the free encyclopedia
/url?q=https://en.wikipedia.org/wiki/Hello&sa=U&ved=0ahUKEwjQw7PF3Z_NAhUQSI8KHfVNDTMQFgh6MA8&usg=AFQjCNEPQPRz_g4ms5UWaM9OQrZ8-qWzew
Hello is a salutation or greeting in the English language. It is first attested in  writing from 1833. Contents.
[hide]. 1 First use; 2 Etymology. 2.1 Telephone; 2.2 ...

5 Images for hello
/search?q=hello&ie=UTF-8&oe=UTF-8&prmd=ivnsfdm&tbm=isch&tbo=u&source=univ&sa=X&ved=0ahUKEwjQw7PF3Z_NAhUQSI8KHfVNDTMQsAQIgAE

6 HELLO! - Facebook
/url?q=https://www.facebook.com/hello/&sa=U&ved=0ahUKEwjQw7PF3Z_NAhUQSI8KHfVNDTMQFgiMATAV&usg=AFQjCNFX6GnLlw8l4pE7PM-eVFUijQimIg
HELLO! 1794375 likes · 54963 talking about this. The official Facebook page for  HELLO! magazine &
http://www.hellomagazine.com/ FOLLOW US @...

7 ADELE LYRICS - Hello - A-Z Lyrics
/url?q=http://www.azlyrics.com/lyrics/adele/hello.html&sa=U&ved=0ahUKEwjQw7PF3Z_NAhUQSI8KHfVNDTMQFgiSATAW&usg=AFQjCNHGIyMSO2JIfxsiYutLKdH21BVmvw
Hello, it's me. I was wondering if after all these years you'd like to meet. To go  over everything. They say
that time's supposed to heal ya. But I ain't done much ...

8 Hello
/url?q=https://hello.is/&sa=U&ved=0ahUKEwjQw7PF3Z_NAhUQSI8KHfVNDTMQFgiXATAX&usg=AFQjCNGwdUWTtCdsDuH8FLfNW3oDxaTyqQ
Sense by Hello wakes you up feeling great, improves your sleep, and helps you  learn how the environment of your
bedroom affects your rest.

9 Hello — Caller ID & Blocking - Android Apps on Google Play
/url?q=https://play.google.com/store/apps/details%3Fid%3Dcom.facebook.phone%26hl%3Den&sa=U&ved=0ahUKEwjQw7PF3Z_NAhUQSI8KHfVNDTMQFgidATAY&usg=AFQjCNHcLO9dQ-fNU0fT-KM6O5TW50jggA
Make your phone smarter with Hello. Built by Messenger just for Android, Hello  combines info from Facebook with
the contact info on your phone so it's easy to ...

a. URLs are malformed b. multiple spaces in results: Show the world, around the world, To go over c. Images for hello show in results

jarun commented 8 years ago

I was doing some profiling for optimization. It seems like we can bring the fetch time below 1 sec (without any redirection) if UA is disabled. I added a test commit https://github.com/jarun/googler/commit/57b3dae244e9f98d6f5b6aec784be589a6cd1a78 (detailed notes) and Travis build passed.

However, I am seeing the above issues when UA is disabled.