search

abenassi / Google-Search-API

Python based api for searching google web, images, calc, and currency conversion.
552 stars 218 forks source link

Google standard search takes over 3 seconds to return answer #59

Open SearchPirat opened 6 years ago

SearchPirat commented 6 years ago

Hey abenassi,

I tried your code and I'm very happy with it. I only have the issue, that each google request takes at minimum 3s. I'm using the function as follows:

import urllib.request as urllib2 from google import google

num_pages =1 Google_result = google.search(query, num_pages, lang = 'de', area = 'de')

I would be very happy if you could help me.

Best regards, SearchPirat

JuaniFilardo commented 6 years ago

Hi @SearchPirat!

As far as I'm concerned, that's the time that each requests takes; I'm not pretty sure what could be done to reduce it (maybe some refactoring of the API itself?). I mean, scraping the whole html returned by Google and processing it takes time, so it's not that you're doing something wrong.

Have a nice day!

PS: the area param is no longer working, check issue 52 for further information.

SearchPirat commented 6 years ago

Hi @JuaniFilardo Thank you for your fast reply. Good to know, that there is nothing that I'm doing wrong. I just thought it may be possible to limit the search results or the time for searching. This would also limit the time it would take to scrape the html results.

Have a nice one SearchPirat

JuaniFilardo commented 6 years ago

By default, the search method will return the ten results from the first page from Google. Would you want to get, for example, just the first 5 results?

SearchPirat commented 6 years ago

If this would speed the search up by a factor of 2 it would be a possible solution.

SearchPirat commented 6 years ago

@JuaniFilardo Is this possible?

JuaniFilardo commented 6 years ago

Um, I really don't know if the improvement would be that significant. Maybe you could try hardcoding the library in order to fetch fewer results, and then measuring the mean time it takes.

lxwang42 commented 5 years ago

ran a quick test, lxml reduces the overall time to less than 0.8 sec(location: DE, VPS: Vultr)