search

GeneralMills / pytrends

Pseudo API for Google Trends
Other
3.2k stars 809 forks source link

Does related queries work? I keep getting 429 #625

Open josh-ashkinaze opened 1 month ago

josh-ashkinaze commented 1 month ago

Here is a MRE, using what is in README

from pytrends.request import TrendReq
kw_list = ["Blockchain"]
pytrends = TrendReq(hl='en-US', tz=360)
pytrends.build_payload(kw_list, cat=0, timeframe='today 5-y', geo='', gprop='')
pytrends.related_queries()

This throws a 429 errors. I tried wrapping this with tenacity and I keep getting 429s no matter how long I wait.

liusaint commented 1 month ago

the same

lapiceroazul4 commented 1 month ago

Same for me, It appers that only a few methods keep working.

The following are working

Suggestions & Categories are not performing that well

hl105 commented 1 month ago

build_payload() doesn't seem to be working for me

lapiceroazul4 commented 1 month ago

How you're doing it?

saadadeo commented 1 month ago

it's same for me I've been attempting to make API calls using PyTrends, but I keep receiving an HTTP error 429, which indicates "Too Many Requests." This error typically occurs when the rate limit for API requests is exceeded. However, the strange part is that I encounter this error even when I make only a single API call.

claudioferrazb commented 1 month ago

the same here =/

Toffee1878 commented 1 month ago

Same for me, It appers that only a few methods keep working.

The following are working

  • realtime_trending_searches()
  • trending_searches(pn='united_states')
  • suggestions(keyword='coronavirus')
  • categories()

Suggestions & Categories are not performing that well

Can I ask what payload you are using please?

lapiceroazul4 commented 1 month ago

Same for me, It appers that only a few methods keep working. The following are working

  • realtime_trending_searches()
  • trending_searches(pn='united_states')
  • suggestions(keyword='coronavirus')
  • categories()

Suggestions & Categories are not performing that well

Can I ask what payload you are using please?

Sure!, here it is

pytrend = TrendReq(hl='en-US', tz=360, retries=3)

pytrend.build_payload(kw_list=['whatever word you want'])

-To get realtime trendings

df = pytrend.realtime_trending_searches(pn='US')
df.head(10)

-To get top searches in US

df = pytrend.trending_searches(pn='united_states')
df.head(10)

-To get suggestions based on a word

df = pytrend.suggestions(keyword='obamacare')
print(df)

-To get related categories

df = pytrend.categories()
print(df)
df = pytrend.today_searches(pn='US')
df.head(2)

Please note that these methods are working but some others are not. This is because Pytrends is not an official API, it is an open source project that actually scraps the data from Google Trends, this implies that often we may find errors in Pytrends every time Google change something in the Google Trend's Page

Toffee1878 commented 1 month ago

Same for me, It appers that only a few methods keep working. The following are working

  • realtime_trending_searches()
  • trending_searches(pn='united_states')
  • suggestions(keyword='coronavirus')
  • categories()

Suggestions & Categories are not performing that well

Can I ask what payload you are using please?

Sure!, here it is

pytrend = TrendReq(hl='en-US', tz=360, retries=3)

pytrend.build_payload(kw_list=['whatever word you want'])

-To get realtime trendings

df = pytrend.realtime_trending_searches(pn='US')
df.head(10)

-To get top searches in US

df = pytrend.trending_searches(pn='united_states')
df.head(10)

-To get suggestions based on a word

df = pytrend.suggestions(keyword='obamacare')
print(df)

-To get related categories

df = pytrend.categories()
print(df)
  • To get today's searches in US
df = pytrend.today_searches(pn='US')
df.head(2)

Please note that these methods are working but some others are not. This is because Pytrends is not an official API, it is an open source project that actually scraps the data from Google Trends, this implies that often we may find errors in Pytrends every time Google change something in the Google Trend's Page

@lapiceroazul4 - Thank you very much.

KatMerrill commented 1 month ago

Are the other methods being fixed currently, or does anyone know of good ways to work around them? I'm trying to use interest_over_time() and it seems to have completely stopped working, probably when Google Trends updated recently. I've been consistently getting 429 errors on my first request, and haven't had luck with changing backoff_factor/retries, using proxies, or even running it on a different computer and network.

javathejhut commented 1 month ago

Starting to see data coming in now, but slowly. No immediate 429.

guaribr commented 1 month ago

I am also having the "TooManyRequestsError: The request failed: Google returned a response with code 429" error

jabeztadesse commented 1 month ago

How the hell are these guys able to do it? 🤔 https://serpapi.com/playground?engine=google_trends&q=Coffee

HumanBot000 commented 1 month ago

interest_over_time() only returns 429 for me also

lapiceroazul4 commented 1 month ago

How the hell are these guys able to do it? 🤔 https://serpapi.com/playground?engine=google_trends&q=Coffee

Idk how they do that, but, it looks awesome, any idea of how can we get that data?

HumanBot000 commented 3 weeks ago

How the hell are these guys able to do it? 🤔 https://serpapi.com/playground?engine=google_trends&q=Coffee

Idk how they do that, but, it looks awesome, any idea of how can we get that data?

hmm..

jabeztadesse commented 3 weeks ago

How the hell are these guys able to do it? 🤔 https://serpapi.com/playground?engine=google_trends&q=Coffee

Idk how they do that, but, it looks awesome, any idea of how can we get that data?

hmm..

Got any idea?

HumanBot000 commented 3 weeks ago

I just know, that if I try to replicate the http calls my browser does in postman, I keep getting an error that this isn't a browser. So my guess would be, that they are just web scraping it. When you use their API (with only 100 free calls/months) my guess would be, that they have headless browsers running on a server.

gl2007 commented 3 weeks ago

Can this library include such a feature where it uses a headless browser to do the scraping just like the API? Possibly, someone can add it

On Sat, Aug 17, 2024 at 2:25 PM Tobias @.***> wrote:

I just know, that if I try to replicate the http calls my browser does in postman, I keep getting an error that this isn't a browser. So my guess would be, that they are just web scraping it. When you use their API (with only 100 free calls/months) my guess would be, that they have headless browsers running on a server.

— Reply to this email directly, view it on GitHub https://github.com/GeneralMills/pytrends/issues/625#issuecomment-2294935238, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7IJS7IH2BKR6FLQKDN3BTZR6ISBAVCNFSM6AAAAABK3TZXKWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJUHEZTKMRTHA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

franz101 commented 3 days ago

My personal workaround for similar data is to use the wikipedia api: see python wikipedia api for page views