GeneralMills / pytrends

Pseudo API for Google Trends
Other
3.28k stars 821 forks source link

Broken for short timeframe - question for experts #605

Open mobilityvalley opened 11 months ago

mobilityvalley commented 11 months ago

Good Morning,

I'am playing with pytrends for some days. Thanks for your work!

I can't find a solution to query trends data for less than 1 month window (ex: 'today 1-m').

Here is my code:

from pytrends.request import TrendReq
pt = TrendReq(retries=3, backoff_factor=0.1)
pt.build_payload(["covid"], geo='FR', timeframe='today 1-m')
df = pt.interest_by_region(resolution='DMA')
print(df.to_json())

I tried any combination of timeframes and I tried to hack by setting a different User-Agent by Google still detect the request as 'SCRAPER'.

Is there a solution based on puppeteer or selenium (seems complex) ?

I HAVE A QUESTION for Trends EXPERTS:

I get the top terms from Big Query Trends Open Data for a specific day and I need to map these datas on a DMA scale map (in France), the Big Query table gives only region level and there are big discrepancies with what I see on Google Trends Web Site.

If I request the trends data for a 1 month window observation it does not match the top of the day meaning I guess. Or it still gives an idea on the dynamic for local sub regions ?

Thanks

praburamWAPKA commented 11 months ago

Use category number and Timestamp along with date start and end will work

mobilityvalley commented 11 months ago

Many thanks @praburamWAPKA it saved my day. It works by setting the timeframe on this format "YYYY-MM-DDT00 YYYY-MM-DDT01" and cat=0 and tz=0;

I still wonder my self what is the best window in time to observe a trend (at the regional DMA level).

Let's take an example:

A trend X appeared yesterday (Day D-1).

A typical trend graph shape seems to be: flat then big peak the day of trend then going flat. It is even true for big international events, things seems to be interesting for people one day, two days at the best and then disappear in the search on Google.

image

My feeling is I can "observe" the date on the last 7 days to have a 'good' view of the trend repartition on regional level.

The pure logical way to do it would be to 'observe the data for the day it appears from D-1 T 00 to D T 00 but I get a lot of 429 error if I try this.

What folks do you think from this window (7 days) approach to analyse the data ?

Thanks by advance

CyberTamilan commented 11 months ago

right now 7 days only working

mobilityvalley commented 11 months ago

I make it works for last day but it seems hazardous.

Also there is a bug in this line:

https://github.com/GeneralMills/pytrends/blob/a9984ffdc9b31d853dde2ab614a77ecbf2bf33a1/pytrends/request.py#L337

You have to remove self.geo == 'US' and if you want GEO different than US, in my case FR