Quota defeating / Authentication

[Dayjo]

Hello!

I've been wondering about overcoming the Quota problem. Currently, I cannot really run any of the calls without getting the Quota error - the first request I did today got the error. Then sometimes it works, other times it doesn't.

I read some suggestions by others about overcoming this by doing a request to login and grabbing the cookies, then sending those cookies with the request to the API so that it thinks you're authenticated and the quota is not so tight.

Not sure if anyone has tried this yet or found other ways to overcome the quota errors?

After grabbing my cookies from the browser after doing a search on google trends, I added them to the trendData.js request and it does actually appear to work, something like this allows me to make more regular requests;

( I have obfuscated the real cookie values )

rp( {
    uri: 'http://www.google.com/trends/fetchComponent?q="' + keyword + '"&cid=TIMESERIES_GRAPH_0&export=3',
    headers: {
        'Cookie': 'APISID=phZcBBd_oF_clAao/A2biC9KLw1Wk42F4H; GMAIL_RTT=223; HSID=AvCAPi1_gcLNPHp04; NID=90\=ljp9bPedFIvbHjyJu_ASSrzAyOjG8bql-141rlIlMUAFUl2whg2O7g6u8Fa0_I5Nf3WteTzfPF4UEp1K5BZw_FLqRXrpdICasdbkRT24zGGkdJHloBLmNZ55hJb2EgUtfsglxlevfPTIeM5bSZ0SmQejORetaXY_yWUvLxQ5NgsxFjMw8eBuqFYruG3aOMhK4jCrc2p8evYPwLMU5t5z1YqdjvBsJw0pWHTHpM7WWWqHhEVvPgZh8SwvAdCTMidd4Q9TTa3bX77lA8hG_kvcaXGApb1ppelSHCE607viCgNYy4pBHAMYypWW9cgX_WxwOmhzvbSjNn21gJZw6TUkadGhEt8vWYjbJUW7q; SAPISID=cZpNlAneFfMDfB3FLx/AB-QxJFQYeWxEY6zB; SID=7gOUsm_siHAB7zmta3cPjuZM-Nos7Ok4pEekdyB_DPpWH3CspzMf4fxnJ7QoWelTrapcMg.; SSID=ASVarA9Iip_lMFg-w;'
            }
});

How reliable this method will be, I'm not sure. Regardless I think something will need to be built in to any app using this to detect the quota and retry later.

[samarv]

Hey man!

what kindda quota limits you are getting now, after cookies? better than 10/ min or 200/ hr ?

[Dayjo]

Far lower than normal without the cookies, literally the first request I did said that I reached my limit. Apparently Google gives reduced quotas if there are no cookies as it recognises that it's a bot.

It just seemed very inconsistent. However when sending the cookies along with the request I had consistent results, (though i've yet to benchmark how many exactly)

[pat310]

@Dayjo sounds like it's worth a shot - good idea!

[Dayjo]

I'm off to Vietnam for a couple of weeks on Friday so won't get a chance to implement it properly, however I may get a chance when I'm back 👍

[Dayjo]

@pat310 Going to probably get a chance to look into this over the next couple of weeks so will keep you posted! Need to come up with a proof of concept for a reliable way of utilising this.

[KelvinCampelo]

@Dayjo can you explain to me, what the cookies I need to send... I would be grateful!

[Dayjo]

Hello @KelvinCampelo

As in the code in my first post, the cookies I was sending were;

GMAIL_RTT=223;
HSID=AvCAPi1_gcLNPHp04;
NID=90\=ljp9bPedFIvbHjyJu_ASSrzAyOjG8bql-141rlIlMUAFUl2whg2O7g6u8Fa0_I5Nf3WteTzfPF4UEp1K5BZw_FLqRXrpdICasdbkRT24zGGkdJHloBLmNZ55hJb2EgUtfsglxlevfPTIeM5bSZ0SmQejORetaXY_yWUvLxQ5NgsxFjMw8eBuqFYruG3aOMhK4jCrc2p8evYPwLMU5t5z1YqdjvBsJw0pWHTHpM7WWWqHhEVvPgZh8SwvAdCTMidd4Q9TTa3bX77lA8hG_kvcaXGApb1ppelSHCE607viCgNYy4pBHAMYypWW9cgX_WxwOmhzvbSjNn21gJZw6TUkadGhEt8vWYjbJUW7q;
SAPISID=cZpNlAneFfMDfB3FLx/AB-QxJFQYeWxEY6zB; 
SID=7gOUsm_siHAB7zmta3cPjuZM-Nos7Ok4pEekdyB_DPpWH3CspzMf4fxnJ7QoWelTrapcMg.; 
SSID=ASaVarA9Iip_lMFg-w;

Which I'd extracted from the browser when I'd logged into google. I don't know what they do, I literlaly just copied them all :)

[Dayjo]

Note to self / anyone else; I'm going to be attempting to max out the quota. I would like to go for around 170k requests a month. Hopefully that's within their limit!

@pat310 @dreyco676 no idea if either of you have tried or had anyone else try anything near this number?

[pat310]

@Dayjo I'm not sure, but I would be curious to see what the number is. I'll let you know as soon as it's updated

[pat310]

@Dayjo Closing this for now. Are you going to run a test to see what the limit is?