Client append userIp=0.1.0.2 for all instances in push queue

[GoogleCodeExporter]

[Use this form for both apiclient and oauth2client issues]

What steps will reproduce the problem?
1. Just try to refresh n > 10 different access tokens in n different queues in 
App Engine (limited for 10 requests/second/user)
2. When those queues try to refresh those tokens, some queues get 403 errors 
due to the limit rate.

What is the expected output? What do you see instead?
There is a logging.info in app engine logs for every queue:
URL being requested: 
https://www.googleapis.com/discovery/v1/apis/analytics/v3/rest?userIp=0.1.0.2
And googleapis response is User Rate Limit Exceeded. Error 403.

What version of the product are you using? On what operating system?
- App Engine
- This client
- GA API
- Push Queues (with rate = 10/s)

Please provide any additional information below:
I wanted to pass to discovery.build a parameter quotaUser or simply does not 
pass userIp, because every queue is passing it. So, if I try to refresh those 
different tokens I'm blocked and need to do exponencial backoff.
Or am I totally wrong and I'm not being blocked due to that ?
Thanks in advance

Original issue reported on code.google.com by gabrielh...@gmail.com on 19 Oct 2012 at 8:58

[GoogleCodeExporter]

Seems like you are hitting the rate limit on requesting the discovery document. 
It would be better to keep the discovery document in a local cache and use 
discovery.build_from_document, passing in that cached discovery document:

http://google-api-python-client.googlecode.com/hg/docs/epy/apiclient.discovery-m
odule.html#build_from_document

That should also be much faster.

Original comment by jcgregorio@google.com on 22 Oct 2012 at 2:06

• Changed state: Waiting

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Hello, jcgregorio. Thanks for the answer.
I'm a bit confused about that solution.
Are you saying to pickle the discovery document into Datastore and save its key 
in memcache ? Then I retrieve it when I want to use ?
What if my token expires or the refresh token becomes invalid ? Will I need to 
refresh it every time ?

The fluxogram of my application is a bit different. I'll try to explain quickly:
1) A server sends to appengine some user_id and its refresh_token
2) So I instanciate a credential:

credentials = OAuth2Credentials(self.access_token,
                                settings.CLIENT_ID,
                                settings.CLIENT_SECRET,
                                self.refresh_token,
                                None,
                                self.token_uri,
                                self.user_agent)
credentials.set_store(StorageByKeyName(CredentialsModel(),
                                       self.user_id,
                                       'credentials',
                                       memcache))

3) After that, I return the build object to be used in a view/facade:

return discovery.build(self.scope, self.scope_version,
                       http=credentials.authorize(self.http))

For me you're saying that before doing all that dance I must save it somewhere. 
But where ?
Thanks in advance

Original comment by gabrielh...@gmail.com on 23 Oct 2012 at 5:31

[GoogleCodeExporter]

"""
Are you saying to pickle the discovery document into Datastore and save its key 
in memcache ? Then I retrieve it when I want to use ?
"""

Yes.

"""
What if my token expires or the refresh token becomes invalid ? Will I need to 
refresh it every time ?
"""

The error you are seeing has nothing to do with authentication. The discovery 
document describes the API you are accessing and is requested each time you 
call discovery.build(). Since you were calling discovery.build() many times 
from a task queue you may occasionally hit a limit in how quickly you can 
retrieve discovery documents. You should instead cache the discovery document 
to avoid generating that many requests.

Original comment by jcgregorio@google.com on 23 Oct 2012 at 5:46

[GoogleCodeExporter]

Oh, I understand now.
Thank you very much for the fast answer :)

Original comment by gabrielh...@gmail.com on 23 Oct 2012 at 5:56

[GoogleCodeExporter]

Just final question:
Is the discovery object bound by the credentials ? I mean, may I store 1 object 
for 1 user/token ?

Original comment by gabrielh...@gmail.com on 23 Oct 2012 at 6:07

[GoogleCodeExporter]

The discovery object is per-API. That is, it does not change based on the user, 
their credentials, etc. So you only need to store one, and that can be used by 
everyone.

Original comment by jcgregorio@google.com on 23 Oct 2012 at 6:29

[GoogleCodeExporter]

What am I supposed to pickle ? I'm really confused right now.
I'm getting this error:

>>> a = discovery.build(self.scope, self.scope_version, 
http=credentials.authorize(self.http))
>>> type(a)
<class 'apiclient.discovery.Resource'>
>>> pickle.dumps(a)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
    save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 562, in save_tuple
    save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <class 'apiclient.discovery.Resource'>: it's not 
found as apiclient.discovery.Resource

Original comment by gabrielh...@gmail.com on 23 Oct 2012 at 7:48

[GoogleCodeExporter]

import httplib2
from apiclient import discovery

h = httplib2.Http()

resp, content = 
h.request('https://www.googleapis.com/discovery/v1/apis/analytics/v3/rest?quotaU
ser=the_name_of_your_app_goes_here')

assert resp.status == 200

# The value of 'content' is what needs to be cached.

# To build the service object for interacting with the api
# you use build_from_document with the cached discovery doc:

service = discovery.build_from_document(content)

Original comment by jcgregorio@google.com on 23 Oct 2012 at 8:00

[GoogleCodeExporter]

It worked! Used ndb so it handles memcache for me.
Thank you very much for your patience! If you come to Brazil remember me to buy 
you some beer. :)

Original comment by gabrielh...@gmail.com on 23 Oct 2012 at 8:33

[GoogleCodeExporter]

Glad to hear it!

Marking this issue as closed.

Original comment by jcgregorio@google.com on 23 Oct 2012 at 8:40

• Changed state: Closed