Invalid_request from credential.refreshToken()

[GoogleCodeExporter]

Version of google-api-java-client (e.g. 1.15.0-rc)?
1.18.0

Java environment (e.g. Java 6, Android 2.3, App Engine)?
Java 7 with AppEngine

Describe the problem.
We're seeing a lot of invalid_request errors happening ( stack trace below ).  
We opened a ticket with the Google API team and they said it was an issue with 
the java client library and to post here.  Below is there response for this and 
the stack trace for the error.

--- GOOGLE SUPPORT ---
Hello Nathan,

Thank you for your response. I do apologize due to the amount of time that has 
taken for us to get back to you. We have isolated the issue and it seems that 
it is being caused by the API Client Library that is being used. Unfortunately, 
we do not offer support for client library issues through this support portal.

Simply post your questions on the issue tracker maintained by the Client 
library team (http://code.google.com/p/google-api-java-client/downloads/list), 
a member of our global developer community will respond shortly.

NOTE: make sure you list the correct client libraries issue tracker for your 
user (Python, Java, Etc.)
--- END GOOGLE SUPPORT ---
--- STACK TRACE ---
com.google.api.client.auth.oauth2.TokenResponseException: 403 OK
{
  "error" : "invalid_request"
}
        at com.google.api.client.auth.oauth2.TokenResponseException.from(TokenResponseException.java:105)
        at com.google.api.client.auth.oauth2.TokenRequest.executeUnparsed(TokenRequest.java:287)
        at com.google.api.client.auth.oauth2.TokenRequest.execute(TokenRequest.java:307)
        at com.google.api.client.googleapis.auth.oauth2.GoogleCredential.executeRefreshToken(GoogleCredential.java:268)
        at com.google.api.client.auth.oauth2.Credential.refreshToken(Credential.java:489)
--- END STACK TRACE ---

How would you expect it to be fixed?
NA

Original issue reported on code.google.com by mike.sch...@promevo.com on 19 May 2014 at 2:39

[GoogleCodeExporter]

Thanks for posting the issue. We are investigating and will get back to you 
shortly. By the way, were you running your application on a large number of 
AppEngine instances simultaneously?

Original comment by wonder...@google.com on 19 May 2014 at 4:51

[GoogleCodeExporter]

Thanks for the quick response.  No, we do run some things simultaneously but we 
get this on just one off user actions ( user clicking a button in our 
application ).  I can send you over a request / response if that will assist in 
narrowing down the issue.

Thanks,
Mike

Original comment by mike.sch...@promevo.com on 19 May 2014 at 5:04

[GoogleCodeExporter]

Here is an example of a failure and a successful call below.  I commented out 
some of the codes that are sent back and forth for security reasons.

ERRORED

CONFIG: -------------- REQUEST  --------------
POST https://accounts.google.com/o/oauth2/token
Accept-Encoding: gzip
User-Agent: Google-HTTP-Java-Client/1.15.0-rc (gzip)
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Content-Length: 568

May 19, 2014 3:45:14 PM com.google.api.client.http.HttpRequest execute
CONFIG: curl -v --compressed -X POST -H 'Accept-Encoding: gzip' -H 'User-Agent: 
Google-HTTP-Java-Client/1.15.0-rc (gzip)' -H 'Content-Type: 
application/x-www-form-urlencoded; charset=UTF-8' -d '@-' -- 
'https://accounts.google.com/o/oauth2/token' << $$$
May 19, 2014 3:45:14 PM com.google.api.client.util.LoggingByteArrayOutputStream 
close
CONFIG: Total: 568 bytes
May 19, 2014 3:45:14 PM com.google.api.client.util.LoggingByteArrayOutputStream 
close
CONFIG: 
grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=XXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX << commented out for security
May 19, 2014 3:45:14 PM 
com.google.apphosting.utils.jetty.AppEngineAuthentication$AppEngineAuthenticator
 authenticate
FINE: Got /_ah/channel/dev, returning NOBODY to imply authentication is in 
progress.
May 19, 2014 3:45:14 PM com.google.api.client.http.HttpResponse <init>
CONFIG: -------------- RESPONSE --------------
403 OK
content-type: application/json; charset=utf-8
cache-control: no-cache, no-store, max-age=0, must-revalidate
pragma: no-cache
expires: Fri, 01 Jan 1990 00:00:00 GMT
date: Mon, 19 May 2014 15:45:00 GMT
content-disposition: attachment; filename="json.txt"; filename*=UTF-8''json.txt
content-encoding: gzip
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
server: GSE
alternate-protocol: 443:quic
transfer-encoding: chunked

May 19, 2014 3:45:14 PM com.google.api.client.util.LoggingByteArrayOutputStream 
close
CONFIG: Total: 33 bytes
May 19, 2014 3:45:14 PM com.google.api.client.util.LoggingByteArrayOutputStream 
close
CONFIG: {
  "error" : "invalid_request"
}

GOOD ONE
CONFIG: -------------- REQUEST  --------------
POST https://accounts.google.com/o/oauth2/token
Accept-Encoding: gzip
User-Agent: Google-HTTP-Java-Client/1.15.0-rc (gzip)
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Content-Length: 568

May 19, 2014 3:59:05 PM com.google.api.client.http.HttpRequest execute
CONFIG: curl -v --compressed -X POST -H 'Accept-Encoding: gzip' -H 'User-Agent: 
Google-HTTP-Java-Client/1.15.0-rc (gzip)' -H 'Content-Type: 
application/x-www-form-urlencoded; charset=UTF-8' -d '@-' -- 
'https://accounts.google.com/o/oauth2/token' << $$$
May 19, 2014 3:59:05 PM com.google.api.client.util.LoggingByteArrayOutputStream 
close
CONFIG: Total: 568 bytes
May 19, 2014 3:59:05 PM com.google.api.client.util.LoggingByteArrayOutputStream 
close
CONFIG: 
grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=XXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX << commented out for security
May 19, 2014 3:59:05 PM 
com.google.apphosting.utils.jetty.AppEngineAuthentication$AppEngineAuthenticator
 authenticate
FINE: Got /_ah/channel/dev, returning NOBODY to imply authentication is in 
progress.
May 19, 2014 3:59:05 PM com.google.api.client.http.HttpResponse <init>
CONFIG: -------------- RESPONSE --------------
200 OK
content-type: application/json; charset=utf-8
cache-control: no-cache, no-store, max-age=0, must-revalidate
pragma: no-cache
expires: Fri, 01 Jan 1990 00:00:00 GMT
date: Mon, 19 May 2014 15:58:51 GMT
content-disposition: attachment; filename="json.txt"; filename*=UTF-8''json.txt
content-encoding: gzip
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
server: GSE
alternate-protocol: 443:quic
transfer-encoding: chunked

May 19, 2014 3:59:05 PM com.google.api.client.util.LoggingByteArrayOutputStream 
close
CONFIG: Total: 120 bytes
May 19, 2014 3:59:05 PM com.google.api.client.util.LoggingByteArrayOutputStream 
close
CONFIG: {
  "access_token" : "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",  <-- replaced for security.
  "token_type" : "Bearer",
  "expires_in" : 3600
}

Original comment by mike.sch...@promevo.com on 19 May 2014 at 5:05

[GoogleCodeExporter]

Great. Thanks.

Original comment by wonder...@google.com on 19 May 2014 at 5:06

[GoogleCodeExporter]

Looks like you were using google-http-java-client 1.15.0-rc. Did you try with 
the latest version, ie., 1.18.0-rc?

Original comment by wonder...@google.com on 19 May 2014 at 5:19

[GoogleCodeExporter]

Sorry about that.  That was from other testing.  Here are some from 1.18.

com.google.api.client.http.HttpRequest execute: curl -v --compressed -X POST -H 
'Accept-Encoding: gzip' -H 'User-Agent: Google-HTTP-Java-Client/1.18.0-rc 
(gzip)' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -d 
'@-' -- 'https://accounts.google.com/o/oauth2/token' << $$$ 
D2014-05-19 11:49:45.370
com.google.api.client.util.LoggingByteArrayOutputStream close: Total: 826 bytes 
D2014-05-19 11:49:45.370
com.google.api.client.util.LoggingByteArrayOutputStream close: 
grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion= 
XXXXXXXXXXXXXXXXXXXXXXXXXXXXX<<---REMOVED FOR SECURITY
D2014-05-19 11:49:45.463
com.google.api.client.http.HttpResponse <init>: -------------- RESPONSE 
-------------- 403 OK content-type: application/json; charset=utf-8 
cache-control: no-cache, no-store, max-age=0, must-revalidate pragma: no-cache 
expires: Fri, 01 Jan 1990 00:00:00 GMT date: Mon, 19 May 2014 15:49:45 GMT 
content-disposition: attachment; filename="json.txt"; filename*=UTF-8''json.txt 
content-encoding: gzip x-content-type-options: nosniff x-frame-options: 
SAMEORIGIN x-xss-protection: 1; mode=block server: GSE alternate-protocol: 
443:quic x-google-cache-control: remote-fetch via: HTTP/1.1 GWA 
D2014-05-19 11:49:45.463

2014-05-19 11:49:45.476
com.google.api.client.http.HttpRequest execute: -------------- REQUEST 
-------------- POST https://accounts.google.com/o/oauth2/token Accept-Encoding: 
gzip User-Agent: Google-HTTP-Java-Client/1.18.0-rc (gzip) Content-Type: 
application/x-www-form-urlencoded; charset=UTF-8 Content-Length: 826 
D2014-05-19 11:49:45.477
com.google.api.client.http.HttpRequest execute: curl -v --compressed -X POST -H 
'Accept-Encoding: gzip' -H 'User-Agent: Google-HTTP-Java-Client/1.18.0-rc 
(gzip)' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -d 
'@-' -- 'https://accounts.google.com/o/oauth2/token' << $$$ 
D2014-05-19 11:49:45.477
com.google.api.client.util.LoggingByteArrayOutputStream close: Total: 826 bytes 
D2014-05-19 11:49:45.477
com.google.api.client.util.LoggingByteArrayOutputStream close: 
grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion= 
XXXXXXXXXXXXXXXXXXXXXXXXXXXXX<<---REMOVED FOR SECURITY
D2014-05-19 11:49:45.559

D2014-05-19 11:49:46.574
com.google.api.client.http.HttpRequest execute: -------------- REQUEST 
-------------- POST https://accounts.google.com/o/oauth2/token Accept-Encoding: 
gzip User-Agent: Google-HTTP-Java-Client/1.18.0-rc (gzip) Content-Type: 
application/x-www-form-urlencoded; charset=UTF-8 Content-Length: 826 
D2014-05-19 11:49:46.574
com.google.api.client.http.HttpRequest execute: curl -v --compressed -X POST -H 
'Accept-Encoding: gzip' -H 'User-Agent: Google-HTTP-Java-Client/1.18.0-rc 
(gzip)' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -d 
'@-' -- 'https://accounts.google.com/o/oauth2/token' << $$$ 
D2014-05-19 11:49:46.574
com.google.api.client.util.LoggingByteArrayOutputStream close: Total: 826 bytes 
D2014-05-19 11:49:46.575
com.google.api.client.util.LoggingByteArrayOutputStream close: 
grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion= 
XXXXXXXXXXXXXXXXXXXXXXXXXXXXX<<---REMOVED FOR SECURITY
D2014-05-19 11:49:46.651

Original comment by mike.sch...@promevo.com on 19 May 2014 at 5:24

[GoogleCodeExporter]

Also, here is the stack trace from 1.18

com.google.api.client.auth.oauth2.Credential handleResponse: unable to refresh 
token
com.google.api.client.auth.oauth2.TokenResponseException: 403 OK
{
  "error" : "invalid_request"
}
    at com.google.api.client.auth.oauth2.TokenResponseException.from(TokenResponseException.java:105)
    at com.google.api.client.auth.oauth2.TokenRequest.executeUnparsed(TokenRequest.java:287)
    at com.google.api.client.auth.oauth2.TokenRequest.execute(TokenRequest.java:307)
    at com.google.api.client.googleapis.auth.oauth2.GoogleCredential.executeRefreshToken(GoogleCredential.java:268)
    at com.google.api.client.auth.oauth2.Credential.refreshToken(Credential.java:489)
    at com.google.api.client.auth.oauth2.Credential.handleResponse(Credential.java:272)
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:999)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:410)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:343)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:460)

Original comment by mike.sch...@promevo.com on 19 May 2014 at 5:25

[GoogleCodeExporter]

If you need anything else just let me know.  We're seeing these errors 
consistently throughout the day so I can get you more debug info if you need.

Thanks,
Mike

Original comment by mike.sch...@promevo.com on 19 May 2014 at 6:17

[GoogleCodeExporter]

FYI, not sure if you can see logs or anything, but were getting this error now 
with the domain gpanel.info.

Thanks,
Mike

Original comment by mike.sch...@promevo.com on 19 May 2014 at 8:30

[GoogleCodeExporter]

Mike, What's the email address of the service account used to create the 
assertion? That would be helpful for our debugging.

Original comment by wonder...@google.com on 19 May 2014 at 10:11

[GoogleCodeExporter]

Can you please shoot me an email and I'll respond to you with the service 
account, I'd rather not post it so everyone can see it.

Thanks,
Mike

Original comment by mike.sch...@promevo.com on 19 May 2014 at 10:16

[GoogleCodeExporter]

Just responded to your email.  If you need anything definitely let us know.  
We're seeing this error a lot lately.

Thanks,
Mike

Original comment by mike.sch...@promevo.com on 19 May 2014 at 10:23

[GoogleCodeExporter]

Also, not sure if you can get into the Google support portal, but the case that 
we had opened up and referred us here is 04248873.

Thanks,
Mike

Original comment by mike.sch...@promevo.com on 19 May 2014 at 10:24

[GoogleCodeExporter]

Just another FYI.  If you can see the backend logs ( not sure if you can or not 
), we're seeing a lot of invalid_requests at the moment.

Thanks,
Mike

Original comment by mike.sch...@promevo.com on 20 May 2014 at 12:46

[GoogleCodeExporter]

Just wondering if you've had any luck in replicating the issue?  

Original comment by mike.sch...@promevo.com on 20 May 2014 at 4:42

[GoogleCodeExporter]

We've got some results but are still investigating. Stay tuned.

Original comment by wonder...@google.com on 20 May 2014 at 6:14

[GoogleCodeExporter]

Awesome.  Is it something that we are doing wrong or is it something on 
Google's side of things?  If there is anything you would like us to try 
definitely let me know.

Thanks,
Mike

Original comment by mike.sch...@promevo.com on 20 May 2014 at 6:18

[GoogleCodeExporter]

Hi Mike, we took a dump of your recent requests history and it turns out that 
you were sending 100+ requests in 5 minutes, which apparently exceeds our quota 
limit for number of active access tokens. Are you having a service account 
impersonating a large number of users?

Original comment by wonder...@google.com on 20 May 2014 at 6:47

[GoogleCodeExporter]

Hi, well we use the same service account ( the one that I emailed you ) for 
accessing resources on different domains ( this is an apps marketplace 
application ).  I thought the limit for tokens was on a per user basis ( not a 
service account basis )?

Original comment by mike.sch...@promevo.com on 20 May 2014 at 6:52

[GoogleCodeExporter]

That explains. When using service accounts, the requests are sent on behalf of 
that particular service account, instead of end users, and thus the limit is 
per service account basis. You can read more on 
https://developers.google.com/accounts/docs/OAuth2ServiceAccount.

We are making changes to our client library so that it will return "Rate limit 
exceeded." when this happens and do an exponential backoff accordingly. At the 
server side, we are also working on a project that will remove this rate limit 
completely and you can expect this to be done pretty soon.

Original comment by wonder...@google.com on 20 May 2014 at 7:01

[GoogleCodeExporter]

Thanks for the update.  We've got a big problem then.  We have hundreds of 
domains that actively use our product ( you can find our product in the chrome 
web store and in the apps marketplace ).  So are you saying that if 100 of 
those domains login and take an action in 5 minutes, we'll see the 
invalid_requests?  Or is it just 100 requests by a service account within 5 
minutes ( no matter the number of different domains )?

Do you know of a work around?  Can we request a quota increase so that we won't 
hit this error until the server project you mentioned is put in place so we 
don't have to worry about the rate limit?

Thanks,
Mike

Original comment by mike.sch...@promevo.com on 20 May 2014 at 7:11

[GoogleCodeExporter]

Hi Mike, from the logs, we suspect that you were using service account with 
impersonation. With impersonation, token requests are sent on behalf of a 
particular domain, specified by an email address, instead of your service 
account. The rate limit is per domain basis in that case. The domain that's 
sending most of the requests is administrator@gpanel.info. You probably want to 
 balance your requests among all domains. You may want to read the "Additional 
claims" section here: 
https://developers.google.com/accounts/docs/OAuth2ServiceAccount#formingclaimset

Original comment by wonder...@google.com on 20 May 2014 at 11:15

[GoogleCodeExporter]

Thanks for the info.  So does this refer to "api requests" or "token requests". 
 for example, if I make an api call to google, does that count towards the rate 
limit or does the rate limit just get impacted when we call credential.refresh, 
or get a new token?

Thanks,
Mike

Original comment by mike.sch...@promevo.com on 20 May 2014 at 11:19

[GoogleCodeExporter]

This rate limit is for "token requests", i.e., when you call 
credential.refresh(). You would have another rate limit for api request but 
that's something you can config/raise through Google Cloud Console.

Original comment by wonder...@google.com on 20 May 2014 at 11:22

[GoogleCodeExporter]

Thanks for the quick response.  So just to confirm, if we request on behalf of 
different users within the same domain, that still counts towards the "domain 
rate limit" correct?

Thanks again,
Mike

Original comment by mike.sch...@promevo.com on 20 May 2014 at 11:30

[GoogleCodeExporter]

I should probably have used the accurate term. Impersonation works for a 
particular "user" specified by an email address, through the "sub" field of a 
JWT. 

Original comment by wonder...@google.com on 20 May 2014 at 11:46

[GoogleCodeExporter]

Thanks for the clarification.  So basically if a domain has 100 users on our 
application, and they all choose to view the documents ( using the drive 
service ) for their own accounts using our application at the same time, we're 
going to hit the limit?  because that would be 100 ( or however many ) requests 
to get a token and that would count to the domain that they are all on.  Am I 
correct in saying this?

( sorry, I'm just trying to understand exactly what will cause the limit so we 
can figure out a way to fix this and not get this error )

Thanks,
Mike

Original comment by mike.sch...@promevo.com on 20 May 2014 at 11:55

[GoogleCodeExporter]

And FYI, I used drive specifically because the only way to get the drive files 
for a specific user is to specify the setServiceAccountUser as the user you 
want to retrieve the files for ( this is similar for other services as well ).

Thanks,
Mike

Original comment by mike.sch...@promevo.com on 20 May 2014 at 11:57

[GoogleCodeExporter]

No, this limit is per user basis. So if you have 100 users within one domain, 
and you impersonate them differently(passing different user emails to 
setServiceAccountUser()), you won't hit that limit as long as you don't send 
requests for one user too quickly.

I see you sent a lot of token requests for user administrator@gpanel.info, you 
probably want to slow down a little bit.

Original comment by wonder...@google.com on 21 May 2014 at 12:16

[GoogleCodeExporter]

Hello,

We've made some changes last night and this morning.  Can you compare the 
requests that you were seeing yesterday with what we're requesting now and see 
if there is a big difference?

Thanks,
Mike

Original comment by mike.sch...@promevo.com on 21 May 2014 at 4:03

[GoogleCodeExporter]

We still see a lot of token requests for administrator@gpanel.info. Do you have 
a log at your side that can help us debugging? What was your use case and why 
were you sending so many requests on behalf of this user?

Original comment by wonder...@google.com on 21 May 2014 at 6:00

[GoogleCodeExporter]

Is this still an issue?

Original comment by wonder...@google.com on 29 Dec 2014 at 10:27