\u003d aka "=" character in json key breaks oauth2client

[felagund]

Reported here: https://github.com/burnash/gspread/issues/239 for details, but in -----BEGIN PRIVATE KEY-----\n*\n-----END PRIVATE KEY-----\n", it seems * must not contain \u003d

This seems to be a problem only for python 2.7 (although possibly python 3 too as I am blocked by https://github.com/google/oauth2client/issues/106 so I might not have got that far with python 3).

[dhermes]

The key should be bytes, not unicode.

[felagund]

Yes, but this is happening with python 2.7 where bytes it just string.

This:

from oauth2client.client import SignedJwtAssertionCredentials
import gspread
key = "-----BEGIN PRIVATE KEY-----\nMIICdwIBADANBgkqhkiG9w0BAQEFAASCAmEwggJdAgEAAoGBAMJKii1hhVxqjGVa\ntXP16aa/nprMxzNLGwk0WjjFpM8pLWj40RQFZnl/LoIiiPOpRfzihZjj/EP+Uh41\nVq6MCx2wpHOmD++HCG0Z4gNeF/brNmXEXNVtw0AgzBn4wZhb/EeGxYN8Rzuu+Rlz\n3gafw7SnlGBut4nRPNF6EiT1Jv8FAgMBAAECgYEAh3YrAtr0fWu4OU5WJuR0pJMD\nRRCzbjrWPcOnh9+dOeaOx5p7d4dEXDdlMMxdSe1iY5+X3/JMxydBH9i2d/oiILgG\naPEIu21bAEER5koOFToAZYKLF10VfGXqQ/MJYNwqPIXxYMBY7nRIcJ7KiEzELwYA\nwFMtFYk3rEupbdyYe50CQQDzRz0Du522UB5Wb2rrO9zK4CINPkWW1Wc/i/hfVA9Q\nx/dFzlJTcLjWUisKZn2qzz5SRgjcCHELPLGCLto49WLLAkEAzHODEWiU5oRuANLH\ncZBAmsRs/XTupdupRbOaDcDErMWbhA2dibV081w85Fv7GN1s7OcVi/EV/ivLWakY\nPolbbwJBAJdUhhjPWVCyT9zWm/zOFQ9CEwyHdwPrpbk62Xp7MLfAWWUQLdfns6Lm\nJA3pKVzaY9sL47Dzs1YZIBZqDKcHxbcCQFD8UKbMrm7BdnGNDMPHSFAGDsY3U3Dc\n5iheBF/+Q+nMPNdKLqUd47Wii9xJMyWeUE9nGfnc/cE4x8w0Vw4uirECQHI4AV5K\nkm97buy4HINn8a+AkCAWabBfdL7eTA2A0cIJ4H5iWnEXPR2oiRSGRpgkZON4RnqN\n7/xjApBLOUJrXxM\u003d\n-----END PRIVATE KEY-----\n"
email = 'bleble@developer.gserviceaccount.com'
scope = ['https://spreadsheets.google.com/feeds']
credentials = SignedJwtAssertionCredentials(key, email, scope)
gc = gspread.authorize(credentials)

(the key and email are made up but it happens with pair generated with google too) It results in this (I am not sure if it is not a bug in gspread, but the error is deep from oauth2client):

Error                                     Traceback (most recent call last)
/tmp/ipython_edit_CRIEbT/ipython_edit_3l8dsZ.py in <module>()
      5 scope = ['https://spreadsheets.google.com/feeds']
      6 credentials = SignedJwtAssertionCredentials(key, email, scope)
----> 7 gc = gspread.authorize(credentials)

/usr/local/lib/python2.7/dist-packages/gspread/client.pyc in authorize(credentials)
    333     """
    334     client = Client(auth=credentials)
--> 335     client.login()
    336     return client
    337 

/usr/local/lib/python2.7/dist-packages/gspread/client.pyc in login(self)
     96 
     97                 http = httplib2.Http()
---> 98                 self.auth.refresh(http)
from oauth2client.client import SignedJwtAssertionCredentials
     99 
    100             self.session.add_header('Authorization', "Bearer " + self.auth.access_token)

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in refresh(self, http)
    596         request.
    597     """
--> 598     self._refresh(http.request)
    599 
    600   def revoke(self, http):

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in _refresh(self, http_request)
    767     """
    768     if not self.store:
--> 769       self._do_refresh_request(http_request)
    770     else:
    771       self.store.acquire_lock()

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in _do_refresh_request(self, http_request)
    793       AccessTokenRefreshError: When the refresh fails.
    794     """
--> 795     body = self._generate_refresh_request_body()
    796     headers = self._generate_refresh_request_headers()
    797 
     97                 http = httplib2.Http()
---> 98                 self.auth.refresh(http)
     99 
    100             self.session.add_header('Authorization', "Bearer " + self.auth.access_token)

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in refresh(self, http)
    596         request.
    597     """
--> 598     self._refresh(http.request)
    599 
    600   def revoke(self, http):

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in _refresh(self, http_request)
    767     """
    768     if not self.store:
--> 769       self._do_refresh_request(http_request)
    770     else:
    771       self.store.acquire_lock()

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in _do_refresh_request(self, http_request)
    793       AccessTokenRefreshError: When the refresh fails.
    794     """
--> 795     body = self._generate_refresh_request_body()
    796     headers = self._generate_refresh_request_headers()
    797 

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in _generate_refresh_request_body(self)
   1423 
   1424   def _generate_refresh_request_body(self):
-> 1425     assertion = self._generate_assertion()
   1426 
   1427     body = urllib.parse.urlencode({

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in _generate_assertion(self)
   1552     private_key = base64.b64decode(self.private_key)
   1553     return crypt.make_signed_jwt(crypt.Signer.from_string(
-> 1554         private_key, self.private_key_password), payload)
   1555 
   1556 # Only used in verify_id_token(), which is always calling to the same URI

/usr/local/lib/python2.7/dist-packages/oauth2client/crypt.pyc in from_string(key, password)
    167         if isinstance(password, six.text_type):
    168           password = password.encode('utf-8')
--> 169         pkey = crypto.load_pkcs12(key, password).get_privatekey()
    170       return OpenSSLSigner(pkey)
    171 

Error: [('asn1 encoding routines', 'ASN1_CHECK_TLEN', 'wrong tag'), ('asn1 encoding routines', 'ASN1_ITEM_EX_D2I', 'nested asn1 error')]

[dhermes]

The error you sent is for an invalid key. Also the line you referenced is on line 146, not 169 so you may have an out of date oauth2client.

[felagund]

Hm, I am on version 1.4.11, that is recent enough, no?

It also happens with a key provided by google - can I somehow securely send it to you (I do not feel comfortable posting it here)? I changed a few characters in the one pasted here. Or is Google producing invalid keys?

[migurski]

I had a similar problem, and found that even in Python 2.7 I needed to encode the key to utf-8.

[felagund]

What do you mean by encoding it to unicode?

[migurski]

Something like this: u'\u003d'.encode('utf8').

u'\u003d' and b'=' are equivalent in Python 2.7, but oauth2client only wants the latter.

[felagund]

Indeed, you are right. When I am not using this (ie. a unicode string encoded as 'utf8' when the key includes '\u003d', I get this error using Python 2.7.9

/tmp/ipython_edit_AuOR2A/ipython_edit_MAAUgC.py in login_using_gspread()
     13         )
     14 
---> 15     return gspread.authorize(credentials)

/usr/local/lib/python2.7/dist-packages/gspread/client.pyc in authorize(credentials)
    333     """
    334     client = Client(auth=credentials)
--> 335     client.login()
    336     return client
    337 

/usr/local/lib/python2.7/dist-packages/gspread/client.pyc in login(self)
     96 
     97                 http = httplib2.Http()
---> 98                 self.auth.refresh(http)
     99 
    100             self.session.add_header('Authorization', "Bearer " + self.auth.access_token)

/home/drew/.local/lib/python2.7/site-packages/oauth2client/client.pyc in refresh(self, http)
    596         request.
    597     """
--> 598     self._refresh(http.request)
    599 
    600   def revoke(self, http):

/home/drew/.local/lib/python2.7/site-packages/oauth2client/client.pyc in _refresh(self, http_request)
    767     """
    768     if not self.store:
--> 769       self._do_refresh_request(http_request)
    770     else:
    771       self.store.acquire_lock()

/home/drew/.local/lib/python2.7/site-packages/oauth2client/client.pyc in _do_refresh_request(self, http_request)
    793       AccessTokenRefreshError: When the refresh fails.
    794     """
--> 795     body = self._generate_refresh_request_body()
    796     headers = self._generate_refresh_request_headers()
    797 

/home/drew/.local/lib/python2.7/site-packages/oauth2client/client.pyc in _generate_refresh_request_body(self)
   1423 
   1424   def _generate_refresh_request_body(self):
-> 1425     assertion = self._generate_assertion()
   1426 
   1427     body = urllib.parse.urlencode({

/home/drew/.local/lib/python2.7/site-packages/oauth2client/client.pyc in _generate_assertion(self)
   1552     private_key = base64.b64decode(self.private_key)
   1553     return crypt.make_signed_jwt(crypt.Signer.from_string(
-> 1554         private_key, self.private_key_password), payload)
   1555 
   1556 # Only used in verify_id_token(), which is always calling to the same URI

/home/drew/.local/lib/python2.7/site-packages/oauth2client/crypt.pyc in from_string(key, password)
    163       parsed_pem_key = _parse_pem_key(key)
    164       if parsed_pem_key:
--> 165         pkey = crypto.load_privatekey(crypto.FILETYPE_PEM, parsed_pem_key)
    166       else:
    167         if isinstance(password, six.text_type):

Error: [('PEM routines', 'PEM_read_bio', 'bad base64 decode')]

Python 3 gives me:

/tmp/ipython_edit_AuOR2A/ipython_edit_MAAUgC.py in login_using_gspread()
     13         )
     14 
---> 15     return gspread.authorize(credentials)

/usr/local/lib/python2.7/dist-packages/gspread/client.pyc in authorize(credentials)
    333     """
    334     client = Client(auth=credentials)
--> 335     client.login()
    336     return client
    337 

/usr/local/lib/python2.7/dist-packages/gspread/client.pyc in login(self)
     96 
     97                 http = httplib2.Http()
---> 98                 self.auth.refresh(http)
     99 
    100             self.session.add_header('Authorization', "Bearer " + self.auth.access_token)

/home/drew/.local/lib/python2.7/site-packages/oauth2client/client.pyc in refresh(self, http)
    596         request.
    597     """
--> 598     self._refresh(http.request)
    599 
    600   def revoke(self, http):

/home/drew/.local/lib/python2.7/site-packages/oauth2client/client.pyc in _refresh(self, http_request)
    767     """
    768     if not self.store:
--> 769       self._do_refresh_request(http_request)
    770     else:
    771       self.store.acquire_lock()

/home/drew/.local/lib/python2.7/site-packages/oauth2client/client.pyc in _do_refresh_request(self, http_request)
    793       AccessTokenRefreshError: When the refresh fails.
    794     """
--> 795     body = self._generate_refresh_request_body()
    796     headers = self._generate_refresh_request_headers()
    797 

/home/drew/.local/lib/python2.7/site-packages/oauth2client/client.pyc in _generate_refresh_request_body(self)
   1423 
   1424   def _generate_refresh_request_body(self):
-> 1425     assertion = self._generate_assertion()
   1426 
   1427     body = urllib.parse.urlencode({

/home/drew/.local/lib/python2.7/site-packages/oauth2client/client.pyc in _generate_assertion(self)
   1552     private_key = base64.b64decode(self.private_key)
   1553     return crypt.make_signed_jwt(crypt.Signer.from_string(
-> 1554         private_key, self.private_key_password), payload)
   1555 
   1556 # Only used in verify_id_token(), which is always calling to the same URI

/home/drew/.local/lib/python2.7/site-packages/oauth2client/crypt.pyc in from_string(key, password)
    163       parsed_pem_key = _parse_pem_key(key)
    164       if parsed_pem_key:
--> 165         pkey = crypto.load_privatekey(crypto.FILETYPE_PEM, parsed_pem_key)
    166       else:
    167         if isinstance(password, six.text_type):

Error: [('PEM routines', 'PEM_read_bio', 'bad base64 decode')]

The code to trigger it is this:

def login_using_gspread():
    GOOGLE_SPREADSHEET_CLIENT_EMAIL = 'bleble@developer.gserviceaccount.com'
    GOOGLE_SPREADSHEET_PRIVATE_KEY  = b"-----BEGIN PRIVATE KEY-----blabla\\u003d\n-----END PRIVATE KEY-----\n"
    credentials = SignedJwtAssertionCredentials(
        service_account_name = GOOGLE_SPREADSHEET_CLIENT_EMAIL,
        private_key          = GOOGLE_SPREADSHEET_PRIVATE_KEY,
        scope                = ['https://spreadsheets.google.com/feeds']
        )
    return gspread.authorize(credentials)
gc = login_using_gspread()

Note that the key must contain 'u003d' - again, I am happy to send a JSON file with this, but google throws such json files about 3 tries out of 5 using this guide: http://gspread.readthedocs.org/en/latest/oauth2.html

[dhermes]

So that error is just a bad decode caused by using a fake private key, hence it doesn't showcase the issue at hand.

We don't need you to send a real key, but we do need to see a real stacktrace from trying to use a real key and failing.

Also, are you copying and pasting your key into your file as above

VAR_NAME = b"..."

If yes, that may the be the issue (a copy-paste error). I recommend taking the file you downloaded from the Google developer's console and then using

with open(filename, 'rb') as file_obj:
    VAR_NAME = file_obj.read()

[felagund]

Yes, opening it like this and not storing it in a file works fine. Is copying it into a file not supported? Or how should I copy it to avoid this error?

The stacktrace that the copypasted real key gives is in my previous comment.

[dhermes]

Pretty much anything is supported, I'm just trying to figure out what is going wrong. Can you send the stacktrace that occurs with the actual key?

[felagund]

Python 3.4 gives me:

IPython will make a temporary file named: /tmp/ipython_edit_j13qjyh2/ipython_edit_7wp7q3ct.py
Editing... done. Executing edited code...
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/tmp/ipython_edit_j13qjyh2/ipython_edit_7wp7q3ct.py in <module>()
      7 scope = ['https://spreadsheets.google.com/feeds']
      8 credentials = SignedJwtAssertionCredentials(private_key=key,service_account_name=email, scope=scope)
----> 9 gc = gspread.authorize(credentials)
     10 
     11 def login_using_gspread():

/usr/local/lib/python3.4/dist-packages/gspread/client.py in authorize(credentials)
    333     """
    334     client = Client(auth=credentials)
--> 335     client.login()
    336     return client
    337 

/usr/local/lib/python3.4/dist-packages/gspread/client.py in login(self)
     96 
     97                 http = httplib2.Http()
---> 98                 self.auth.refresh(http)
     99 
    100             self.session.add_header('Authorization', "Bearer " + self.auth.access_token)

/usr/local/lib/python3.4/dist-packages/oauth2client/client.py in refresh(self, http)
    596         request.
    597     """
--> 598     self._refresh(http.request)
    599 
    600   def revoke(self, http):

/usr/local/lib/python3.4/dist-packages/oauth2client/client.py in _refresh(self, http_request)
    767     """
    768     if not self.store:
--> 769       self._do_refresh_request(http_request)
    770     else:
    771       self.store.acquire_lock()

/usr/local/lib/python3.4/dist-packages/oauth2client/client.py in _do_refresh_request(self, http_request)
    793       AccessTokenRefreshError: When the refresh fails.
    794     """
--> 795     body = self._generate_refresh_request_body()
    796     headers = self._generate_refresh_request_headers()
    797 

/usr/local/lib/python3.4/dist-packages/oauth2client/client.py in _generate_refresh_request_body(self)
   1423 
   1424   def _generate_refresh_request_body(self):
-> 1425     assertion = self._generate_assertion()
   1426 
   1427     body = urllib.parse.urlencode({

/usr/local/lib/python3.4/dist-packages/oauth2client/client.py in _generate_assertion(self)
   1552     private_key = base64.b64decode(self.private_key)
   1553     return crypt.make_signed_jwt(crypt.Signer.from_string(
-> 1554         private_key, self.private_key_password), payload)
   1555 
   1556 # Only used in verify_id_token(), which is always calling to the same URI

/usr/local/lib/python3.4/dist-packages/oauth2client/crypt.py in from_string(key, password)
    298       parsed_pem_key = _parse_pem_key(key)
    299       if parsed_pem_key:
--> 300         pkey = RSA.importKey(parsed_pem_key)
    301       else:
    302         raise NotImplementedError(

/usr/lib/python3/dist-packages/Crypto/PublicKey/RSA.py in importKey(self, externKey, passphrase)
    663                     padding = bord(der[-1])
    664                     der = der[:-padding]
--> 665                 return self._importKeyDER(der)
    666 
    667         if externKey.startswith(b('ssh-rsa ')):

/usr/lib/python3/dist-packages/Crypto/PublicKey/RSA.py in _importKeyDER(self, externKey)
    586             pass
    587 
--> 588         raise ValueError("RSA key format is not supported")
    589 
    590     def importKey(self, externKey, passphrase=None):

ValueError: RSA key format is not supported

After executing this:

# -*- coding: utf-8 -*-
from oauth2client.client import SignedJwtAssertionCredentials
import gspread
key1 = b"-----BEGIN PRIVATE KEY-----\nBLABLA_WITH_NO_\u003d\n-----END PRIVATE KEY-----\n"
key = b"-----BEGIN PRIVATE KEY-----\nBLABLA_INCLUDING_\u003d\n-----END PRIVATE KEY-----\n"
email = "blabla@developer.gserviceaccount.com"
scope = ['https://spreadsheets.google.com/feeds']
credentials = SignedJwtAssertionCredentials(private_key=key,service_account_name=email, scope=scope)
gc = gspread.authorize(credentials)

(using key1 in the code above works)

Running the same code on 2.7 keeps giving me:

Error                                     Traceback (most recent call last)
/tmp/ipython_edit_9Kfkcr/ipython_edit_MzUr6r.py in <module>()
      7 scope = ['https://spreadsheets.google.com/feeds']
      8 credentials = SignedJwtAssertionCredentials(private_key=key,service_account_name=email, scope=scope)
----> 9 gc = gspread.authorize(credentials)

/usr/local/lib/python2.7/dist-packages/gspread/client.pyc in authorize(credentials)
    333     """
    334     client = Client(auth=credentials)
--> 335     client.login()
    336     return client
    337 

/usr/local/lib/python2.7/dist-packages/gspread/client.pyc in login(self)
     96 
     97                 http = httplib2.Http()
---> 98                 self.auth.refresh(http)
     99 
    100             self.session.add_header('Authorization', "Bearer " + self.auth.access_token)

/home/drew/.local/lib/python2.7/site-packages/oauth2client/client.pyc in refresh(self, http)
    596         request.
    597     """
--> 598     self._refresh(http.request)
    599 
    600   def revoke(self, http):

/home/drew/.local/lib/python2.7/site-packages/oauth2client/client.pyc in _refresh(self, http_request)
    767     """
    768     if not self.store:
--> 769       self._do_refresh_request(http_request)
    770     else:
    771       self.store.acquire_lock()

/home/drew/.local/lib/python2.7/site-packages/oauth2client/client.pyc in _do_refresh_request(self, http_request)
    793       AccessTokenRefreshError: When the refresh fails.
    794     """
--> 795     body = self._generate_refresh_request_body()
    796     headers = self._generate_refresh_request_headers()
    797 

/home/drew/.local/lib/python2.7/site-packages/oauth2client/client.pyc in _generate_refresh_request_body(self)
   1423 
   1424   def _generate_refresh_request_body(self):
-> 1425     assertion = self._generate_assertion()
   1426 
   1427     body = urllib.parse.urlencode({

/home/drew/.local/lib/python2.7/site-packages/oauth2client/client.pyc in _generate_assertion(self)
   1552     private_key = base64.b64decode(self.private_key)
   1553     return crypt.make_signed_jwt(crypt.Signer.from_string(
-> 1554         private_key, self.private_key_password), payload)
   1555 
   1556 # Only used in verify_id_token(), which is always calling to the same URI

/home/drew/.local/lib/python2.7/site-packages/oauth2client/crypt.pyc in from_string(key, password)
    163       parsed_pem_key = _parse_pem_key(key)
    164       if parsed_pem_key:
--> 165         pkey = crypto.load_privatekey(crypto.FILETYPE_PEM, parsed_pem_key)
    166       else:
    167         if isinstance(password, six.text_type):

Error: [('PEM routines', 'PEM_read_bio', 'bad base64 decode')]

[dhermes]

The values

key1 = b"-----BEGIN PRIVATE KEY-----\nBLABLA_WITH_NO_\u003d\n-----END PRIVATE KEY-----\n"
key = b"-----BEGIN PRIVATE KEY-----\nBLABLA_INCLUDING_\u003d\n-----END PRIVATE KEY-----\n"

are not real encoded keys. Please test this with a real key and let us know if it fails.

---

Also note that using a bytestring won't convert your unicode character as you expect:

>>> b'\u003d'
'\\u003d'
>>> u'\u003d'
u'='

---

The things you are seeing work are just based on the fake contents you were using.

>>> import base64
>>> base64.b64decode(b'BLABLA_WITH_NO_\u003d')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/base64.py", line 76, in b64decode
    raise TypeError(msg)
TypeError: Incorrect padding
>>> base64.b64decode(b'BLABLA_INCLUDING_\u003d')
'\x04\xb0\x01,\x02\r\x08\xb5\x03 \xd1\xae\xd3M\xdd'

[felagund]

I am using real keys all along, of course. I posted the code snippets so that it would be clear what I was doing. I replaced the real keys so that somebody would not misuse them.

If I get a key from Google (as explained here: http://gspread.readthedocs.org/en/latest/oauth2.html#custom-credentials-objects ) that contains '\u003d' the code stops working.

As migurski pointed out above, one needs to use (again, with a valid code, which I am doing) this:

key = u"-----BEGIN PRIVATE KEY-----\nBLABLA_INCLUDING_\u003d\n-----END PRIVATE KEY-----\n".encode("utf-8")

So that the code works for both keys that contain '\u003d' and for those that don't contain it. This still very much feels like shotting myself in the foot

[dhermes]

So can you confirm that using a real key you are seeing

Error: [('PEM routines', 'PEM_read_bio', 'bad base64 decode')]

and

ValueError: RSA key format is not supported

---

How are you loading this variables? You won't see anything like \u003d in the file you downloaded since it is just bytes. It may be a copy-paste error, or maybe you are actually loading from the file and your example of key = ... was just for illustration?

Do you want to email me the key file so we can verify?

[felagund]

What is your e-mail address?

I actually have a .json file that I can open in a text editor. I copy pasted the key into my file. I can open it from a file like this:

import json
json_key = json.load(open('gspread-april-2cd … ba4.json'))
json_key['private_key'].encode("utf-8")

Yes, using the real key (as much as what I seen in the json file is real), I get the errors above, provided that I use the very code that I posted above with actual keys (using 'b''', not encode()).

[dhermes]

Can you try the following:

from oauth2client.client import _get_application_default_credential_from_file
json_credentials_path = 'gspread-april-2cd … ba4.json'
credentials = _get_application_default_credential_from_file(
    json_credentials_path)

and post a stack trace if there are issues?

You could also set the GOOGLE_APPLICATION_CREDENTIALS environment variable in bash / other shell:

export GOOGLE_APPLICATION_CREDENTIALS="gspread-april-2cd … ba4.json"

and then just execute

from oauth2client.client import GoogleCredentials
credentials = GoogleCredentials.get_application_default()

[felagund]

That seems to work fine. Maybe there is an issue with theinstructions here: http://gspread.readthedocs.org/en/latest/oauth2.html ?

[dhermes]

Yes. Closing this out. I'm happy to offer suggestions on the gspread issue you filed.

It seems your transcription of the key from the JSON file was the issue, but it's not necessary to do, given that you can just use the code above.

[felagund]

Yeah, I opened an issue about it: https://github.com/burnash/gspread/issues/244

BTW: this is what I see when I open the file I downloaded from googe in Leafpad (a text editor) - see the \u003d characters at the end of the key:

[dhermes]

Thanks a ton for showing this!

Those are base 64 padding characters (=) and there may actually be an error in Google somewhere in the creation of these files.

I'm not really sure where to report it. @craigcitro may know who to talk to.

[craigcitro]

summoning @anthmgoogle for all things auth and service account keys.

anthony, can you look into this?

[dhermes]

I'm curious that your editor (Leafpad) displays the newline characters are literal \n. There is no equivalent escape character \u and it is clear that the last two characters are meant to be base64 padding anyhow.

[felagund]

It is not just Leafpad, Gedit, Geany and even notepad.exe through wine and Libreoffice Writer display it the same. I am on Ubuntu 15.04, my locale is LANG=en_US.UTF-8, if it is relevant.

[dhermes]

Thanks for being comprehensive! I'm curious if you open the file as bytes in Python (open(filename, 'rb')) what content you see. I'd wager it's a backslash (\) character, then the character u, then 0, then 0, then 3, then d, but would like to see.

[felagund]

Yes:

In [39]: a = open('gspread-april-2cd … ba4.json', 'rb')

In [40]: s[993:]
Out[40]: b'003d\\n-----END PRIVATE KEY-----\\n",\n  "client_email": "704538...'

In [41]: s[994:]
Out[41]: b'03d\\n-----END PRIVATE KEY-----\\n",\n  "client_email": "704538...'

[dhermes]

We miss out on the \u, but it seems the \n is also literal. Can you print s[985:1049] as well?

PS I edited out the client email for you, not

[dhermes]

>>> import json
>>> json.dumps(u'\nSOMESTUFF\u003d')
'"\\nSOMESTUFF="'
>>> json.dumps(b'\nSOMESTUFF\u003d')
'"\\nSOMESTUFF\\\\u003d"'
>>> print json.dumps(b'\nSOMESTUFF\u003d')
"\nSOMESTUFF\\u003d"

[felagund]

Oh thanks, I thought that since I forgot it in the image I attached above, it would not matter anyway.

As requested:

In [13]: s[985:1049]
Out[13]: b'\\u003d\\u003d\\n-----END PRIVATE KEY-----\\n",\n  "client_email": "7'

In [14]: s[997:1049]
Out[14]: b'\\n-----END PRIVATE KEY-----\\n",\n  "client_email": "7'

In [15]: s[998:1049]
Out[15]: b'n-----END PRIVATE KEY-----\\n",\n  "client_email": "7'

[felagund]

Also note that I use Python 3 in these examples (but it should be the same, no?)

[dhermes]

Python 3 makes a difference most of the time when processing text, but not likely here since using 'rb' as our mode.

It does indeed seem to be an issue with the result returned @anthmgoogle

[dhermes]

Has an internal Google bug been filed with the identity team yet?

Easy to file issue:

Token base64 padding seems to be mixing bytes and unicode in Python.

User reported a private key which couldn't be read because the base64 padding bytes [1]
were `\u003d\u003d` (12 bytes total) instead of `==` (2 bytes total).

It seems something bad is happening, note that 0x3d == 61 and ord('=') = 61, so
these are the same character, e.g. in Python `u'\u003d' == '='`.

See https://github.com/google/oauth2client/issues/192.

[1]: https://cloud.githubusercontent.com/assets/1218098/8058106/4f71930e-0eb7-11e5-9929-b39c38718175.png

/cc @nathanielmanistaatgoogle @anthmgoogle

[felagund]

Any news on this?

[dhermes]

The issue is resolved. I'm unclear what you're asking for.

[felagund]

Well, then I guess it should be reopened. You closed it but then we discovered that the issue was beacuse google was giving out files with \u003d instead of = - we tried to contact google about it and I am not sure if it was resolved or not, but we never reopened the issue (see the discusion after June 9).

[dhermes]

It's not an issue. The issue was the way you copied and pasted.

Check out https://gist.github.com/dhermes/2e4137ccf717b98cfe5e to see that the \u003d characters are loaded correctly.

[felagund]

So Google claims that it is fine if \u003d is shown in the file instead of = when one opens the file in a text editor as per https://github.com/google/oauth2client/issues/192#issuecomment-110348735 ?

[migurski]

\u003d and = are equivalent in JSON, yes.

[dhermes]

As @migurski says, they are equivalent. I don't know what Google claims, but the gist I linked to shows that it works just fine (as long as you don't copy-paste and get something different).

[felagund]

Hm, but in the text file that I downloaded from Google, \u003d is 6 characters, not one. You yourself, dhermes, claimed it is a problem in this comment: https://github.com/google/oauth2client/issues/192#issuecomment-114617984 and subsequently tried to contact google, but @anthmgoogle never responded. I am confused as what has changed?

Anyway, when I try to follow the gist you linked, I get:

<ipython-input-8-675e9bdd2720> in <module>()
      1 with open('vytvor-zakazku-afe9a29f6563.json', 'rb') as fh:
----> 2     contents = json.load(fh)
      3 

/usr/lib/python3.4/json/__init__.py in load(fp, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    266         cls=cls, object_hook=object_hook,
    267         parse_float=parse_float, parse_int=parse_int,
--> 268         parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
    269 
    270 

/usr/lib/python3.4/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    310     if not isinstance(s, str):
    311         raise TypeError('the JSON object must be str, not {!r}'.format(
--> 312                             s.__class__.__name__))
    313     if s.startswith(u'\ufeff'):
    314         raise ValueError("Unexpected UTF-8 BOM (decode using utf-8-sig)")

TypeError: the JSON object must be str, not 'bytes'

I did not copypasted anything. I downloaded a new json file from https://console.developers.google.com/project using Firefox on Ubuntu (might that be a problem?) and opened it with

with open('new_file.json', 'rb') as fh:
        contents = json.load(fh)

[dhermes]

Yes I did say that before, but I have since actually tried to load the file and noticed that Python's JSON library handles it just fine. I still contend that Google should fix it, but am / was just saying it's something you can work around.

I didn't run it in Python 3 and the previous version of the gist failed (for different reasons in 3.3 and 3.4). Thanks for pointing it out! I have updated the gist and it works just fine.

[felagund]

Ok, now I understand. Of course this can be worked-around, we know that since at least https://github.com/google/oauth2client/issues/192#issuecomment-107700166 Indeed, GIST works now:-).

Anyway, I was asking about news on Google's side, if they know about it and if they are going to fix it. Since I tried to download a new key today, they have apparently not fixed it and are unresponsive so far:-/.

[dhermes]

Hopefully someone will weigh in (I wrote the bug report for them)

[nathanielmanistaatgoogle]

@dhermes sorry for this not getting the attention it needs. Thank you for the bug report text but against what team/product/public API would you like that report filed?

[dhermes]

File it with the auth people (or whoever else owns https://accounts.google.com/o/oauth2/auth and similar).

They are base64 padding with \u003d instead of =.

As mentioned, libraries can handle this, but it makes no sense since = (i.e. 0x3d == 61) is an ascii char.

[nathanielmanistaatgoogle]

Filed internally, finally, and sorry again for the wait.

[dhermes]

Thanks a lot. There was also a pre-written bug to file about error messages in https://github.com/google/oauth2client/issues/193#issuecomment-114618536.

[anthmgoogle]

Sorry I did not notice the ping to me on this issue earlier.

I just tested the downloaded files from the Google Developer's console, and I'm not reproducing the indicated problem with \u003d instead of "=". I've also not seen this in other languages. Is it possible that there is conversion when the JSON is loaded up in Python?

Note that the usage scenario above is with the SignJwtAccessCredentials, which is not in the intended usge. Intended usage is with GoogleCredentials.from_stream or get_application_default. If the files work with this but not SignedJwtAccessCrdentials that would be why. It may be possible to special case this to work around it, but there is also a plan to deprecate SignedJwtAccessCredentials and make _ServiceAccountCredentials public.

[anthmgoogle]

Update. I was able to reproduce this from the console. I'll poke on the internal version.

[dhermes]

@anthmgoogle There is no issue with the library. The JSON parser is able to properly convert the chracters to =.

The only remaining issue was that it shouldn't be occurring. (IIRC the real issue here was a copy-paste that accidentally changed an encoding.)