google.auth.default() returns None with Python 3.12, but works fine with Python 3.11

[mezhaka]

Please run down the following list and make sure you've tried the usual "quick fixes":

• OS: MacOS 13.5.2
• Python version: 3.12
• google-auth version: 2.23.0

Steps to reproduce

I use conda.

1. Create environment.yml file with the following content:

   name: auth-bug channels:

   • conda-forge dependencies:
   • python ~= 3.12.0
   • google-auth=2.23.3

2. Create environment conda env create and activate it conda activate auth-bug

3. Run python and try:

   import google.auth print(google.auth.default()) (<google.oauth2.credentials.Credentials object at 0x1057e1640>, None)

As soon as I switch to Python 3.11 in the environment, I get back a valid GCP project instead of None.

Happy to provide more details.

[clundin25]

I am not so familiar with conda, but I was unable to reproduce this. My steps:

Checking 3.12

pyenv install 3.12.0
pyenv shell 3.12.0 && createpyenv
python -m pip install google-auth
python --version && python -c "import google.auth;print(google.auth.default())"
Python 3.12.0
(<google.oauth2.credentials.Credentials object at 0x7f7e011a8680>, '$REDACTED')

Checking 3.11

pyenv install 3.11.6
pyenv shell 3.11.6 && createpyenv
python -m pip install google-auth
python --version && python -c "import google.auth;print(google.auth.default())"
Python 3.11.6
(<google.oauth2.credentials.Credentials object at 0x7f7848bd6510>, '$REDACTED')

Can you tell me a bit more about your setup? It seems you are running locally on MacOS, are you using the application default credentials created by gcloud?

https://cloud.google.com/docs/authentication/application-default-credentials

[clundin25]

Note: We do not yet have an official release that supports 3.12. There may be some broken features.

[mezhaka]

@clundin25 Hey, thank you for taking your time to look into it. It is indeed remarkable that you get a different result. I have no idea, however, if this is conda related or somehow a side effect of other installed libs...

I am using application default credentials, that I have obtained running gcloud auth application-default login.

The only thing that occures to me is I can provide the export of all the libs installed obtained with conda list --export

I generate both environments from the same environment file that I have provided, the only difference is the Python version:

3.11

$ cat environment.yml
name: auth-bug-ok
channels:
  - conda-forge
dependencies:
  - python ~= 3.11.0
  - google-auth=2.23.3

$ cat conda.export
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: osx-64
aiohttp=3.8.6=py311he705e18_1
aiosignal=1.3.1=pyhd8ed1ab_0
async-timeout=4.0.3=pyhd8ed1ab_0
attrs=23.1.0=pyh71513ae_1
brotli-python=1.1.0=py311hdf8f085_1
bzip2=1.0.8=h0d85af4_4
ca-certificates=2023.7.22=h8857fd0_0
cachetools=5.3.2=pyhd8ed1ab_0
certifi=2023.7.22=pyhd8ed1ab_0
cffi=1.16.0=py311hc0b63fd_0
charset-normalizer=3.3.1=pyhd8ed1ab_0
cryptography=41.0.5=py311hd51016d_0
frozenlist=1.4.0=py311h2725bcf_1
google-auth=2.23.3=pyhca7485f_0
idna=3.4=pyhd8ed1ab_0
libcxx=16.0.6=hd57cbcb_0
libexpat=2.5.0=hf0c8a7f_1
libffi=3.4.2=h0d85af4_5
libsqlite=3.43.2=h92b6c6a_0
libzlib=1.2.13=h8a1eda9_5
multidict=6.0.4=py311h5547dcb_1
ncurses=6.4=hf0c8a7f_0
openssl=3.1.4=hd75f5a5_0
pip=23.3.1=pyhd8ed1ab_0
pyasn1=0.5.0=pyhd8ed1ab_0
pyasn1-modules=0.3.0=pyhd8ed1ab_0
pycparser=2.21=pyhd8ed1ab_0
pyopenssl=23.2.0=pyhd8ed1ab_1
pysocks=1.7.1=pyha2e5f31_6
python=3.11.6=h30d4d87_0_cpython
python_abi=3.11=4_cp311
pyu2f=0.1.5=pyhd8ed1ab_0
readline=8.2=h9e318b2_1
requests=2.31.0=pyhd8ed1ab_0
rsa=4.9=pyhd8ed1ab_0
setuptools=68.2.2=pyhd8ed1ab_0
six=1.16.0=pyh6c4a22f_0
tk=8.6.13=hef22860_0
typing-extensions=4.8.0=hd8ed1ab_0
typing_extensions=4.8.0=pyha770c72_0
tzdata=2023c=h71feb2d_0
urllib3=2.0.7=pyhd8ed1ab_0
wheel=0.41.3=pyhd8ed1ab_0
xz=5.2.6=h775f41a_0
yarl=1.9.2=py311he705e18_1

with this setup, I am able to get the project:

$ python --version && python -c "import google.auth;print(google.auth.default())"
Python 3.11.6
(<google.oauth2.credentials.Credentials object at 0x109e08990>, 'something-meaningful')

3.12

name: auth-bug-not-ok
channels:
  - conda-forge
dependencies:
  - python ~= 3.12.0
  - google-auth=2.23.3

full environment:

$ cat conda.export
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: osx-64
aiohttp=3.9.0b0=py312h41838bb_0
aiosignal=1.3.1=pyhd8ed1ab_0
attrs=23.1.0=pyh71513ae_1
brotli-python=1.1.0=py312heafc425_1
bzip2=1.0.8=h0d85af4_4
ca-certificates=2023.7.22=h8857fd0_0
cachetools=5.3.2=pyhd8ed1ab_0
certifi=2023.7.22=pyhd8ed1ab_0
cffi=1.16.0=py312h38bf5a0_0
charset-normalizer=3.3.1=pyhd8ed1ab_0
cryptography=41.0.5=py312h68f415e_0
frozenlist=1.4.0=py312h104f124_1
google-auth=2.23.3=pyhca7485f_0
idna=3.4=pyhd8ed1ab_0
libcxx=16.0.6=hd57cbcb_0
libexpat=2.5.0=hf0c8a7f_1
libffi=3.4.2=h0d85af4_5
libsqlite=3.43.2=h92b6c6a_0
libzlib=1.2.13=h8a1eda9_5
multidict=6.0.4=py312h97956c7_1
ncurses=6.4=hf0c8a7f_0
openssl=3.1.4=hd75f5a5_0
pip=23.3.1=pyhd8ed1ab_0
pyasn1=0.5.0=pyhd8ed1ab_0
pyasn1-modules=0.3.0=pyhd8ed1ab_0
pycparser=2.21=pyhd8ed1ab_0
pyopenssl=23.2.0=pyhd8ed1ab_1
pysocks=1.7.1=pyha2e5f31_6
python=3.12.0=h30d4d87_0_cpython
python_abi=3.12=4_cp312
pyu2f=0.1.5=pyhd8ed1ab_0
readline=8.2=h9e318b2_1
requests=2.31.0=pyhd8ed1ab_0
rsa=4.9=pyhd8ed1ab_0
setuptools=68.2.2=pyhd8ed1ab_0
six=1.16.0=pyh6c4a22f_0
tk=8.6.13=hef22860_0
tzdata=2023c=h71feb2d_0
urllib3=2.0.7=pyhd8ed1ab_0
wheel=0.41.3=pyhd8ed1ab_0
xz=5.2.6=h775f41a_0
yarl=1.9.2=py312h41838bb_1

with this setup I get:

$ python --version && python -c "import google.auth;print(google.auth.default())"
Python 3.12.0
(<google.oauth2.credentials.Credentials object at 0x10ee8aa20>, None)

[clundin25]

Hmm interesting. Have you set the quota project in the default credentials?

Here is how:$ gcloud auth application-default set-quota-project $YOUR_PROJECT_ID.

The auth code will then inspect the file to determine the project. It is curious that 3.11 works and 3.12 does not.

I did reproduce in a slightly different environment, so if the above does not solve the issue I will do the same steps on my Mac.

[arithmetic1728]

For authorized user credentials, auth lib fetches the project id using gcloud config get project command. If you run this gcloud command, what do you see in 3.11 and 3.12?

https://github.com/googleapis/google-auth-library-python/blob/main/google/auth/_cloud_sdk.py#L92

[mezhaka]

@clundin25 I just tried to run the set-quota-project command that you have provided, but I still get None for the project:

$ gcloud auth application-default set-quota-project happy-panda-161123

Credentials saved to file: [/Users/ant/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).

Quota project "happy-panda-161123" was added to ADC which can be used by Google client libraries for billing and quota. Note that some services may still bill the project owning the resource.

$ python --version && python -c "import google.auth;print(google.auth.default())"
Python 3.12.0
(<google.oauth2.credentials.Credentials object at 0x107870f50>, None)

[mezhaka]

@arithmetic1728 I substitute my actual project name for happy-panda-161123

$ python --version && python -c "import google.auth._cloud_sdk;print(google.auth._cloud_sdk.get_project_id())"
Python 3.12.0
None

$ python --version && python -c "import google.auth._cloud_sdk;print(google.auth._cloud_sdk.get_project_id())"
Python 3.11.6
happy-panda-161123

[arithmetic1728]

@mezhaka can you run gcloud config get project command directly under these two environments? It's likely a gcloud issue instead of auth lib issue.

[mezhaka]

@arithmetic1728 Indeed, I get:

$ python --version && gcloud config get project
Python 3.12.0
Traceback (most recent call last):
  File "/Users/ant/google-cloud-sdk/lib/gcloud.py", line 137, in <module>
    main()
  File "/Users/ant/google-cloud-sdk/lib/gcloud.py", line 90, in main
    from googlecloudsdk.core.util import encoding
  File "/Users/ant/google-cloud-sdk/lib/googlecloudsdk/__init__.py", line 23, in <module>
    from googlecloudsdk.core.util import importing
  File "/Users/ant/google-cloud-sdk/lib/googlecloudsdk/core/util/importing.py", line 23, in <module>
    import imp
ModuleNotFoundError: No module named 'imp'

while in 3.11 it works just fine. The gcloud --version was 449.0.0. I have updated to the latest 453.0.0 and everything works just fine now. It did not occur to me that the Python module was actually invoking gcloud.