gcsfs does not work with google-auth-oauthlib v0.5.3+

[andersy005]

When using gcsfs with the latest version of google-auth-oauthlib, i run into this error.

In [1]: import gcsfs

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In [1], line 1
----> 1 import gcsfs

File /srv/conda/envs/notebook/lib/python3.9/site-packages/gcsfs/__init__.py:5
      3 __version__ = get_versions()["version"]
      4 del get_versions
----> 5 from .core import GCSFileSystem
      6 from .mapping import GCSMap
      8 __all__ = ["GCSFileSystem", "GCSMap"]

File /srv/conda/envs/notebook/lib/python3.9/site-packages/gcsfs/core.py:23
     21 from .retry import retry_request, validate_response
     22 from .checkers import get_consistency_checker
---> 23 from .credentials import GoogleCredentials
     24 from . import __version__ as version
     25 from urllib.parse import quote as quote_urllib

File /srv/conda/envs/notebook/lib/python3.9/site-packages/gcsfs/credentials.py:8
      6 import google.auth.exceptions
      7 from google.oauth2.credentials import Credentials
----> 8 from google_auth_oauthlib.flow import InstalledAppFlow
      9 from google.oauth2 import service_account
     10 from google.auth.transport.requests import Request

File /srv/conda/envs/notebook/lib/python3.9/site-packages/google_auth_oauthlib/__init__.py:21
      1 # Copyright 2019 Google LLC
      2 #
      3 # Licensed under the Apache License, Version 2.0 (the "License");
   (...)
     12 # See the License for the specific language governing permissions and
     13 # limitations under the License.
     15 """oauthlib integration for Google Auth
     16 
     17 This library provides `oauthlib <https://oauthlib.readthedocs.io/>`__
     18 integration with `google-auth <https://google-auth.readthedocs.io/>`__.
     19 """
---> 21 from .interactive import get_user_credentials
     23 __all__ = ["get_user_credentials"]

File /srv/conda/envs/notebook/lib/python3.9/site-packages/google_auth_oauthlib/interactive.py:27
     24 import contextlib
     25 import socket
---> 27 import google_auth_oauthlib.flow
     30 LOCALHOST = "localhost"
     31 DEFAULT_PORTS_TO_TRY = 100

File /srv/conda/envs/notebook/lib/python3.9/site-packages/google_auth_oauthlib/flow.py:69
     66 import google.auth.transport.requests
     67 import google.oauth2.credentials
---> 69 import google_auth_oauthlib.helpers
     72 _LOGGER = logging.getLogger(__name__)
     73 _OOB_REDIRECT_URIS = [
     74     "urn:ietf:wg:oauth:2.0:oob",
     75     "urn:ietf:wg:oauth:2.0:oob:auto",
     76     "oob",
     77 ]

File /srv/conda/envs/notebook/lib/python3.9/site-packages/google_auth_oauthlib/helpers.py:27
     24 import datetime
     25 import json
---> 27 from google.auth import external_account_authorized_user
     28 import google.oauth2.credentials
     29 import requests_oauthlib

ImportError: cannot import name 'external_account_authorized_user' from 'google.auth' (/srv/conda/envs/notebook/lib/python3.9/site-packages/google/auth/__init__.py)

I have tried downgrading to version 0.6.0 of "google-auth-oauthlib" without success. i was able to import gcsfs after downgrading google-auth-oauthlib to v0.5.3. The newer versions of Google-auth-oauthlib were released after this repository's latest CI runs. This is probably why the issue didn't arise sooner.

Broken Environment

```bash (notebook) root@80c07463551d:~# conda list gcsfs # packages in environment at /srv/conda/envs/notebook: # # Name Version Build Channel gcsfs 2022.10.0 pyhd8ed1ab_0 conda-forge (notebook) root@80c07463551d:~# conda list fsspec # packages in environment at /srv/conda/envs/notebook: # # Name Version Build Channel fsspec 2022.10.0 pyhd8ed1ab_0 conda-forge (notebook) root@80c07463551d:~# conda list google # packages in environment at /srv/conda/envs/notebook: # # Name Version Build Channel google-api-core 2.10.1 pyhd8ed1ab_0 conda-forge google-api-core-grpc 2.10.1 hd8ed1ab_0 conda-forge google-apitools 0.5.31 pyh9f0ad1d_0 conda-forge google-auth 1.35.0 pyh6c4a22f_0 conda-forge google-auth-httplib2 0.1.0 pyhd8ed1ab_1 conda-forge google-auth-oauthlib 0.7.0 pyhd8ed1ab_0 conda-forge google-cloud-bigquery-core 2.34.4 pyh44b312d_0 conda-forge google-cloud-bigquery-storage 2.11.0 pyh6c4a22f_0 conda-forge google-cloud-bigquery-storage-core 2.11.0 pyh6c4a22f_0 conda-forge google-cloud-bigtable 1.7.1 pyhd8ed1ab_0 conda-forge google-cloud-core 2.3.2 pyhd8ed1ab_0 conda-forge google-cloud-datastore 1.15.5 pyhd8ed1ab_0 conda-forge google-cloud-dlp 3.9.0 pyhd8ed1ab_0 conda-forge google-cloud-language 1.3.1 pyh6c4a22f_0 conda-forge google-cloud-pubsub 2.13.7 pyhd8ed1ab_0 conda-forge google-cloud-pubsublite 1.6.0 pyhd8ed1ab_0 conda-forge google-cloud-recommendations-ai 0.7.1 pyhd8ed1ab_0 conda-forge google-cloud-spanner 1.19.3 pyhd8ed1ab_0 conda-forge google-cloud-storage 2.5.0 pyh6c4a22f_0 conda-forge google-cloud-videointelligence 1.16.2 pyhd8ed1ab_0 conda-forge google-cloud-vision 1.0.1 pyhd8ed1ab_0 conda-forge google-crc32c 1.1.2 py39h8db88ab_4 conda-forge google-resumable-media 2.4.0 pyhd8ed1ab_0 conda-forge googleapis-common-protos 1.56.4 py39hf3d152e_1 conda-forge googleapis-common-protos-grpc 1.56.4 hf3d152e_1 conda-forge grpc-google-iam-v1 0.12.4 py39hf3d152e_0 conda-forge libgoogle-cloud 2.1.0 h9ebe8e8_2 conda-forge ```

Working environment

```bash (notebook) root@9c6cc1082cce:~# conda list google # packages in environment at /srv/conda/envs/notebook: # # Name Version Build Channel google-api-core 2.10.1 pyhd8ed1ab_0 conda-forge google-api-core-grpc 2.10.1 hd8ed1ab_0 conda-forge google-apitools 0.5.31 pyh9f0ad1d_0 conda-forge google-auth 1.35.0 pyh6c4a22f_0 conda-forge google-auth-httplib2 0.1.0 pyhd8ed1ab_1 conda-forge google-auth-oauthlib 0.5.3 pyhd8ed1ab_0 conda-forge google-cloud-bigquery-core 2.34.4 pyh44b312d_0 conda-forge google-cloud-bigquery-storage 2.11.0 pyh6c4a22f_0 conda-forge google-cloud-bigquery-storage-core 2.11.0 pyh6c4a22f_0 conda-forge google-cloud-bigtable 1.7.1 pyhd8ed1ab_0 conda-forge google-cloud-core 2.3.2 pyhd8ed1ab_0 conda-forge google-cloud-datastore 1.15.5 pyhd8ed1ab_0 conda-forge google-cloud-dlp 3.9.0 pyhd8ed1ab_0 conda-forge google-cloud-language 1.3.1 pyh6c4a22f_0 conda-forge google-cloud-pubsub 2.13.7 pyhd8ed1ab_0 conda-forge google-cloud-pubsublite 1.6.0 pyhd8ed1ab_0 conda-forge google-cloud-recommendations-ai 0.7.1 pyhd8ed1ab_0 conda-forge google-cloud-spanner 1.19.3 pyhd8ed1ab_0 conda-forge google-cloud-storage 2.5.0 pyh6c4a22f_0 conda-forge google-cloud-videointelligence 1.16.2 pyhd8ed1ab_0 conda-forge google-cloud-vision 1.0.1 pyhd8ed1ab_0 conda-forge google-crc32c 1.1.2 py39h8db88ab_4 conda-forge google-resumable-media 2.4.0 pyhd8ed1ab_0 conda-forge googleapis-common-protos 1.56.4 py39hf3d152e_1 conda-forge googleapis-common-protos-grpc 1.56.4 hf3d152e_1 conda-forge grpc-google-iam-v1 0.12.4 py39hf3d152e_0 conda-forge libgoogle-cloud 2.1.0 h9ebe8e8_2 conda-forge (notebook) root@9c6cc1082cce:~# ```

[martindurant]

Appears to work if you also install google-auth==2.13. Given that 0.7.0 so closely followed 0.6.0, seems like google messed something up here.

[andersy005]

Appears to work if you also install google-auth==2.13

My attempt at install google-auth=2.13 fails due to another dependency in my environment that seems not to support v2+

Encountered problems while solving:
  - package apache-beam-with-gcp-2.42.0-py39hf3d152e_0 requires google-auth >=1.18.0,<2, but none of the providers can be installed

This is good to know though.

[martindurant]

Ah, ok. So that ^ is the principle problem; but also google got wrong the version constraints in its release.

[yuvipanda]

Anyone know if this has been fixed? :(

[weiji14]

Looking at the most recent CI run at https://github.com/fsspec/gcsfs/actions/runs/5244478235/jobs/9470552946#step:4:41, I see google-auth-oauthlib=1.0.0 installed and the tests seem to be passing. Also tried this locally:

mamba create --name temp python=3.10 google-auth-oauthlib=1.0.0 gcsfs=2023.6.0
mamba activate temp
python -c "import gcsfs; print(gcsfs.__version__)"

which printed 2023.6.0, so I guess import gcsfs should be working again?

[martindurant]

@andersy005 , please close if all is well.