Failed to access to GCP bucket

[rkakamilan]

I tried to load a model from gs://molfeat-store-prod/artifacts/ by dm.fs.glob, but I got the following error:

2023-05-10 10:01:35 | ERROR | gcsfs | _request out of retries on exception: ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})
Traceback (most recent call last):
  File "/Users/rkakamilan/miniconda3/envs/molfeathype/lib/python3.10/site-packages/gcsfs/retry.py", line 114, in retry_request
    return await func(*args, **kwargs)
  File "/Users/rkakamilan/miniconda3/envs/molfeathype/lib/python3.10/site-packages/gcsfs/core.py", line 413, in _request
    headers=self._get_headers(headers),
  File "/Users/rkakamilan/miniconda3/envs/molfeathype/lib/python3.10/site-packages/gcsfs/core.py", line 392, in _get_headers
    self.credentials.apply(out)
  File "/Users/rkakamilan/miniconda3/envs/molfeathype/lib/python3.10/site-packages/gcsfs/credentials.py", line 185, in apply
    self.maybe_refresh()
  File "/Users/rkakamilan/miniconda3/envs/molfeathype/lib/python3.10/site-packages/gcsfs/credentials.py", line 180, in maybe_refresh
    self.credentials.refresh(req)
  File "/Users/rkakamilan/miniconda3/envs/molfeathype/lib/python3.10/site-packages/google/oauth2/credentials.py", line 335, in refresh
    ) = reauth.refresh_grant(
  File "/Users/rkakamilan/miniconda3/envs/molfeathype/lib/python3.10/site-packages/google/oauth2/reauth.py", line 351, in refresh_grant
    _client._handle_error_response(response_data, retryable_error)
  File "/Users/rkakamilan/miniconda3/envs/molfeathype/lib/python3.10/site-packages/google/oauth2/_client.py", line 73, in _handle_error_response
    raise exceptions.RefreshError(
google.auth.exceptions.RefreshError: ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})

I can access there by gsutil as follows:

$ gsutil ls gs://molfeat-store-prod/artifacts/
gs://molfeat-store-prod/artifacts/all/
gs://molfeat-store-prod/artifacts/dgllife/
gs://molfeat-store-prod/artifacts/fp/
gs://molfeat-store-prod/artifacts/graphormer/
gs://molfeat-store-prod/artifacts/huggingface/
gs://molfeat-store-prod/artifacts/pharmacophore/
gs://molfeat-store-prod/artifacts/rdkit/
gs://molfeat-store-prod/artifacts/shape/

I would greatly appreciate your guidance on how to proceed in this situation.

[maclandrol]

Hello @rkakamilan,

Could you share your version of fsspec, gcsfs and the datamol packages you are using ?

If possible also share the output of mamba list | grep google

My hunch is that this is due to an ongoing issue with google authentication system. Could you try installing the following in your environment google-auth >=2.13.0 and see if it fixes it ?

[rkakamilan]

@maclandrol Thanks for your comments.

The results of mamba list | grep xxxx

google-api-core           2.11.0             pyhd8ed1ab_0    conda-forge
google-auth               2.18.0             pyh1a96a4e_0    conda-forge
google-auth-oauthlib      1.0.0              pyhd8ed1ab_0    conda-forge
google-cloud-core         2.3.2              pyhd8ed1ab_0    conda-forge
google-cloud-storage      2.9.0              pyh1a96a4e_0    conda-forge
google-crc32c             1.1.2           py310h6f7428f_4    conda-forge
google-resumable-media    2.5.0              pyhd8ed1ab_0    conda-forge
googleapis-common-protos  1.57.1             pyhd8ed1ab_0    conda-forge
libgoogle-cloud           2.10.0               h20eaa6e_0    conda-forge

fsspec                    2023.5.0           pyh1a96a4e_0    conda-forge
gcsfs                     2023.5.0           pyhd8ed1ab_0    conda-forge

I tried the below, but it didn't worked.

• my conda env again and installation by pip.
• downgrading google-auth to 2.13.0

[maclandrol]

@rkakamilan I am trying to reproduce your error, but I can't. Can you post your full conda env.yml file ? Also are you using both gcloud and gsutil ?

[hadim]

@rkakamilan I cannot reproduce on my local machine, but it might be due to my local machine being logged with a gcloud account. So, I tried to reproduce in a GitHub Codespace where gcloud is not available and I cannot reproduce. Looking at the libs on the codespace session, all the versions are the same except for libgoogle-cloud (but I also cannot reproduce using the same version as you).

$ micromamba list | grep google
  google-api-core                2.11.0        pyhd8ed1ab_0          conda-forge
  google-auth                    2.18.0        pyh1a96a4e_0          conda-forge
  google-auth-oauthlib           1.0.0         pyhd8ed1ab_0          conda-forge
  google-cloud-core              2.3.2         pyhd8ed1ab_0          conda-forge
  google-cloud-storage           2.9.0         pyh1a96a4e_0          conda-forge
  google-crc32c                  1.1.2         py311h98db957_4       conda-forge
  google-resumable-media         2.5.0         pyhd8ed1ab_0          conda-forge
  googleapis-common-protos       1.57.1        pyhd8ed1ab_0          conda-forge
  libgoogle-cloud                2.8.0         hac9eb74_2            conda-forge

[rkakamilan]

@maclandrol @hadim

Thank you for conducting various reproduction experiments.

Based on your comments, I suspected that there might be an issue with my gcloud configuration. I performed the following steps, which resolved the problem. Firstly, on my PC, I executed gcloud auth revoke and then used gcloud config configuration delete to remove all unnecessary configurations. As a result, I was able to access the bucket. It appears that the problem stemmed from expired information that remained in my gcloud settings.

Thanks again for all your efforts on my issue.