gsutil does not maintain correct encoding for file path to adc.json

[codekoenig]

I have an umlaut in my Windows User Profile name & path:

C:\Users\bernhardkönig\

When I try to use gsutil I get an authentication error despite being correctly logged in:

Your "OAuth 2.0 Service Account" credentials are invalid. Please run
  $ gcloud auth login
OSError: No such file or directory.

When looking at the debug output of gsutil, it can be seen it does not use the correct file path encoding for at least the adc.json file and the Umlaut in my path is replaced by gibberish:

FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\bernhardkönig\\AppData\\Roaming\\gcloud\\legacy_credentials\\***\\adc.json'

Obviously, the file cannot be found with this incorrect path which leads to the error.

Here is the full debug log:

gsutil version: 5.10
boto version: 2.49.0
python version: 3.9.12 (tags/v3.9.12:b28265d, Mar 23 2022, 23:52:46) [MSC v.1929 64 bit (AMD64)]
OS: Windows 10
multiprocessing available: False
using cloud sdk: True
pass cloud sdk credentials to gsutil: True
config path(s): C:\Users\bernhardkönig\.boto, C:\Users\bernhardkönig\AppData\Roaming\gcloud\legacy_credentials\***\.boto
gsutil path: C:\Users\bernhardkönig\AppData\Local\Google\Cloud SDK\google-cloud-sdk\bin\gsutil
compiled crcmod: True
installed via package manager: False
editable install: False
Command being run: C:\Users\bernhardkönig\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform/gsutil\gsutil -o GSUtil:default_project_id=*** -D ls
config_file_list: ['C:\\Users\\bernhardkönig\\.boto', 'C:\\Users\\bernhardkönig\\AppData\\Roaming\\gcloud\\legacy_credentials\\***\\.boto']
config: [('working_dir', '/mnt/pyami'), ('debug', '0'), ('https_validate_certificates', 'True'), ('working_dir', '/mnt/pyami'), ('debug', '0'), ('content_language', 'en'), ('default_api_version', '2'), ('default_project_id', '***')]
Traceback (most recent call last):
  File "C:\Users\bernhardkönig\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\gsutil\gslib\gcs_json_credentials.py", line 213, in _CheckAndGetCredentials
    service_account_creds = _GetOauth2ServiceAccountCredentials()
  File "C:\Users\bernhardkönig\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\gsutil\gslib\gcs_json_credentials.py", line 302, in _GetOauth2ServiceAccountCredentials
    with io.open(private_key_filename, 'rb') as private_key_file:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\bernhardkönig\\AppData\\Roaming\\gcloud\\legacy_credentials\\***\\adc.json'

Your "OAuth 2.0 Service Account" credentials are invalid. Please run
  $ gcloud auth login
DEBUG: Exception stack trace:
    Traceback (most recent call last):
      File "C:\Users\bernhardkönig\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\gsutil\gslib\__main__.py", line 632, in _RunNamedCommandAndHandleExceptions
        return command_runner.RunNamedCommand(command_name,
      File "C:\Users\bernhardkönig\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\gsutil\gslib\command_runner.py", line 421, in RunNamedCommand
        return_code = command_inst.RunCommand()
      File "C:\Users\bernhardkönig\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\gsutil\gslib\commands\ls.py", line 616, in RunCommand
        for blr in self.WildcardIterator(
      File "C:\Users\bernhardkönig\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\gsutil\gslib\wildcard_iterator.py", line 521, in IterBuckets
        for blr in self._ExpandBucketWildcards(bucket_fields=bucket_fields):
      File "C:\Users\bernhardkönig\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\gsutil\gslib\wildcard_iterator.py", line 437, in _ExpandBucketWildcards
        for bucket in self.gsutil_api.ListBuckets(
      File "C:\Users\bernhardkönig\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\gsutil\gslib\cloud_api_delegator.py", line 238, in ListBuckets
        return self._GetApi(provider).ListBuckets(project_id=project_id,
      File "C:\Users\bernhardkönig\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\gsutil\gslib\cloud_api_delegator.py", line 119, in _GetApi
        self._LoadApi(provider, api_selector)
      File "C:\Users\bernhardkönig\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\gsutil\gslib\cloud_api_delegator.py", line 137, in _LoadApi
        self.api_map[ApiMapConstants.API_MAP][provider][api_selector](
      File "C:\Users\bernhardkönig\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\gsutil\gslib\gcs_json_api.py", line 217, in __init__
        SetUpJsonCredentialsAndCache(self, logger, credentials=credentials)
      File "C:\Users\bernhardkönig\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\gsutil\gslib\gcs_json_credentials.py", line 139, in SetUpJsonCredentialsAndCache
        api.credentials = (credentials or _CheckAndGetCredentials(logger) or
      File "C:\Users\bernhardkönig\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\gsutil\gslib\gcs_json_credentials.py", line 213, in _CheckAndGetCredentials
        service_account_creds = _GetOauth2ServiceAccountCredentials()
      File "C:\Users\bernhardkönig\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\gsutil\gslib\gcs_json_credentials.py", line 302, in _GetOauth2ServiceAccountCredentials
        with io.open(private_key_filename, 'rb') as private_key_file:
    FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\bernhardkönig\\AppData\\Roaming\\gcloud\\legacy_credentials\\***\\adc.json'

OSError: No such file or directory.

[thomasmaclean]

Thanks for reporting this. I believe this is an issue with the way the CloudSDK authentication is passed to gsutil. I notified the team who maintains this code.

[jbcpollak]

relatedly, we are recently getting errors where we map the .config/gcloud directory into a Docker container with a different home path, and when we run gsutil it fails with a similar issue to the above. Is there a way to work around this?