gsutil always says "Reauthentication required" when using a security key

[tsuna]

I use a YubiKey and have to reauth once a day with gcloud, however even immediately after reauthenticating with gcloud, gsutil keeps prompting for it on each usage.

$ gsutil --debug cp gs://whatever/foo.json /tmp/foo.json
***************************** WARNING *****************************
*** You are running gsutil with debug output enabled.
*** Be aware that debug output includes authentication credentials.
*** Make sure to remove the value of the Authorization header for
*** each HTTP request printed to the console prior to posting to
*** a public medium such as a forum post or Stack Overflow.
***************************** WARNING *****************************
gsutil version: 5.23
checksum: cea0c6d70aecfa4525c7abab95b75dcf (OK)
boto version: 2.49.0
python version: 3.11.3 (main, Apr  7 2023, 19:25:52) [Clang 14.0.0 (clang-1400.0.29.202)]
OS: Darwin 22.4.0
multiprocessing available: True
using cloud sdk: True
pass cloud sdk credentials to gsutil: True
[...]
DEBUG 0512 13:25:48.249412 multiprocess_file_storage.py] Read credential file
DEBUG 0512 13:25:48.249971 multiprocess_file_storage.py] Read credential file
INFO 0512 13:25:48.251158 base_api.py] Calling method storage.objects.get with StorageObjectsGetRequest: <StorageObjectsGetRequest
 bucket: 'whatever'
 object: 'foo.json'
 projection: ProjectionValueValuesEnum(noAcl, 1)>
INFO 0512 13:25:48.252010 base_api.py] Making http GET to https://storage.googleapis.com/storage/v1/b/whatever/o/foo.json?alt=json&fields=crc32c%2CcustomerEncryption%2CcontentEncoding%2Cgeneration%2Cetag%2CmediaLink%2Cmd5Hash%2Cname%2CcontentType%2Csize&projection=noAcl
INFO 0512 13:25:48.252429 base_api.py] Headers: {'accept': 'application/json',
 'accept-encoding': 'gzip, deflate',
 'content-length': '0',
 'user-agent': 'apitools Python/3.11.3 gsutil/5.23 (darwin) analytics/disabled '
               'interactive/True command/cp google-cloud-sdk/430.0.0'}
INFO 0512 13:25:48.252702 base_api.py] Body: (none)
INFO 0512 13:25:48.252846 transport.py] Attempting refresh to obtain initial access_token
DEBUG 0512 13:25:48.253362 multiprocess_file_storage.py] Read credential file
DEBUG 0512 13:25:48.253572 multiprocess_file_storage.py] Read credential file
INFO 0512 13:25:48.253670 reauth_creds.py] Refreshing access_token
connect: (oauth2.googleapis.com, 443)
send: b'POST /token HTTP/1.1\r\nHost: oauth2.googleapis.com\r\nContent-Length: 234\r\ncontent-type: application/x-www-form-urlencoded\r\nuser-agent: Python-httplib2/0.20.4 (gzip)\r\naccept-encoding: gzip, deflate\r\n\r\n'
send: b'grant_type=refresh_token&client_id=XXX.apps.googleusercontent.com&client_secret=XXX'
reply: 'HTTP/1.1 400 Bad Request\r\n'
header: Date: Fri, 12 May 2023 11:25:48 GMT
header: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
header: Pragma: no-cache
header: Expires: Mon, 01 Jan 1990 00:00:00 GMT
header: Content-Type: application/json; charset=utf-8
header: Vary: Origin
header: Vary: X-Origin
header: Vary: Referer
header: Content-Encoding: gzip
header: Server: scaffolding on HTTPServer2
header: X-XSS-Protection: 0
header: X-Frame-Options: SAMEORIGIN
header: X-Content-Type-Options: nosniff
header: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
header: Transfer-Encoding: chunked
Reauthentication required.
[...]
Please insert your security key and press enter...^CDEBUG: Exception stack trace:
    Traceback (most recent call last):
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/pyu2f/pyu2f/convenience/localauthenticator.py", line 36, in Authenticate
        device = u2f.GetLocalU2FInterface(origin=self.origin)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/pyu2f/pyu2f/u2f.py", line 45, in GetLocalU2FInterface
        raise errors.NoDeviceFoundError()
    pyu2f.errors.NoDeviceFoundError

DEBUG: Caught CTRL-C (signal 2) - Exception stack trace:
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gsutil", line 21, in <module>
        gsutil.RunMain()
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gsutil.py", line 151, in RunMain
        sys.exit(gslib.__main__.main())
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 436, in main
        return _RunNamedCommandAndHandleExceptions(
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 633, in _RunNamedCommandAndHandleExceptions
        return command_runner.RunNamedCommand(command_name,
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 421, in RunNamedCommand
        return_code = command_inst.RunCommand()
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 1054, in RunCommand
        name_expansion_iterator = CopyObjectsIterator(
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/name_expansion.py", line 677, in __init__
        name_expansion_dest_tuple = next(self.name_expansion_dest_iter)
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 990, in _ConstructNameExpansionIteratorDstTupleIterator
        NameExpansionIterator(
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/name_expansion.py", line 476, in NameExpansionIterator
        if name_expansion_iterator.IsEmpty():
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/plurality_checkable_iterator.py", line 100, in IsEmpty
        return not self._PopulateHead()
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/plurality_checkable_iterator.py", line 70, in _PopulateHead
        e = next(self.base_iterator)
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/name_expansion.py", line 242, in __iter__
        src_url_expands_to_multi = post_step1_iter.HasPlurality()
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/plurality_checkable_iterator.py", line 105, in HasPlurality
        return self._PopulateHead(num_elements=2) > 1
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/plurality_checkable_iterator.py", line 70, in _PopulateHead
        e = next(self.base_iterator)
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/wildcard_iterator.py", line 538, in IterAll
        for blr in self.__iter__(bucket_listing_fields=bucket_listing_fields,
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/wildcard_iterator.py", line 189, in __iter__
        get_object = self.gsutil_api.GetObjectMetadata(
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/cloud_api_delegator.py", line 319, in GetObjectMetadata
        return self._GetApi(provider).GetObjectMetadata(bucket_name,
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 1041, in GetObjectMetadata
        object_metadata = self._GetObjectMetadataHelper(bucket_name,
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 998, in _GetObjectMetadataHelper
        return self.api_client.objects.Get(apitools_request,
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/third_party/storage_apitools/storage_v1_client.py", line 1124, in Get
        return self._RunMethod(
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/base_api.py", line 734, in _RunMethod
        http_response = http_wrapper.MakeRequest(
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 348, in MakeRequest
        return _MakeRequestNoRetry(
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 397, in _MakeRequestNoRetry
        info, content = http.request(
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/vendored/oauth2client/oauth2client/transport.py", line 159, in new_request
        credentials._refresh(orig_request_method)
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/vendored/oauth2client/oauth2client/client.py", line 761, in _refresh
        self._do_refresh_request(http)
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/google-reauth-python/google_reauth/reauth_creds.py", line 112, in _do_refresh_request
        self._update(*reauth.refresh_access_token(
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/google-reauth-python/google_reauth/reauth.py", line 283, in refresh_access_token
        rapt = get_rapt_token(
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/google-reauth-python/google_reauth/reauth.py", line 194, in get_rapt_token
        rapt_token = _obtain_rapt(
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/google-reauth-python/google_reauth/reauth.py", line 144, in _obtain_rapt
        msg = _run_next_challenge(msg, http_request, access_token)
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/google-reauth-python/google_reauth/reauth.py", line 93, in _run_next_challenge
        client_input = c.obtain_challenge_input(challenge)
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/google-reauth-python/google_reauth/challenges.py", line 110, in obtain_challenge_input
        response = api.Authenticate(app_id, challenge_data,
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/pyu2f/pyu2f/convenience/authenticator.py", line 44, in Authenticate
        result = authenticator.Authenticate(app_id,
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/pyu2f/pyu2f/convenience/localauthenticator.py", line 39, in Authenticate
        six.input()
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/sig_handling.py", line 92, in _SignalHandler
        _final_signal_handlers[signal_num](signal_num, cur_stack_frame)
      File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 520, in _HandleControlC
        stack_trace = ''.join(traceback.format_list(traceback.extract_stack()))

This makes gsutil super annoying to work with. Other people on my team are reporting the same problem since we turned on security key with daily refreshes for gcloud and this problem only seems to impact gsutil.

[tsuna]

Ping? Nobody at Google is using gsutil with a security key and running into this annoying issue?

[tsuna]

@NickGoog you guys don't run into this internally with security keys?

[NickGoog]

@tsuna , I've never actually seen someone use a security key with gsutil specifically. I'll bring this to the attention of our OnDuty @rrauber

rrauber fixed a reauth bug recently, but, if your GCP project is configured with an aggressive reauth policy (e.g. daily reauth), we can't help from the client-side.

[tsuna]

It's not like we really have a choice, as soon as we turned on YubiKeys for gcloud, this became a problem.

[NickGoog]

Using the new storage CLI, gcloud storage, may be a workaround

[tsuna]

Nice, this does help, thanks!

[ryukinix]

rrauber fixed a reauth bug recently, but, if your GCP project is configured with an aggressive reauth policy (e.g. daily reauth), we can't help from the client-side.

The company I work for configure not daily, but hourly reauth. It's just a nightmare in any place beyond gsutil, the gsutil (funny enough) is the only place where reauth through yubikey works. The rest I need to gcloud auth application-default login opening browser authentication stuff, confirmation and etc.

I wish there is some command to refresh the token using the same method gsutil does, which request the yubikey being pressed (much more ok to me than browser authentication stuff).

[covalspace]

Using the new storage CLI, gcloud storage, may be a workaround

This worked perfectly for me thank you!