[Spanner] Randomly getting ServiceException: Request is missing required authentication credential.

Environment details

OS: Alpine Linux 3.16.15
PHP version: PHP 8.1.18
Package name and version:
- grpc (c extension) v1.54.0
- google/gax v1.25.0
- google/auth v1.32.1
- google/cloud-spanner v1.67.0

We are seeing the following error occur randomly throughout the day (~20 per day) in production.

Google\Cloud\Core\Exception\ServiceException: {
    "message": "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https:\/\/developers.google.com\/identity\/sign-in\/web\/devconsole-project.",
    "code": 16,
    "status": "UNAUTHENTICATED",
    "details": []
} in /project/vendor/google/cloud-core/src/GrpcRequestWrapper.php:263

I tried to reproduce this error by not sending the credentials by returning an empty array here. Unfortunately, I got a slightly different error which contained more context in the details section (see below).

Google\Cloud\Core\Exception\ServiceException: {
    "reason": "CREDENTIALS_MISSING",
    "domain": "googleapis.com",
    "errorInfoMetadata": {
        "method": "google.spanner.v1.Spanner.ExecuteStreamingSql",
        "service": "spanner.googleapis.com"
    },
    "message": "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https:\/\/developers.google.com\/identity\/sign-in\/web\/devconsole-project.",
    "code": 16,
    "status": "UNAUTHENTICATED",
    "details": [
        {
            "@type": "google.rpc.errorinfo-bin",
            "reason": "CREDENTIALS_MISSING",
            "domain": "googleapis.com",
            "metadata": {
                "method": "google.spanner.v1.Spanner.ExecuteStreamingSql",
                "service": "spanner.googleapis.com"
            }
        },
        {
            "@type": "grpc-status-details-bin",
            "data": "<Unknown Binary Data>"
        }
    ]
}

I then tried returning an expired token but that also resulted in a different message (see below).

Google\Cloud\Core\Exception\ServiceException: {
    "message": "Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https:\/\/developers.google.com\/identity\/sign-in\/web\/devconsole-project.",
    "code": 16,
    "status": "UNAUTHENTICATED",
    "details": []
}

So I have not been able to reproduce the error locally.

I also want to note that I added a customized FetchAuthTokenCache and deployed it to production to see if the token was actually being returned and did confirm that the access_token did exist and was NOT expired.

What I fail to understand is that when we deployed the customized FetchAuthTokenCache, we saw fewer errors throughout the day (~1 per day) even though no data is being altered. The error rate went back up again once we've reverted the code a week later.

I'm wondering if it's a similar case to https://github.com/grpc/grpc/issues/15441?

Below is the customized FetchAuthTokenCache

<?php

declare(strict_types=1);

namespace App\Support\SpannerInvestigation;

use Colopl\App\Log\AppMetadataProvider;
use Google\Auth\CredentialsLoader;
use Google\Auth\FetchAuthTokenInterface;
use Google\Cloud\Spanner\SpannerClient;
use Psr\Cache\CacheItemPoolInterface;
use ReflectionClass;
use ReflectionMethod;

class FetchAuthTokenCache extends \Google\Auth\FetchAuthTokenCache
{
    private ReflectionMethod $fetchAuthTokenFromCacheMethod;
    private ReflectionMethod $saveAuthTokenInCacheMethod;

    /**
     * @param array<string, mixed> $cacheConfig Configuration for the cache
     */
    public function __construct(
        // AppMetadataProvider is just an object that stores values that will be 
        // logged to cloud logging as context when an exception is thrown
        private AppMetadataProvider $appMetadataProvider,
        array $cacheConfig,
        CacheItemPoolInterface $cache
    ) {
        $scopes = [SpannerClient::FULL_CONTROL_SCOPE, SpannerClient::ADMIN_SCOPE];
        $creds = CredentialsLoader::fromEnv();
        $fetcher = CredentialsLoader::makeCredentials($scopes, $creds);

        parent::__construct($fetcher, $cacheConfig, $cache);

        $parentReflection = (new ReflectionClass($this))
            ->getParentClass();

        $methodReflection = $parentReflection->getMethod('fetchAuthTokenFromCache');
        $methodReflection->setAccessible(true);
        $this->fetchAuthTokenFromCacheMethod = $methodReflection;

        $methodReflection = $parentReflection->getMethod('saveAuthTokenInCache');
        $methodReflection->setAccessible(true);
        $this->saveAuthTokenInCacheMethod = $methodReflection;
    }

    /**
     * Updates metadata with the authorization token.
     *
     * @param array<mixed> $metadata metadata hashmap
     * @param string $authUri optional auth uri
     * @param callable $httpHandler callback which delivers psr7 request
     * @return array<mixed> updated metadata hashmap
     * @throws \RuntimeException If the fetcher does not implement
     *     `Google\Auth\UpdateMetadataInterface`.
     */
    public function updateMetadata(
        $metadata,
        $authUri = null,
        callable $httpHandler = null
    ) {
        $cached = $this->fetchAuthTokenFromCacheMethod->invoke($this, $authUri);

        if ($cached) {
            // Set the access token in the `Authorization` metadata header so
            // the downstream call to updateMetadata know they don't need to
            // fetch another token.
            if (isset($cached['access_token'])) {
                $metadata[self::AUTH_METADATA_KEY] = [
                    'Bearer ' . $cached['access_token']
                ];
            }
        }

        $newMetadata = $this->getFetcher()->updateMetadata(
            $metadata,
            $authUri,
            $httpHandler
        );

        if (!$cached && $token = $this->getFetcher()->getLastReceivedToken()) {
            $this->saveAuthTokenInCacheMethod->invoke($this, $token, $authUri);
        }

        $this->appMetadataProvider['spanner_auth'] = [
            'cached' => $cached,
            'token' => $token ?? null,
        ];

        return $newMetadata;
    }
}

googleapis / google-cloud-php

[Spanner] Randomly getting ServiceException: Request is missing required authentication credential. #7068

Environment details