secretmanager: rpc error while iterating secret version using secretIterator.New() post ListSecrets() after new secretManager version upgrade

[patels30]

Client

secretManager Client

Environment

OS (github runner): 15~22.04.1-Ubuntu SMP

$ go version
go version go1.23.1 linux/amd64

Code and Dependencies

type SecretVersionDetails struct {
    SecretVersion *secretmanagerpb.SecretVersion
    Type          string
    Status        string
}

func (smc *SecretManagerClient) FetchSecrets(ctx context.Context) ([]config.SecretVersionDetails, error) {

    reqAllSecrets := &secretmanagerpb.ListSecretsRequest{
        Parent: "projects/abc-sample",
        Filter: secretLabelFilter,
    }

    secretIterator := smc.client.ListSecrets(ctx, reqAllSecrets)

    secretVersionDetails := make([]config.SecretVersionDetails, 0)
    for {
        secret, err := secretIterator.Next()
        if err == iterator.Done {
            break
        }

        if err != nil {
            log.Error().Err(err).Msg("error at secretIterator.Next()")
            return nil, err
        }

        // Check only latest versions
        req := &secretmanagerpb.GetSecretVersionRequest{
            Name: secret.Name + "/versions/latest",
        }

        secretVersion, err := smc.client.GetSecretVersion(ctx, req)

        if err != nil {
            log.Error().Err(err).Msg("error at getting latest secret version")
        } else if secretVersion.State == secretmanagerpb.SecretVersion_ENABLED {
            secretVersionDetails = append(secretVersionDetails, config.SecretVersionDetails{
                SecretVersion: secretVersion,
                Type:          secret.Labels["secrettype"],
            })
        } else {
            log.Info().Str("Version State", secretVersion.State.String()).Str("SecretVersion Name", secretVersion.Name).Msg("secretVersion may be Disabled or Destroyed")
        }
    }

    return secretVersionDetails, nil
}

go.mod

```text module github.com/sample/apps go 1.23 toolchain go1.23.1 require ( cloud.google.com/go/secretmanager v1.14.1 google.golang.org/api v0.196.0 google.golang.org/grpc v1.66.0 ) ```

Expected behavior

It always returned the version details.

Actual behavior

Getting error

{"level":"error","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: EOF\"","time":"2024-10-03T13:27:16+10:00","message":"error at secretIterator.Next()"}

Screenshots

Additional context

We are seeing this error after upgrading the secretManager to 1.41.0 -> 1.41.1. and below

[patels30]

It is working fine with below versions.

cloud.google.com/go/secretmanager v1.14.0
google.golang.org/api v0.193.0
google.golang.org/grpc v1.65.0

// indirect
cloud.google.com/go/auth v0.9.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
google.golang.org/genproto v0.0.0-20240814211410-ddb44dafa142 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240814211410-ddb44dafa142 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240814211410-ddb44dafa142 // indirect

[codyoss]

Can please share how you authenticate your client today? Do you use any custom auth options or just default? I am unable to reproduce this as is. This sounds like a TLS issue if I had to guess. Can you verify what version of cloud.google.com/go/auth you are pulling in when you are getting this error?

[patels30]

It's downloading cloud.google.com/go/auth v0.9.3 for auth. And I’m using the default credentials set by GOOGLE_APPLICATION_CREDENTIALS, which are generated during the OIDC step with google-github-actions/auth for authentication. The same credentials work perfectly fine with the gcloud CLI though.

Edited: I ran a few more tests with different versions of the auth package. I can confirm that it works fine with cloud.google.com/go/auth v0.9.0 and v0.9.1 (using Secret Manager v1.14.0). However, the issue starts occurring from auth v0.9.2 onwards.

[codyoss]

Of the changes in the release notes for 0.9.2 this is the only one that stood out as it may affect something at this layer: https://github.com/googleapis/google-cloud-go/pull/10733

Are you by chance doing anything with the http.DefaultTransport?

[patels30]

No, the code is using Application Default Credentials (ADC) in Go. There are no specific changes related to the transport mode.