secretmanager: New version upgrade of cloud.google.com/go/secretmanager to v1.14.0 is breaking while fetching the secret with below error

raghvendra-dixit commented 2 months ago

Client

secret manager client issue with - cloud.google.com/go/secretmanager v1.14.0, getting below error: Error: level=error msg="GCP Secret Manager: failed to create secretManager's NewClient: open /dev/null/.config/gcloud/certificate_config.json: not a directory"

Environment

cbl mariner distroless image $ go version - go 1.22.6

Code and Dependencies

func (smc *GCPSMCClient) InitializeClient() (err error) {
    smc.traceLog.Debugf("GCP Secret Manager: In InitializeClient()")

    // Create the client.
    ctx := context.Background()
    var smClient *secretmanager.Client
    smClient, err = secretmanager.NewClient(ctx)
    if err != nil {
        smc.traceLog.Errorf("GCP Secret Manager: failed to create secretManager's NewClient: %v", err)
        return err
    }
    smc.client = smClient

    return err
}

go.mod

```text module modname go 1.22.6 require ( "context" "fmt" "hash/crc32" secretmanager "cloud.google.com/go/secretmanager/apiv1" "cloud.google.com/go/secretmanager/apiv1/secretmanagerpb" ) ```

Expected behavior

issue with - cloud.google.com/go/secretmanager v1.14.0, getting below error: Error: level=error msg="GCP Secret Manager: failed to create secretManager's NewClient: open /dev/null/.config/gcloud/certificate_config.json: not a directory" client works just fine with - cloud.google.com/go/secretmanager v1.12.0 As of now had to downgrade to v1.12.0 to make it work. with below changes in direct and indirect dependencies in go.mod Please note that below go.mod is the revert from v1.14.0 to v1.12.0 in order to make secret fetching work. - denotes removal and + denotes addition, pasted the git diff of the reverted change.

go 1.22.6

require (

cloud.google.com/go/secretmanager v1.14.0
cloud.google.com/go/secretmanager v1.12.0

-cloud.google.com/go/iam v1.2.0 // indirect +cloud.google.com/go/iam v1.1.8 // indirect

-github.com/googleapis/gax-go/v2 v2.13.0 // indirect +github.com/googleapis/gax-go/v2 v2.12.4 // indirect

-google.golang.org/api v0.194.0 // indirect -google.golang.org/genproto v0.0.0-20240826202546-f6391c0de4c7 // indirect +google.golang.org/api v0.182.0 // indirect +google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda // indirect )

Actual behavior

with cloud.google.com/go/secretmanager v1.12.0 to cloud.google.com/go/secretmanager v1.14.0, it should not break with error - Error: level=error msg="GCP Secret Manager: failed to create secretManager's NewClient: open /dev/null/.config/gcloud/certificate_config.json: not a directory"

Screenshots

Let me know if any other details is required, will be happy to provide.

Additional context

No additional code in the client was touched, the only diff is upgrade of cloud.google.com/go/secretmanager v1.12.0 to cloud.google.com/go/secretmanager v1.14.0, which result in this break.

codyoss commented 2 months ago

Can you run go get cloud.google.com/go/auth@latest to see if that fixes the issue? I believe this weeks release should fix the issue here. Related: #10696

raghvendra-dixit commented 2 months ago

running:go get cloud.google.com/go/auth@latest

go get cloud.google.com/go/auth@latest go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.3 go: upgraded github.com/googleapis/enterprise-certificate-proxy v0.3.2 => v0.3.3: error finding sum for github.com/googleapis/enterprise-certificate-proxy@v0.3.3: github.com/googleapis/enterprise-certificate-proxy@v0.3.3: verifying module: checksum mismatch downloaded: h1:G6q7VHBoU74wQHXFsZSLMPl0rFw0ZDrlZ3rt6/aTBII= sum.golang.org: h1:QRje2j5GZimBzlbhGA2V2QlGNgL8G6e+wGo/+/2bWI0=

SECURITY ERROR This download does NOT match the one reported by the checksum server. The bits may have been replaced on the origin server, or an attacker may have intercepted the download attempt.

For more information, see 'go help module-auth'.

go help module-auth When the go command downloads a module zip file or go.mod file into the module cache, it computes a cryptographic hash and compares it with a known value to verify the file hasn't changed since it was first downloaded. Known hashes are stored in a file in the module root directory named go.sum. Hashes may also be downloaded from the checksum database depending on the values of GOSUMDB, GOPRIVATE, and GONOSUMDB.

For details, see https://golang.org/ref/mod#authenticating.

Do I need to clear cache in order to make it work?

codyoss commented 2 months ago

I would try to run that command again and/or clear your cache. I am not able to reproduce that error though. Maybe you had a corrupted during download. I see the following in my .sum

github.com/googleapis/enterprise-certificate-proxy v0.3.3 h1:QRje2j5GZimBzlbhGA2V2QlGNgL8G6e+wGo/+/2bWI0=
github.com/googleapis/enterprise-certificate-proxy v0.3.3/go.mod h1:YKe7cfqYXjKGpGvmSg28/fFvhNzinZQm8DGnaburhGA=

raghvendra-dixit commented 2 months ago

not sure, i did a fresh install, cleaned cache, restarted the system go clean --cache followed by go clean --modcache After restart i ran go mod tidy to get the required dependencies and then ran the above command to pull latest go auth still getting the same error: go get cloud.google.com/go/auth@latest go: downloading cloud.google.com/go/auth v0.9.4 go: downloading golang.org/x/sys v0.25.0 go: downloading google.golang.org/grpc v1.66.0 go: downloading golang.org/x/net v0.29.0 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.3 go: downloading golang.org/x/crypto v0.27.0 go: downloading golang.org/x/text v0.18.0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 go: upgraded github.com/googleapis/enterprise-certificate-proxy v0.3.2 => v0.3.3: error finding sum for github.com/googleapis/enterprise-certificate-proxy@v0.3.3: github.com/googleapis/enterprise-certificate-proxy@v0.3.3: verifying module: checksum mismatch downloaded: h1:G6q7VHBoU74wQHXFsZSLMPl0rFw0ZDrlZ3rt6/aTBII= sum.golang.org: h1:QRje2j5GZimBzlbhGA2V2QlGNgL8G6e+wGo/+/2bWI0=

SECURITY ERROR This download does NOT match the one reported by the checksum server. The bits may have been replaced on the origin server, or an attacker may have intercepted the download attempt.

For more information, see 'go help module-auth'. And i see :

github.com/googleapis/enterprise-certificate-proxy v0.3.2 h1:Vie5ybvEvT75RniqhfFxPRy3Bf7vr3h0cechB90XaQs=
github.com/googleapis/enterprise-certificate-proxy v0.3.2/go.mod h1:VLSiSSBs/ksPL8kq3OBOQ6WRI2QnaFynd1DCjZ62+V0=

Let me try in another machine/colleague's machine and update in next thread if I am able to go through pas this upgrade.

raghvendra-dixit commented 2 months ago

Thanks @codyoss for looking into this..

Update: I tested in my colleague's machine, and go get worked fine and i am able to fetch the latest version of cloud.google.com/go/auth@latest gosum also shows v0.3.3

github.com/googleapis/enterprise-certificate-proxy v0.3.3 h1:QRje2j5GZimBzlbhGA2V2QlGNgL8G6e+wGo/+/2bWI0=
github.com/googleapis/enterprise-certificate-proxy v0.3.3/go.mod h1:YKe7cfqYXjKGpGvmSg28/fFvhNzinZQm8DGnaburhGA=

After upgrading, I verified, still throws the same error for not directory

time="2024-09-11T20:49:13.419841209Z" level=error msg="token rotation: getSMCLicenseFromCloud failed with
err=InitializeClient failed in getSMCLicenseFromCloud() with error open
 /dev/null/.config/gcloud/certificate_config.json: not a directory"

Could this be related to the recent PR merged : https://github.com/googleapis/google-cloud-go/pull/10697/files cc: @jba

codyoss commented 2 months ago

That is the change I would have suspected would fix issue, yes. That is why I was wondering if pulling cloud.google.com/go/auth @ latest would fix it. Can you share your go.mod from your example

raghvendra-dixit commented 2 months ago

sure, I am shairing only the impacted libs rather than entire go.mod and go.sum go.mod

go 1.22.6
require (
    cloud.google.com/go/secretmanager v1.14.0
    golang.org/x/net v0.29.0
)
require (
    cloud.google.com/go/auth v0.9.4 // indirect
    cloud.google.com/go/iam v1.1.13 // indirect
    github.com/googleapis/enterprise-certificate-proxy v0.3.3 // indirect
    github.com/googleapis/gax-go/v2 v2.13.0 // indirect
    golang.org/x/crypto v0.27.0 // indirect
    google.golang.org/api v0.193.0 // indirect
    google.golang.org/genproto v0.0.0-20240814211410-ddb44dafa142 // indirect
    google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 // indirect
    google.golang.org/grpc v1.66.0 // indirect
)
require (
    golang.org/x/sys v0.25.0 // indirect
    golang.org/x/text v0.18.0 // indirect
)

go.sum

cloud.google.com/go v0.115.1 h1:Jo0SM9cQnSkYfp44+v+NQXHpcHqlnRJk2qxh6yvxxxQ=
cloud.google.com/go v0.115.1/go.mod h1:DuujITeaufu3gL68/lOFIirVNJwQeyf5UXyi+Wbgknc=
cloud.google.com/go/auth v0.9.4 h1:DxF7imbEbiFu9+zdKC6cKBko1e8XeJnipNqIbWZ+kDI=
cloud.google.com/go/auth v0.9.4/go.mod h1:SHia8n6//Ya940F1rLimhJCjjx7KE17t0ctFEci3HkA=
cloud.google.com/go/iam v1.1.13 h1:7zWBXG9ERbMLrzQBRhFliAV+kjcRToDTgQT3CTwYyv4=
cloud.google.com/go/iam v1.1.13/go.mod h1:K8mY0uSXwEXS30KrnVb+j54LB/ntfZu1dr+4zFMNbus=
cloud.google.com/go/secretmanager v1.14.0 h1:P2RRu2NEsQyOjplhUPvWKqzDXUKzwejHLuSUBHI8c4w=
cloud.google.com/go/secretmanager v1.14.0/go.mod h1:q0hSFHzoW7eRgyYFH8trqEFavgrMeiJI4FETNN78vhM=
github.com/googleapis/enterprise-certificate-proxy v0.3.3 h1:QRje2j5GZimBzlbhGA2V2QlGNgL8G6e+wGo/+/2bWI0=
github.com/googleapis/enterprise-certificate-proxy v0.3.3/go.mod h1:YKe7cfqYXjKGpGvmSg28/fFvhNzinZQm8DGnaburhGA=
github.com/googleapis/gax-go/v2 v2.13.0 h1:yitjD5f7jQHhyDsnhKEBU52NdvvdSeGzlAnDPT0hH1s=
github.com/googleapis/gax-go/v2 v2.13.0/go.mod h1:Z/fvTZXF8/uw7Xu5GuslPw+bplx6SS338j1Is2S+B7A=
go.opentelemetry.io/otel/sdk v1.28.0 h1:b9d7hIry8yZsgtbmM0DKyPWMMUMlK9NEKuIG4aBqWyE=
go.opentelemetry.io/otel/sdk v1.28.0/go.mod h1:oYj7ClPUA7Iw3m+r7GeEjz0qckQRJK2B8zjcZEfu7Pg=
golang.org/x/crypto v0.27.0 h1:GXm2NjJrPaiv/h1tb2UH8QfgC/hOf/+z0p6PT8o1w7A=
golang.org/x/crypto v0.27.0/go.mod h1:1Xngt8kV6Dvbssa53Ziq6Eqn0HqbZi5Z6R0ZpwQzt70=
golang.org/x/net v0.29.0 h1:5ORfpBpCs4HzDYoodCDBbwHzdR5UrLBZ3sOnUJmFoHo=
golang.org/x/net v0.29.0/go.mod h1:gLkgy8jTGERgjzMic6DS9+SP0ajcu6Xu3Orq/SpETg0=
golang.org/x/sys v0.25.0 h1:r+8e+loiHxRqhXVl6ML1nO3l1+oFoWbnlu2Ehimmi34=
golang.org/x/sys v0.25.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/text v0.18.0 h1:XvMDiNzPAl0jr17s6W9lcaIhGUfUORdGCNsuLmPG224=
golang.org/x/text v0.18.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY=
google.golang.org/api v0.193.0 h1:eOGDoJFsLU+HpCBaDJex2fWiYujAw9KbXgpOAMePoUs=
google.golang.org/api v0.193.0/go.mod h1:Po3YMV1XZx+mTku3cfJrlIYR03wiGrCOsdpC67hjZvw=
google.golang.org/genproto v0.0.0-20240814211410-ddb44dafa142 h1:oLiyxGgE+rt22duwci1+TG7bg2/L1LQsXwfjPlmuJA0=
google.golang.org/genproto v0.0.0-20240814211410-ddb44dafa142/go.mod h1:G11eXq53iI5Q+kyNOmCvnzBaxEA2Q/Ik5Tj7nqBE8j4=
google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 h1:pPJltXNxVzT4pK9yD8vR9X75DaWYYmLGMsEvBfFQZzQ=
google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1/go.mod h1:UqMtugtsSgubUsoxbuAoiCXvqvErP7Gf0so0mK9tHxU=
google.golang.org/grpc v1.66.0 h1:DibZuoBznOxbDQxRINckZcUvnCEvrW9pcWIE2yF9r1c=
google.golang.org/grpc v1.66.0/go.mod h1:s3/l6xSSCURdVfAnL+TqCNMyTDAGN6+lZeVxnZR128Y=

raghvendra-dixit commented 2 months ago

hi @codyoss @quartzmo Wanted to check if we have noticed this issue, or there are any prospects of fixing this in coming releases. I am thinking this is broken in current release of secret manager APIs.

codyoss commented 1 month ago

Would you mind upgrading to https://github.com/googleapis/google-cloud-go/releases/tag/auth/v0.9.9. I believe this should fix the issue.

googleapis / google-cloud-go