googleapis / google-cloud-go

Google Cloud Client Libraries for Go.
https://cloud.google.com/go/docs/reference
Apache License 2.0
3.72k stars 1.27k forks source link

bigtable/bttest: default credentials error despite using insecure connection #10688

Closed jakobatfreiheit closed 4 days ago

jakobatfreiheit commented 1 month ago

Client

Bigtable

Go Environment

go 1.22.1

Code

e.g.

server, err := bttest.NewServer("localhost:0")
if err != nil {
    t.Fatalf("start bigtable emulator server: %v", err)
}
t.Cleanup(server.Close)

conn, err := grpc.NewClient(server.Addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
    t.Fatalf("connect to bigtable emulator instance: %v", err)
}

adminClient, err := bigtable.NewAdminClient(ctx, "emulator-test", "instance", option.WithGRPCConn(conn))
if err != nil {
    t.Fatalf("create admin client: %v", err)
}

Expected behavior

No error, client is successfully created.

Actual behavior

It works until bigtable v1.25.0, but since v1.27.1 I get an error:

create admin client: credentials: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information

Additional context

Same happens with bigtable.NewClient, is not specific to the admin client. I do not know which specific version between 1.25.0 and 1.27.1 broke it.

codyoss commented 4 weeks ago

@jakobatfreiheit could you please share your go.mod. I wrote the following program with the follow go.mod and have no issues:

package main

import (
    "context"
    "log/slog"
    "os"
    "strings"

    "cloud.google.com/go/bigtable"
    "cloud.google.com/go/bigtable/bttest"
    "google.golang.org/api/option"
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials/insecure"
)

func main() {
    if err := run(); err != nil {
        slog.Error("shutting down", "err", err)
        os.Exit(1)
    }
}

func run() error {
    ctx := context.Background()
    server, err := bttest.NewServer("localhost:0")
    if err != nil {
        return err
    }
    defer server.Close()

    conn, err := grpc.NewClient(server.Addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
    if err != nil {
        return err
    }

    adminClient, err := bigtable.NewAdminClient(ctx, "emulator-test", "instance", option.WithGRPCConn(conn))
    if err != nil {
        return err
    }
    defer adminClient.Close()

    resp, err := adminClient.Tables(ctx)
    if err != nil {
        return err
    }
    slog.Info(strings.Join(resp, ","))
    return nil
}
go.mod ```text module example.com/dialconn go 1.23.0 require ( cloud.google.com/go/bigtable v1.27.1 google.golang.org/api v0.189.0 google.golang.org/grpc v1.64.1 ) require ( cel.dev/expr v0.15.0 // indirect cloud.google.com/go v0.115.0 // indirect cloud.google.com/go/auth v0.7.2 // indirect cloud.google.com/go/auth/oauth2adapt v0.2.3 // indirect cloud.google.com/go/compute/metadata v0.5.0 // indirect cloud.google.com/go/iam v1.1.10 // indirect cloud.google.com/go/longrunning v0.5.9 // indirect cloud.google.com/go/monitoring v1.20.1 // indirect github.com/census-instrumentation/opencensus-proto v0.4.1 // indirect github.com/cespare/xxhash/v2 v2.3.0 // indirect github.com/cncf/xds/go v0.0.0-20240423153145-555b57ec207b // indirect github.com/envoyproxy/go-control-plane v0.12.0 // indirect github.com/envoyproxy/protoc-gen-validate v1.0.4 // indirect github.com/felixge/httpsnoop v1.0.4 // indirect github.com/go-logr/logr v1.4.2 // indirect github.com/go-logr/stdr v1.2.2 // indirect github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect github.com/golang/protobuf v1.5.4 // indirect github.com/google/btree v1.1.2 // indirect github.com/google/s2a-go v0.1.7 // indirect github.com/google/uuid v1.6.0 // indirect github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect github.com/googleapis/gax-go/v2 v2.13.0 // indirect go.opencensus.io v0.24.0 // indirect go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 // indirect go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect go.opentelemetry.io/otel v1.24.0 // indirect go.opentelemetry.io/otel/metric v1.24.0 // indirect go.opentelemetry.io/otel/sdk v1.24.0 // indirect go.opentelemetry.io/otel/sdk/metric v1.24.0 // indirect go.opentelemetry.io/otel/trace v1.24.0 // indirect golang.org/x/crypto v0.25.0 // indirect golang.org/x/net v0.27.0 // indirect golang.org/x/oauth2 v0.21.0 // indirect golang.org/x/sync v0.7.0 // indirect golang.org/x/sys v0.22.0 // indirect golang.org/x/text v0.16.0 // indirect golang.org/x/time v0.5.0 // indirect google.golang.org/genproto v0.0.0-20240722135656-d784300faade // indirect google.golang.org/genproto/googleapis/api v0.0.0-20240722135656-d784300faade // indirect google.golang.org/genproto/googleapis/rpc v0.0.0-20240722135656-d784300faade // indirect google.golang.org/protobuf v1.34.2 // indirect rsc.io/binaryregexp v0.2.0 // indirect ) ```
jakobatfreiheit commented 3 weeks ago

Sorry, I just realized I had made a mistake in my original message: The problem does not happen with bigtable.NewAdminClient but with bigtable.NewClient.

So for example running this in a Go docker image (where no credentials are set, I used this one: https://hub.docker.com/r/arm64v8/golang/) reproduces the problem:

package main

import (
        "context"
        "log/slog"
        "os"

        "cloud.google.com/go/bigtable"
        "cloud.google.com/go/bigtable/bttest"
        "google.golang.org/api/option"
        "google.golang.org/grpc"
        "google.golang.org/grpc/credentials/insecure"
)                                                    

func main() {
        if err := run(); err != nil {
                slog.Error("shutting down", "err", err)
                os.Exit(1)                             
        }                 
}        

func run() error {
        ctx := context.Background()
        server, err := bttest.NewServer("localhost:0")
        if err != nil {                               
                return err
        }                 
        defer server.Close()

        conn, err := grpc.NewClient(server.Addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
        if err != nil {                                                                                   
                return err
        }                 

        client, err := bigtable.NewClient(ctx, "emulator-test", "instance", option.WithGRPCConn(conn))
        if err != nil {                                                                                    
                return err
        }                 
        defer client.Close()

        slog.Info("Hi")
        return nil
} 

This is the go.mod:

module example

go 1.23.0

require (
    cloud.google.com/go/bigtable v1.29.0
    google.golang.org/api v0.192.0
    google.golang.org/grpc v1.65.0
)

require (
    cel.dev/expr v0.15.0 // indirect
    cloud.google.com/go v0.115.0 // indirect
    cloud.google.com/go/auth v0.8.1 // indirect
    cloud.google.com/go/auth/oauth2adapt v0.2.3 // indirect
    cloud.google.com/go/compute/metadata v0.5.0 // indirect
    cloud.google.com/go/iam v1.1.12 // indirect
    cloud.google.com/go/longrunning v0.5.11 // indirect
    cloud.google.com/go/monitoring v1.20.3 // indirect
    github.com/census-instrumentation/opencensus-proto v0.4.1 // indirect
    github.com/cespare/xxhash/v2 v2.3.0 // indirect
    github.com/cncf/xds/go v0.0.0-20240423153145-555b57ec207b // indirect
    github.com/envoyproxy/go-control-plane v0.12.0 // indirect
    github.com/envoyproxy/protoc-gen-validate v1.0.4 // indirect
    github.com/felixge/httpsnoop v1.0.4 // indirect
    github.com/go-logr/logr v1.4.2 // indirect
    github.com/go-logr/stdr v1.2.2 // indirect
    github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
    github.com/golang/protobuf v1.5.4 // indirect
    github.com/google/btree v1.1.2 // indirect
    github.com/google/s2a-go v0.1.8 // indirect
    github.com/google/uuid v1.6.0 // indirect
    github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
    github.com/googleapis/gax-go/v2 v2.13.0 // indirect
    go.opencensus.io v0.24.0 // indirect
    go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 // indirect
    go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect
    go.opentelemetry.io/otel v1.24.0 // indirect
    go.opentelemetry.io/otel/metric v1.24.0 // indirect
    go.opentelemetry.io/otel/sdk v1.24.0 // indirect
    go.opentelemetry.io/otel/sdk/metric v1.24.0 // indirect
    go.opentelemetry.io/otel/trace v1.24.0 // indirect
    golang.org/x/crypto v0.25.0 // indirect
    golang.org/x/net v0.27.0 // indirect
    golang.org/x/oauth2 v0.22.0 // indirect
    golang.org/x/sync v0.8.0 // indirect
    golang.org/x/sys v0.22.0 // indirect
    golang.org/x/text v0.16.0 // indirect
    golang.org/x/time v0.6.0 // indirect
    google.golang.org/genproto v0.0.0-20240730163845-b1a4ccb954bf // indirect
    google.golang.org/genproto/googleapis/api v0.0.0-20240725223205-93522f1f2a9f // indirect
    google.golang.org/genproto/googleapis/rpc v0.0.0-20240730163845-b1a4ccb954bf // indirect
    google.golang.org/protobuf v1.34.2 // indirect
    rsc.io/binaryregexp v0.2.0 // indirect
)
codyoss commented 3 weeks ago

I added the following dockerfile, updated to use your provided code, and ran it:

FROM golang

COPY . /app/

WORKDIR /app
CMD ["go", "run", "main.go"]

Output:

2024/08/19 14:10:00 INFO Hi

I am still unable to reproduce the issue.

codyoss commented 3 weeks ago

Actually I think I see the issue. I believe this was introduced with #10046. @bhshkh it looks like we need the metrics provider stuff to be more of a no-op in some cases. This flow tries to dial instead of doing nothings and/or using the provided connection.

bhshkh commented 3 weeks ago

I believe https://github.com/googleapis/google-cloud-go/pull/10658 should fix this

jakobatfreiheit commented 1 week ago

Any update on when we will likely see the fix merged? We're still stuck on v1.25, we have not found a quick-fix to work around this issue.

bhshkh commented 4 days ago

https://github.com/googleapis/google-cloud-go/pull/10815 has the fix bigtable 1.32.0

bhshkh commented 4 days ago

Tested with this code:

package main

import (
    "context"
    "testing"

    "cloud.google.com/go/bigtable"
    "cloud.google.com/go/bigtable/bttest"
    "google.golang.org/api/option"
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials/insecure"
)

func TestIssue10688(t *testing.T) {
    server, err := bttest.NewServer("localhost:0")
    if err != nil {
        t.Fatalf("start bigtable emulator server: %v", err)
    }
    t.Cleanup(server.Close)

    conn, err := grpc.NewClient(server.Addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
    if err != nil {
        t.Fatalf("connect to bigtable emulator instance: %v", err)
    }

    ctx := context.Background()
    _, err = bigtable.NewAdminClient(ctx, "emulator-test", "instance", option.WithGRPCConn(conn))
    if err != nil {
        t.Fatalf("create admin client: %v", err)
    }
}
$ go test -race -v -timeout 45m   ./... 
=== RUN   TestIssue10688
--- PASS: TestIssue10688 (0.01s)
PASS
ok      issue-10688     1.572s