googleapis / google-cloud-go

Google Cloud Client Libraries for Go.
https://cloud.google.com/go/docs/reference
Apache License 2.0
3.72k stars 1.28k forks source link

storage: SetAttrSelection not returning checksums when using customer-managed encryption keys #10922

Closed arriven closed 3 days ago

arriven commented 4 days ago

Client

Storage

Environment

go version go1.22.7 linux/amd64

Code and Dependencies

package main

import (
    "context"
    "encoding/hex"
    "fmt"

    "cloud.google.com/go/storage"
    "google.golang.org/api/iterator"
)

func main() {
    ctx := context.Background()
    client, err := storage.NewClient(ctx)
    bucketName := "<REPLACEME>"

    q := &storage.Query{
        Prefix:    "",
        Delimiter: "/",
    }
    err = q.SetAttrSelection([]string{"Name", "Created", "MD5", "Size"})
    if err != nil {
        panic(err)
    }

    it := client.Bucket(bucketName).Objects(ctx, q)
    for {
        attrs, err := it.Next()
        if err == iterator.Done {
            break
        }
        if err != nil {
            panic(err)
        }

        fmt.Println("query", hex.EncodeToString(attrs.MD5))
        attrs, err = client.Bucket(bucketName).Object(attrs.Name).Attrs(ctx)
        if err != nil {
            panic(err)
        }
        fmt.Println("direct", hex.EncodeToString(attrs.MD5))
    }
}
go.mod ```text module storage-bug-mvp go 1.22 toolchain go1.22.7 require ( cloud.google.com/go/storage v1.43.0 google.golang.org/api v0.199.0 ) require ( cloud.google.com/go v0.115.1 // indirect cloud.google.com/go/auth v0.9.5 // indirect cloud.google.com/go/auth/oauth2adapt v0.2.4 // indirect cloud.google.com/go/compute/metadata v0.5.2 // indirect cloud.google.com/go/iam v1.2.0 // indirect github.com/felixge/httpsnoop v1.0.4 // indirect github.com/go-logr/logr v1.4.2 // indirect github.com/go-logr/stdr v1.2.2 // indirect github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect github.com/google/s2a-go v0.1.8 // indirect github.com/google/uuid v1.6.0 // indirect github.com/googleapis/enterprise-certificate-proxy v0.3.4 // indirect github.com/googleapis/gax-go/v2 v2.13.0 // indirect go.opencensus.io v0.24.0 // indirect go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.54.0 // indirect go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.54.0 // indirect go.opentelemetry.io/otel v1.29.0 // indirect go.opentelemetry.io/otel/metric v1.29.0 // indirect go.opentelemetry.io/otel/trace v1.29.0 // indirect golang.org/x/crypto v0.27.0 // indirect golang.org/x/net v0.29.0 // indirect golang.org/x/oauth2 v0.23.0 // indirect golang.org/x/sync v0.8.0 // indirect golang.org/x/sys v0.25.0 // indirect golang.org/x/text v0.18.0 // indirect golang.org/x/time v0.6.0 // indirect google.golang.org/genproto v0.0.0-20240903143218-8af14fe29dc1 // indirect google.golang.org/genproto/googleapis/api v0.0.0-20240827150818-7e3bb234dfed // indirect google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 // indirect google.golang.org/grpc v1.67.0 // indirect google.golang.org/protobuf v1.34.2 // indirect ) ```

Expected behavior

Both query and direct attributes retrieval returning the same checksum

Actual behavior

Correct behavior for buckets with google-managed encryption, but using buckets with customer-managed encryption keys results in the following output

query 
direct 20a71a61525818fe731e9984bb4d6b3a

Additional information

I tested it with files uploaded via gcp console and files uploaded via gsutil cp/gsutil rsync. The gcp console shows -- instead of checksums for affected buckets when navigating to file versions

BrennaEpp commented 3 days ago

Hi @arriven, thanks for reaching out.

This is intended behaviour. From our documentation on customer managed keys:

The CRC32C checksum and MD5 hash of objects encrypted with CMEKs are not returned when listing objects with the JSON API.

You can obtain the key by getting the Attrs of the object, as you are doing already in that code snippet.

Feel free to reopen this issue if you have any more questions.