storage: RawPath weird behaviour when retrieving bucket's blob

[GuyAfik]

Question

We reported a bug to https://github.com/googleapis/google-cloud-go/issues/10911 (google-cloud-storage) It seems like the workaround of using storage.WithJSONReads() works for us, however that means we need to use the storage module of google-cloud directly which we want to avoid.

Now by default when go-cloud interacts with google-buckets, it uses the XML api and not the JSON api, however the XML api doesn't work in our use case.

Is it possible to make sure that go-cloud when interacting with google buckets would use the JSON api?

[vangent]

It looks like that's an Option passed to NewClient.

If you are using gcsblob.OpenBucket, you are passing in the client, so just apply that option when you are creating it.

If you are using the URL opener, then no -- there's currently no way to ask the default URLOpener to create a client with that option. However, you can make your own URLOpener (https://github.com/google/go-cloud/blob/master/blob/gcsblob/gcsblob.go#L217) and provide the client there.

It is probably worth following up with google-cloud-storage about why using JSON makes a difference -- it doesn't seem like it should. They will want a repro case that just uses google-cloud-storage directly, without the Go Cloud wrapper.

[GuyAfik]

@vangent Thanks for the help, we are indeed using the gcsblob.OpenBucket, however it doesn't seem like I can pass in the needed option, storage.WithJSONReads() even if i pass in a custom client directly by implementing my own URL Opener.

// openBucket returns a GCS Bucket that communicates using the given HTTP client.
func openBucket(ctx context.Context, client *gcp.HTTPClient, bucketName string, opts *Options) (*bucket, error) {
    if client == nil {
        return nil, errors.New("gcsblob.OpenBucket: client is required")
    }
    if bucketName == "" {
        return nil, errors.New("gcsblob.OpenBucket: bucketName is required")
    }

    clientOpts := []option.ClientOption{option.WithHTTPClient(useragent.HTTPClient(&client.Client, "blob"))}
    if host := os.Getenv("STORAGE_EMULATOR_HOST"); host != "" {
        clientOpts = []option.ClientOption{
            option.WithoutAuthentication(),
            option.WithEndpoint("http://" + host + "/storage/v1/"),
            option.WithHTTPClient(http.DefaultClient),
        }
    }

    // We wrap the provided http.Client to add a Go CDK User-Agent.
    c, err := storage.NewClient(ctx, clientOpts...)
    if err != nil {
        return nil, err
    }
    if opts == nil {
        opts = &Options{}
    }
    return &bucket{name: bucketName, client: c, opts: opts}, nil
}

Looks like there are some "hardcoded" options. As being said, our current workaround is to work with google-cloud-go directly, but we would prefer to use go-cloud.

[vangent]

Oh I see, it's an option to the storage client, not to the HTTPClient.

OK, I can expose that in Options.

[vangent]

My advice to repro this for google-cloud-storage and get them to fix it stands; using this option seems like a workaround, not a fix.

[GuyAfik]

Thank you for handling it so quickly! when will a new official version be available?