Issue with PublicHost and Golang client

[ivanovaleksey]

Hey!

Thank you for the great GCS emulator! I am using this server as a local storage while working with DuckDB.

It has a feature working with files in S3 storages directly https://duckdb.org/docs/guides/import/s3_import To use it I need to configure endpoint SET s3_endpoint='some-host'; As far as I understand that should be the same host that is specified in public-host.

I just picked up value for public host from source code comment, i.e. storage.gcs.127.0.0.1.nip.io:4443 It works fine when I run fake-gcs-server as a binary but fails when I run it within Golang unit-test. I took this example https://github.com/fsouza/fake-gcs-server/blob/main/fakestorage/example_test.go#LL41C13-L41C13 and added PublicHost: "storage.gcs.127.0.0.1.nip.io:8081" and then got object not found error.

After some debugging I realized that the issue is probably here https://github.com/fsouza/fake-gcs-server/blob/main/fakestorage/server.go#L418 r.Host was storage.googleapis.com while s.publicHost was storage.gcs.127.0.0.1.nip.io:8081.

I got it working by initialing GCS client manually

client, err := storage.NewClient(context.Background(), option.WithEndpoint("http://storage.gcs.127.0.0.1.nip.io:8081/storage/v1/"))

and specifying Scheme = "http"

I wonder if it's intended behaviour? My impression was that (*Server).Client() method should have handled this automatically as it was stated in the comment

Client returns a GCS client configured to talk to the server.

This issue might be related to #982

[fsouza]

I wonder if it's intended behaviour? My impression was that (*Server).Client() method should have handled this automatically as it was stated in the comment

Hmm, that's interesting. (*Server).Client() should definitely handle this correctly. Do you have an example of a unit test that fails with that error? We should add those tests to fake-gcs-server itself and fix the created client to make sure it either sets all headers or accepts any requests.

[ivanovaleksey]

@fsouza thank you for the reply! Please see my commit here

I am not sure what is happening but in ExampleServer_with_public_host function panic doesn't really work. If you debug it you will see that err != nil but panic just handle it silently. I changed it to log.Fatal and also added another "proper" test. There panic actually works as expected.

Weird thing is that if I copy ExampleServer_with_public_host to my actual project it fails. Maybe something is wrong with my environment? Can't even imagine what it could be.

Could you please run it yourself to see if it works/fails?

P.S.

I guess it has something do to with it being example. This doesn't panic

func ExamplePanic() {
    panic("error")
    // Output: whatever
}

Still don't understand why it fails in some files and not in the others. If I put the code above to internal/backend/backend_test.go it fails but if I put it any *_test.go file under fakestorage directory it just runs silently

[jboyens]

I had a very very similar issue with Google's Go client. I don't know that it's the same issue, but this stumped me for longer than I'll care to admit.

By default, the Go client code will use the XML range reader rather than the JSON range reader (see https://github.com/googleapis/google-cloud-go/blob/cf3452eba7d02b28343c9e6c2425288f80e42d3c/storage/http_client.go#L780-L790).

If I, on the official client, say storage.WithJSONReads() as an option alongside option.WithEndpoint() everything is fine and I receive no 404.

[bliuchak]

@jboyens you've saved me a lot of time! storage.WithJSONReads() works like a charm.

[pocockn]

Thanks @jboyens !