search

cloudyr / googleCloudStorageR

Google Cloud Storage API to R
https://code.markedmondson.me/googleCloudStorageR
Other
104 stars 29 forks source link

Add support for GCS emulators #177

Closed manuteleco closed 1 year ago

manuteleco commented 1 year ago

Allow configuring a Google Cloud Storage emulator to be used instead of the real service. This can be achieved by providing a STORAGE_EMULATOR_HOST environment variable with the scheme, host and port of the emulator server (e.g.: http://localhost:8080).

Besides using a different API endpoint, configuring an emulator also has the effect of lifting the requirement of providing valid authentication credentials. Emulators don't require or even implement authentication.

This behavior is similar to the one implemented in official Google Cloud Storage client libraries developed and maintained by Google.

Closes #176

It also has a weak dependency with https://github.com/MarkEdmondson1234/googleAuthR/pull/228. Without that PR in googleAuthR, using an emulator will still work in googleCloudStorageR but it will unfortunately require valid credentials for the real GCS.

Just like in https://github.com/cloudyr/googleCloudStorageR/pull/175#issuecomment-1646794932, I wonder whether I have to build the pkgdown documentation here or not. I also wonder what else should I include in this PR (e.g., an entry in NEWS.md, etc.).

Also, be aware that I'm new to R, so feel free to point out any unidiomatic or plain odd code that needs to be fixed.

Finally, for testing I have used https://github.com/fsouza/fake-gcs-server, but it currently requires a few adjustments for it to work with googleCloudStorageR: https://github.com/fsouza/fake-gcs-server/pull/1258 and https://github.com/fsouza/fake-gcs-server/pull/1252. With some luck, they will be accepted and merged eventually.

I did not test every single GCS operation; it is possible some are still not working with an emulator yet. But I believe this PR already brings value even if it doesn't happen to provide 100% support. Any operation that is still not working could be refined later, once encountered.

MarkEdmondson1234 commented 1 year ago

Please do add a note to the NEWS.md and feel free to add as a contributor to DESCRIPTION - thank you! Looks like a useful addition. The rebuild of the documentation is usually done before CRAN launch

manuteleco commented 1 year ago

Just in case you didn't get a notification from my last change, I have adjusted NEWS.md and DESCRIPTION. Let me know if there is anything else I should do :+1: .

MarkEdmondson1234 commented 1 year ago

@manuteleco great thank you! I was on holiday

MarkEdmondson1234 commented 1 year ago

@manuteleco I am getting some bug reports since this was merged, is it possible this is related?

https://github.com/cloudyr/googleCloudStorageR/issues/183