RhodiumGroup / rhg_compute_tools

Tools for using compute.rhg.com and compute.impactlab.org
MIT License
1 stars 4 forks source link

Fix need for storage.bucket.get GCS permissions #111

Closed brews closed 1 year ago

brews commented 1 year ago

This fixes issues with users requiring additional GCS permissions to operate on objects within GCS buckets.

The trouble is that google.storage.client's get_bucket() method requires storage.bucket.get permissions to get a successful response from the GCS API. To work around this I switch to use the client.bucket() method to instantiate Buckets without needing these extra permissions on GCS.

Another hitch is that this removes calls to bucket.exists() because it needs storage.bucket.get permissions. This means that these functions won't throw a "NotFound: 404 The specified bucket does not exist" until they try to upload a blob or check that blob.exists().

Requiring storage.bucket.get permission usually means people give accounts god-like admin roles. The alternative is users need to create a custom GCP IAM role which includes storage.bucket.get. All this is a headache because the code otherwise doesn't actually use the bucket data this permission is for.

brews commented 1 year ago

I don't think the tests failing in CI are related to this PR.

brews commented 1 year ago

I am having some internet trouble, switching this to "draft" and checking this manually.

I think bucket.exists() requires storage.bucket.get permissions also.

brews commented 1 year ago

Dropped all bucket.exists() and un-drafting this.

brews commented 1 year ago

Offline, boss said to merge this.