Open jywarren opened 3 years ago
@jywarren so this looks like something on server-side, at least to me, from the docs:
Caution: If you enable uniform bucket-level access, you revoke access from users
who gain their access solely through object ACLs and prevent users from administrating
permissions using bucket ACLs. Be sure that you read considerations when migrating
an existing bucket prior to enabling uniform bucket-level access.
Can you post the output of:
gsutil uniformbucketlevelaccess get gs://BUCKET_NAME
Hi!
Yes we've been using gcsfuse
with this bucket and have found uniform bucket-level access desirable.
The output of the command is:
$ gsutil uniformbucketlevelaccess get gs://production-public-files
Uniform bucket-level access setting for gs://production-public-files:
Enabled: True
LockedTime: 2021-05-20 16:23:28.446000+00:00
We can't change the Uniform Bucket Level access unless we clone the bucket, but that's our plan B. Thanks for your assistance.
@icarito then in this case the problem is indeed because of the Uniform Bucket Level access. You should still be able to manage ACL's for files in the bucket, but paperclip needs to use the new ACL model, not a legacy one.
In theory the support for fog_public
that paperclip uses is there: https://github.com/fog/fog-google/pull/318/files
Can you try deleting the XML credentials from the config (thus forcing the JSON driver) and let me know how it goes?
google_storage_access_key_id: ENV["GOOGLE_STORAGE_KEY"] || '' ,
google_storage_secret_access_key: ENV["GOOGLE_STORAGE_SECRET"] || ''
We tried deleting those creds but still got this, unfortunately:
invalid: Cannot insert legacy ACL for an object when uniform bucket-level access is enabled. Read more at https://cloud.google.com/storage/docs/uniform-bucket-level-access
Is there any way to turn off fog's setting of ACL, and just push the file up respecting whatever the bucket setting is?
i.e. predefined_acl
could be unset or left as default?
Sorry, busy week.
@icco can you take a look into it? IIRC you know the storage part of things a bit better than me.
Hmm, we currently don't have that functionality... but lemme see how hard it would be.
This issue has been marked inactive and will be closed if no further activity occurs.
@icco any luck?
@Temikus Hello, I have same problems when using CarrierWave. Do you have some good ideas to resolve this.
@guppy0356 I had the same issue.
My solution is ...
config.fog_public
is unset or true in CarrierWave.configure do |config| ...
, change public access to "REMOVE PUBLIC ACCESS PREVENTION".config.fog_public = false
in CarrierWave.configure do |config| ...
, change public access to "PREVENT PUBLIC ACCESS" (for your security).Thanks.
@Temikus
In SAP we are using fog-google CF Cloud Controller Blobstore to access Google storage buckets.
The missing uniformBucketLevelAccess support in fog-google has been known for some time but never has been a blocker for us. With the Sovereign Cloud Restrictions the uniformBucketLevelAccess becomes hot topic for everyone who would like to use the offering.
Therefore, could you please update the status of uniformBucketLevelAccess support.
Appreciate your support.
@Temikus @icco
As @div-co has brought to our attention, with the Restrictions and limitations in EU Regions Sovereignty Controls, fog-google's missing uniformBucketLevelAccess support is a blocker for those who would like to use Sovereign cloud offering.
Hence really appreciate your support, if you can provide an update here. Thanks in advance.
@div-co @harinigunabalan is there a chance you folks can provide a PR for this or collaborate? Happy to review and help.
I'm asking since both myself and @icco are pretty short on time right now and don't have an active CarrierWave/PaperClip app we maintain, which I assume you do.
@Temikus also https://github.com/kreeti/kt-paperclip/issues/121
Thanks @cwjenkins - merged and released 🙌
Do we need to wait for paperclip fork to change as well or is this enough to work around things? 🤔
Thanks @cwjenkins, @Temikus! We will be testing the release in CF too and provide feedback.
@Temikus this is enough to work around things.
If one uses paperclip, setting fog_options: { uniform: true }
will set the new var to NOT set predefined_acls allowing the request to succeed against a bucket with uniform access configured.
Currently one can set fog_public: ->(x) { nil }
to achieve the same, however I don't find this solution obvious hence the issue being raised to them to discuss it.
Hi @cwjenkins , @Temikus -
@Temikus this is enough to work around things.
If one uses paperclip, setting
fog_options: { uniform: true }
will set the new var to NOT set predefined_acls allowing the request to succeed against a bucket with uniform access configured.
⚠️ Not setting predefined_acls
(<your fog option>: { uniform: true }
of https://github.com/fog/fog-google/pull/600 workaround) seems to have the effect of also allowing the request to succeed against ACL buckets -- kinda confusing for a uniform
option:
what security concerns, if any?
Example meta of an ACL bucket that works with { uniform: true }
fog-google:
$ gsutil ubla get gs://<bucket>
Uniform bucket-level access setting for gs://<bucket>:
Enabled: False
$ gcloud storage ls -L -b gs://<bucket>
Storage Class: STANDARD
Location Type: region
Location Constraint: EUROPE-WEST1
Versioning Enabled: None
Logging Configuration: Present
Website Configuration: None
CORS Configuration: []
Lifecycle Configuration: None
Requester Pays Enabled: None
Labels: { <redacted> }
Default KMS Key: None
Time Created: 2023-11-26T23:38:19Z
Time Updated: 2023-11-29T11:14:46Z
Metageneration: 33
Bucket Policy Only Enabled: False
Public Access Prevention: inherited
ACL: [ <redacted> ]
Default ACL: [ <redacted> ]
Detection activity: migration of CloudFoundry blobstore from ACL to IAM (UBLA). I must note that support-both-accesses effect nicely serves the zero-downtime migration of CF blobstore ACL <-> UBLA, hence the concern on the stability(⚠️) of the feature.
Hey @nikolaydrm, did you read https://github.com/kreeti/kt-paperclip/issues/121 referenced in the above comment?
Not sure I'm following the 'detection activity' comment. Could you elaborate?
Hi @cwjenkins -
Hey @nikolaydrm, did you read kreeti/kt-paperclip#121 referenced in the above comment?
Yes, however I don't use kreeti/kt-paperclip (no experience configuring, deploying and using it, sorry!) but fog-google for https://github.tools.sap/cloudfoundry/cloud_controller_ng#blobstore. PR https://github.com/fog/fog-google/pull/600 nicely supports the case of CF consuming Uniform buckets when uniform support is enabled at fog-google: https://github.com/fog/fog-google/blob/20816216aad4993a1734f174859d1ddd8f0d056e/lib/fog/storage/google_json/models/file.rb#L120
I got surprised that enabling the access to uniform buckets (uniform: true
in fog-google config) would accept ACL buckets as well. Hence the questions: "is it the expected behavior?" and "are there security concerns?". Can't tell if these concern kreeti/kt-paperclip, my understanding is the PR was driven by the piperclip usecase ...
Not sure I'm following the 'detection activity' comment. Could you elaborate?
The motivation to ask for clarification is the usecase of migrating existing CF buckets from ACL to uniform access. By setting uniform: true
at fog-google side a zero-downtime migration could be achieved. Hence the concerns:
Thanks @nikolaydrm for the clarification.
When you say 'would accept ACL buckets as well', are you meaning you'd expect the code to look something like...
if @uniform
options[:predefined_acl] = nil
else
options[:predefined_acl] ||= @predefined_acl
end
If so, I think that better aligns with the server side given they state they disable ACL when uniform is enabled. https://cloud.google.com/storage/docs/uniform-bucket-level-access
If you are concerned from a security standpoint that the client, fog-google, can pass ACLs to a GCS bucket with uniform access control and ignore the uniform control then that isn't the case. This ticket was created due to the fact that some gems, (e.g. kt-paperclip), configured fog-google in a way that would always pass ACL causing the exception mentioned in the title.
@cwjenkins - thanks!
If there are no concerns that fog-google configured for uniform access plays fine with ACL buckets as well (seems to be the case, per your input) then it's "case closed" for me. Otherwise I'd expect the psudo flow:
if uniform option
if bucket not uniform
raise Exception
end
else
set ACL
end
We're using this via Paperclip and seeing this error on upload:
Google::Apis::ClientError
invalid: Cannot insert legacy ACL for an object when uniform bucket-level access is enabled. Read more at https://cloud.google.com/storage/docs/uniform-bucket-level-access
https://sentry.io/share/issue/393f9e2786b543be9b2061a933268129/
Our config is:
Has anyone seen this error? I can't find any mention of uniform bucket level access in this repository.
https://cloud.google.com/storage/docs/uniform-bucket-level-access
Thank you very much!! cc @icarito