Object Storage - bucket policies

Izaia64 commented 2 years ago

As a customer, I need to set S3 bucket policies. As an example, I need to set a public access for all my objects in my bucket.

API: https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucket-policies.html

On Object Storage S3, High Performance and Standard Performance

In S3 there 2 ways to defined permissions on a bucket: via ACLs (legacy but not deprecated) and bucket policies using AWS IAM syntax.

/!\ important ACLs and bucket policies are not mutually exclusive i.e you can use ACLs and bucket policies at the same time, however, please note that the principle of least privilege applies and can summerized as "allow only if there is an explicit allow and no explicit deny, else, deny all"

We will provide an S3 compatible REST endpoint to put, delete, dowload a policy configuration file that respects the AWS IAM policy syntax for a given bucket.

The following configuration elements will be supported:

statement definition with support for "effect", "principal", "action" and "resource" definition
ip whitelisting with conditional keys

The following configuration elements will not be supported:

AWS IAM roles will not be supported as it is a feature specific to AWS IAM
"groups" in "principal" defintion (a global overhaul of the identity management system in the OVH ecosystem is discussed)
conditional keys (for now): as it is an advanced functionality of policies, do not hesitate to submit an improvement request and upvote it if you want it to be put in the pipeline (except for IP whitelisting)

genjudev commented 1 year ago

There is a s3 policy import function, im not quite sure how its working. Importing a policy does not change anything.

Izaia64 commented 1 year ago

Already available are user policy (not yet bucket policy) You will read example of S3 User policy and the structure of the json here : https://docs.ovh.com/ie/en/storage/s3/identity-and-access-management/

genjudev commented 1 year ago

Yeah we tried the user policy. We imported an JSON but it didn't and no error is seen. I try again with this example. Thanks.

univ3rse commented 1 year ago

@larsonnn Same here, no effect when adding a user policy for a bucket. Want to restrict, upload by this user to BUCKET_B. First gave read-write to BUCKET_A, nothing. Restricted BUCKET_B to read only, nothing. Can still upload to BUCKET_B without any issues using multer-s3 tool.


  "Statement": [
    {
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:ListMultipartUploadParts",
        "s3:ListBucketMultipartUploads",
        "s3:AbortMultipartUpload"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::BUCKET_A",
        "arn:aws:s3:::BUCKET_A/*"
      ],
      "Sid": "RWContainer"
    },
    {
      "Action": [
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListMultipartUploadParts",
        "s3:ListBucketMultipartUploads"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::BUCKET_B",
        "arn:aws:s3:::BUCKET_B/*"
      ],
      "Sid": "ROContainer"
    }
  ]
}

Docteur-RS commented 1 year ago

Have tried to make this work for 6 hours straight... The policies have no effect. Wtf is OVH doing releasing functionnalities like that... At least add a "beta" or "wip" label in the UI damn it !

Docteur-RS commented 1 year ago

Also I can always list the swift/openstack based buckets. This is a problem because you can't assign a user in "DENY" mode to this kind of buckets...

mcartoixa commented 1 year ago

It is my understanding that public access on AWS S3 is granted with a bucket policy (cf. https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-use-case-2) and that we can't have the same level of access on an entire Object Storage bucket until this functionnaliity is implemented. Any idea of an ETA at this point?

rgdev commented 1 year ago

@Izaia64 What is the ETA on this ?

gustoune commented 1 year ago

+1

pierresouchay commented 1 year ago

Clearly, we don't get how to have read-only access. Here is our policy (set on user):

{
  "Statement": [
    {
      "Sid": "RContainer",
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:HeadObject",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": [
        "arn:aws:s3:::my-container",
        "arn:aws:s3:::my-container/*"
      ]
    }
  ]
}

How to get this working?

mscheer93 commented 1 year ago

some ETA yet?

rgdev commented 1 year ago

From to the objectstore-s3 OVH discord channel : Hi, there is a high probability that bucket policies will be available by mid-quarter 2

antoine-de commented 1 year ago

I confirm that the bucket wide policy does not work but the object one do.

It's a pity this is not clearly stated in the documentation (there is even put-bucket-acl commands)!

lason-ovh commented 1 year ago

what do you mean by "does not work"?

antoine-de commented 1 year ago

If I set a bucket wide policy (as stated in the OVH documentation:

❯ aws s3api put-bucket-acl --bucket my_bucket --acl public-read

The ACL seems to have been correctly set:

❯ aws s3api get-bucket-acl --bucket my_bucket
{
    "Owner": {
        "DisplayName": "sniped",
        "ID": "sniped"
    },
    "Grants": [
        {
            "Grantee": {
                "Type": "Group",
                "URI": "http://acs.amazonaws.com/groups/global/AllUsers"
            },
            "Permission": "READ"
        },
        {
            "Grantee": {
                "DisplayName": "sniped",
                "ID": "sniped,
                "Type": "CanonicalUser"
            },
            "Permission": "FULL_CONTROL"
        }
    ]
}

but I cannot access the files in it:

❯ http -h https://my_bucket.s3.gra.io.cloud.ovh.net/derivates/00/1e/95/6e/0066-45a5-8361-00980e89a1aa/sd.jpg
HTTP/1.1 403 Forbidden

If I set the ACL on each file it's ok

❯ aws s3api put-object-acl --bucket my_bucket --key derivates/00/1e/95/6e/0066-45a5-8361-00980e89a1aa/sd.jpg --acl public-read

~ 
❯ http -h https://mubucket-public.s3.gra.io.cloud.ovh.net/derivates/00/1e/95/6e/0066-45a5-8361-00980e89a1aa/sd.jpg
HTTP/1.1 200 OK

devlifealways commented 1 year ago

Same thing over here, started a huge migration work to use these new s3 API coming from old swift containers to be faced with this issue, I think it's not explained enough on the docs. I guess we'll have to wait for this to be implemented

# names are redacted for confidentiality reasons
$ curl -I https://MY_BUCKET.s3.gra.io.cloud.ovh.net/payloads/MY_PAYLOAD

HTTP/1.1 403 Forbidden
Content-Type: application/xml
...
Date: Thu, 27 Apr 2023 15:15:15 GMT

when I use object-acl, indeed everything works

$ aws --endpoint-url https://s3.gra.io.cloud.ovh.net s3api put-object-acl --bucket MY_BUCKET --acl public-read  --key MY_ARTIFACT --acl public-read

$ curl -I https://MY_BUCKET.s3.gra.io.cloud.ovh.net/payloads/MY_PAYLOAD

HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Length: 4306
x-amz-storage-class: STANDARD
....
Date: Thu, 27 Apr 2023 15:18:10 GMT

lason-ovh commented 1 year ago

I think there was a mistake in the ACL doc as it was recently updated. READ at bucket level allows the grantee to only list the objects, you have to set the permission at object level as you did.

We are sorry for the mistake as it was extremely misleading :(

clementdugal commented 1 year ago

I think there was a mistake in the ACL doc as it was recently updated. READ at bucket level allows the grantee to only list the objects, you have to set the permission at object level as you did.

We are sorry for the mistake as it was extremely misleading :(

Hello, is it possible to apply default ACLs for all new uploaded objects? Thank you!

lason-ovh commented 1 year ago

I think there was a mistake in the ACL doc as it was recently updated. READ at bucket level allows the grantee to only list the objects, you have to set the permission at object level as you did. We are sorry for the mistake as it was extremely misleading :(

Hello, is it possible to apply default ACLs for all new uploaded objects? Thank you!

Hi, you can directly set ACLs on the object when you upload it. Examples:

if you are using the cli: aws s3api put-object --bucket my-bucket --key my-object --body path/to/object --ACL public-read
if you are using boto library in Python: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/put_object.html

Wengiel31 commented 11 months ago

I'd really love to see that implemented

lason-ovh commented 11 months ago

I'd really love to see that implemented

Do not hesitate to like the issue to help us prioritize ;)

kedare commented 9 months ago

Indeed the current documentation is very misleading, I have been looking for this for hours. It may be interesting to add documentation about this not being supported yet.

root-io commented 8 months ago

I have ~800k objects in my OVH S3 bucket, am I supposed to iterate over each one of them with put-object-acl ? It will take ages...

Edit: a quick fix for this could be having a UI on the manager, you type a prefix (can be empty for root), you a select an available ACLs then apply to all objects.

lason-ovh commented 8 months ago

I have ~800k objects in my OVH S3 bucket, am I supposed to iterate over each one of them with put-object-acl ? It will take ages...

Edit: a quick fix for this could be having a UI on the manager, you type a prefix (can be empty for root), you a select an available ACLs then apply to all objects.

Thanks for the suggestion, we will take that into consideration, stay tuned :)

lbcd commented 8 months ago

@lason-ovh any update about that?

It is a simple use case for website:

readonly for public for all bucket files
read/write for one user used by the website to do uploads

I guess we have no choice to iterate on all object to set acl or going to AWS to have a complete S3 api.

pinguin999 commented 7 months ago

What's the status of this? Is it already possible to have files / buckets with public access? If so please link me a step by step guide.

I want to host my Podcast Mp3s via OVHs S3.

Thanks a lot.

lason-ovh commented 5 months ago

What's the status of this? Is it already possible to have files / buckets with public access? If so please link me a step by step guide.

I want to host my Podcast Mp3s via OVHs S3.

Thanks a lot.

Hi @pinguin999, first of all, thank you for your commitment with OVHcloud and your interest in this feature. We are currently working very hard to make sure this feature meets the highest quality standards and is delivered very soon.

Back to your use case, currently, it is quite possible to make your bucket "public" by setting the ACLs on your bucket and its objects. You could use the "public-read" predefined ACL as mentioned in our doc.

You will have to set the ACLs for all your objects though which can be a little tedious that's why bucket policies is great because it offers an elegant solution to do that.

ogmkp commented 5 months ago

In addition to having no effect on the objects from the ACLs bucket, I can't get the ACLs on an object and I cannot put acl on the object when its uploaded from the OVH web manager:

$ aws s3api get-object-acl --bucket myhollybucket --key test1.txt
An error occurred (AccessDenied) when calling the GetObjectAcl operation: Access Denied.

$ aws s3api put-object-acl --bucket myhollybucket --acl public-read --key test1.txt
An error occurred (AccessDenied) when calling the PutObjectAcl operation: Access Denied.

But I can download the object with my user. My user have "FULL_CONTROL" on the bucket. Putting ACL on objects uploaded with awscli works. Do the OVH manager set bad user on files ?

lason-ovh commented 5 months ago

In addition to having no effect on the objects from the ACLs bucket, I can't get the ACLs on an object and I cannot put acl on the object when its uploaded from the OVH web manager:
$ aws s3api get-object-acl --bucket myhollybucket --key test1.txt
An error occurred (AccessDenied) when calling the GetObjectAcl operation: Access Denied.

$ aws s3api put-object-acl --bucket myhollybucket --acl public-read --key test1.txt
An error occurred (AccessDenied) when calling the PutObjectAcl operation: Access Denied.
But I can download the object with my user. My user have "FULL_CONTROL" on the bucket. Putting ACL on objects uploaded with awscli works. Do the OVH manager set bad user on files ?

The OVH web manager uses a different user and a different set of credentials than the user you associate with your bucket during creation. Hence, when you upload an object via the UI, its owner is not the same as the user you have configured in your cli. Moreover, the web manager does not offer the option to set the acl on objects, you will have to set them via the cli.

ogmkp commented 5 months ago

In addition to having no effect on the objects from the ACLs bucket, I can't get the ACLs on an object and I cannot put acl on the object when its uploaded from the OVH web manager:
$ aws s3api get-object-acl --bucket myhollybucket --key test1.txt
An error occurred (AccessDenied) when calling the GetObjectAcl operation: Access Denied.

$ aws s3api put-object-acl --bucket myhollybucket --acl public-read --key test1.txt
An error occurred (AccessDenied) when calling the PutObjectAcl operation: Access Denied.
But I can download the object with my user. My user have "FULL_CONTROL" on the bucket. Putting ACL on objects uploaded with awscli works. Do the OVH manager set bad user on files ?
The OVH web manager uses a different user and a different set of credentials than the user you associate with your bucket during creation. Hence, when you upload an object via the UI, its owner is not the same as the user you have configured in your cli. Moreover, the web manager does not offer the option to set the acl on objects, you will have to set them via the cli.

I don't understand, the bucket was created from the OVH manager and my user have FULL CONTROL on the bucket with get-bucket-acl, If different credentials are used, why I can see the bucket ACLs, I can set bucket ACLs, can download / upload objects but can't put acl on objects (becauses objects upload from OVH manager don't inherit the user selected on bucket creation)? So avoid OVH manager.

lason-ovh commented 5 months ago

In addition to having no effect on the objects from the ACLs bucket, I can't get the ACLs on an object and I cannot put acl on the object when its uploaded from the OVH web manager:
$ aws s3api get-object-acl --bucket myhollybucket --key test1.txt
An error occurred (AccessDenied) when calling the GetObjectAcl operation: Access Denied.

$ aws s3api put-object-acl --bucket myhollybucket --acl public-read --key test1.txt
An error occurred (AccessDenied) when calling the PutObjectAcl operation: Access Denied.
But I can download the object with my user. My user have "FULL_CONTROL" on the bucket. Putting ACL on objects uploaded with awscli works. Do the OVH manager set bad user on files ?
The OVH web manager uses a different user and a different set of credentials than the user you associate with your bucket during creation. Hence, when you upload an object via the UI, its owner is not the same as the user you have configured in your cli. Moreover, the web manager does not offer the option to set the acl on objects, you will have to set them via the cli.
I don't understand, the bucket was created from the OVH manager and my user have FULL CONTROL on the bucket with get-bucket-acl, If different credentials are used, why I can see the bucket ACLs, I can set bucket ACLs, can download / upload objects but can't put acl on objects (becauses objects upload from OVH manager don't inherit the user selected on bucket creation)? So avoid OVH manager.

Basically, yes, for now, avoid uploading objects using the OVH manager. When you create a bucket with the manager, there is a step where you associate a S3 user to it. That S3 user may have full control over the bucket and therefore, using that user, you can set bucket/object acls, download objects, ...etc. but if you upload objects via the manager, you are using another user and therefore another set of credentials.

ogmkp commented 5 months ago

Thank you for your feedback.

rgdev commented 3 months ago

What is the ETA on this ? per-object policies are prone to errors (every app interacting with a bucket needs to be aware of the rules) so bucket policies would be very useful to us.

xyba1337 commented 2 months ago

Why in the world isn't this available yet? If you're offering "s3 compatible object storage," why wouldn't you include such a crucial option? Almost everyone uses buckets to showcase public images and more to their users.

Well, thanks for wasting my time, I suppose. Time to cancel my subscription...

xyba1337 commented 2 months ago

For anyone looking for a solution to easily bulk update all their stored objects, here's a solution I've come up with:

Download the software Cyberduck.
Establish a new connection, and in the dropdown, select "Amazon S3".

Fill in your credentials and connect to your bucket.
Right click on the folder where all your files are stored in, go to Info > Permissions.
Click the small chevron-down icon on the cog wheel at the bottom left-hand corner and select "Everyone".

Set the Permission for "Everyone" to "Read".

Wait for the update operation to complete, took quite a while for my ~10k files. Now you'll need to figure out how to set the ACL (Access Control List) correctly when uploading new objects. I will cancel my subscription until they've fixed this issue.

kimlop commented 1 month ago

@agoude @lason-ovh any updates regarding this feature? it is really necessary to compete with other providers

lason-ovh commented 1 month ago

@agoude @lason-ovh any updates regarding this feature? it is really necessary to compete with other providers

Hi, this feature is expected to come within the next 9 months, as of now, we are fully committed to deliver the lifecycle feature which is expected to come in the next 3 months.

ovh / public-cloud-roadmap

Object Storage - bucket policies #260