search

peak / s5cmd

Parallel S3 and local filesystem execution tool.
MIT License
2.72k stars 241 forks source link

cp: acl: seems acl set broken in 2.2.1 #657

Closed k0ste closed 1 year ago

k0ste commented 1 year ago

Hi, the acl seems was omitted, when I was try cp with 2.2.1 version

√ /tmp % s5cmd --stat cp --acl "public-read" "*rpm" s3://k0ste/rpm-legacy/
cp netdata-plugin-perf-1.42.2-1.el8.x86_64.rpm s3://k0ste/rpm-legacy/netdata-plugin-perf-1.42.2-1.el8.x86_64.rpm
cp netdata-plugin-pythond-1.42.2-1.el8.x86_64.rpm s3://k0ste/rpm-legacy/netdata-plugin-pythond-1.42.2-1.el8.x86_64.rpm
cp netdata-plugin-apps-1.42.2-1.el8.x86_64.rpm s3://k0ste/rpm-legacy/netdata-plugin-apps-1.42.2-1.el8.x86_64.rpm
cp netdata-plugin-debugfs-1.42.2-1.el8.x86_64.rpm s3://k0ste/rpm-legacy/netdata-plugin-debugfs-1.42.2-1.el8.x86_64.rpm
cp netdata-plugin-ebpf-1.42.2-1.el8.x86_64.rpm s3://k0ste/rpm-legacy/netdata-plugin-ebpf-1.42.2-1.el8.x86_64.rpm
cp netdata-ebpf-legacy-code-1.42.2-1.el8.x86_64.rpm s3://k0ste/rpm-legacy/netdata-ebpf-legacy-code-1.42.2-1.el8.x86_64.rpm
cp netdata-1.42.2-1.el8.x86_64.rpm s3://k0ste/rpm-legacy/netdata-1.42.2-1.el8.x86_64.rpm

Operation   Total   Error   Success
cp      1   0   1
√ 111 % s3cmd info s3://k0ste/rpm-legacy/netdata-1.42.2-1.el8.x86_64.rpm
s3://k0ste/rpm-legacy/netdata-1.42.2-1.el8.x86_64.rpm (object):
   File size: 13778140
   Last mod:  Wed, 30 Aug 2023 13:40:07 GMT
   MIME type: application/octet-stream
   Storage:   STANDARD
   MD5 sum:   10885f5846e179ae10e3b764662e9f05
   SSE:       none
   Policy:    none
   CORS:      none
   ACL:       <account>: FULL_CONTROL <------ no ACL for anon-read

Checked with s3cmd:

√ /tmp % s3cmd --recursive --acl-public put 111/ s3://k0ste/rpm-legacy/
upload: '111/netdata-1.42.2-1.el8.x86_64.rpm' -> 's3://k0ste/rpm-legacy/netdata-1.42.2-1.el8.x86_64.rpm'  [1 of 7]
 13778140 of 13778140   100% in    1s    10.12 MB/s  done
upload: '111/netdata-ebpf-legacy-code-1.42.2-1.el8.x86_64.rpm' -> 's3://k0ste/rpm-legacy/netdata-ebpf-legacy-code-1.42.2-1.el8.x86_64.rpm'  [2 of 7]
 3694284 of 3694284   100% in    0s    15.77 MB/s  done
upload: '111/netdata-plugin-apps-1.42.2-1.el8.x86_64.rpm' -> 's3://k0ste/rpm-legacy/netdata-plugin-apps-1.42.2-1.el8.x86_64.rpm'  [3 of 7]
 385860 of 385860   100% in    0s     4.23 MB/s  done
upload: '111/netdata-plugin-debugfs-1.42.2-1.el8.x86_64.rpm' -> 's3://k0ste/rpm-legacy/netdata-plugin-debugfs-1.42.2-1.el8.x86_64.rpm'  [4 of 7]
 348728 of 348728   100% in    0s     3.05 MB/s  done
upload: '111/netdata-plugin-ebpf-1.42.2-1.el8.x86_64.rpm' -> 's3://k0ste/rpm-legacy/netdata-plugin-ebpf-1.42.2-1.el8.x86_64.rpm'  [5 of 7]
 599716 of 599716   100% in    0s     5.07 MB/s  done
upload: '111/netdata-plugin-perf-1.42.2-1.el8.x86_64.rpm' -> 's3://k0ste/rpm-legacy/netdata-plugin-perf-1.42.2-1.el8.x86_64.rpm'  [6 of 7]
 347276 of 347276   100% in    0s  1944.90 KB/s  done
upload: '111/netdata-plugin-pythond-1.42.2-1.el8.x86_64.rpm' -> 's3://k0ste/rpm-legacy/netdata-plugin-pythond-1.42.2-1.el8.x86_64.rpm'  [7 of 7]
 270216 of 270216   100% in    0s     2.80 MB/s  done
√ /tmp % s3cmd info s3://k0ste/rpm-legacy/netdata-1.42.2-1.el8.x86_64.rpm
s3://k0ste/rpm-legacy/netdata-1.42.2-1.el8.x86_64.rpm (object):
   File size: 13778140
   Last mod:  Wed, 30 Aug 2023 13:46:13 GMT
   MIME type: application/x-rpm
   Storage:   STANDARD
   MD5 sum:   10885f5846e179ae10e3b764662e9f05
   SSE:       none
   Policy:    none
   CORS:      none
   ACL:       *anon*: READ  <----- ACL for anon-read present
   ACL:       <account>: FULL_CONTROL
denizsurmeli commented 1 year ago

Hi, couldn't reproduce it. Here are the steps I have followed: 1) Copy objects without setting acl with s5cmd and check one of the files information:

./s5cmd cp "vendor/*" "s3://tmpbd/vendor/"
s3cmd info s3://tmpbd/vendor/golang.org/x/sys/windows/zerrors_windows.go

Got the result:

s3://tmpbd/vendor/golang.org/x/sys/windows/zerrors_windows.go (object):
   File size: 945502
   Last mod:  Thu, 31 Aug 2023 06:22:47 GMT
   MIME type: text/plain; charset=utf-8
   Storage:   STANDARD
   MD5 sum:   3bbd2e1b04b33a1007929d928ac6a7d9
   SSE:       AES256
   Policy:    none
   CORS:      none
   ACL:       denizsurmeli: FULL_CONTROL

Flushed the contents, repeating with:

2) Copy the objects with your use case using the following commands:

./s5cmd cp --acl "public-read" "vendor/*" "s3://tmpbd/vendor/"
s3cmd info s3://tmpbd/vendor/golang.org/x/sys/windows/zerrors_windows.go

Got the result:

s3://tmpbd/vendor/golang.org/x/sys/windows/zerrors_windows.go (object):
   File size: 945502
   Last mod:  Thu, 31 Aug 2023 06:26:20 GMT
   MIME type: text/plain; charset=utf-8
   Storage:   STANDARD
   MD5 sum:   3bbd2e1b04b33a1007929d928ac6a7d9
   SSE:       AES256
   Policy:    none
   CORS:      none
   ACL:       denizsurmeli: FULL_CONTROL
   ACL:       *anon*: READ
   URL:       http://tmpbd.s3.amazonaws.com/vendor/golang.org/x/sys/windows/zerrors_windows.go

If you think that I wasn't following the right steps, we can try to reproduce it together.

k0ste commented 1 year ago

Step-by-step with your tests:

  1. No ACL
√ s5cmd % s5cmd cp "vendor/*" "s3://tmpbd/vendor/"
√ s5cmd % s3cmd info s3://tmpbd/vendor/golang.org/x/sys/windows/zerrors_windows.go
s3://tmpbd/vendor/golang.org/x/sys/windows/zerrors_windows.go (object):
   File size: 945502
   Last mod:  Thu, 31 Aug 2023 10:15:22 GMT
   MIME type: text/plain; charset=utf-8
   Storage:   STANDARD
   MD5 sum:   3bbd2e1b04b33a1007929d928ac6a7d9
   SSE:       none
   Policy:    none
   CORS:      none
   ACL:       Konstantin Shalygin: FULL_CONTROL

Flushed the contents, repeating with:

  1. With ACL
√ s5cmd cp --acl "public-read" "vendor/*" "s3://tmpbd/vendor/"
√ s5cmd % s3cmd info s3://tmpbd/vendor/golang.org/x/sys/windows/zerrors_windows.go
s3://tmpbd/vendor/golang.org/x/sys/windows/zerrors_windows.go (object):
   File size: 945502
   Last mod:  Thu, 31 Aug 2023 10:17:39 GMT
   MIME type: text/plain; charset=utf-8
   Storage:   STANDARD
   MD5 sum:   3bbd2e1b04b33a1007929d928ac6a7d9
   SSE:       none
   Policy:    none
   CORS:      none
   ACL:       Konstantin Shalygin: FULL_CONTROL

The binary & Kernel:

√ s5cmd % s5cmd version
v2.2.1-be63977
√ s5cmd % uname -a
Darwin Kunteynir.local 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul  5 22:17:35 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T8112 arm64
denizsurmeli commented 1 year ago

Hi, sorry for late reply. I have looked at the documentation a bit and couldn't find what might be causing the issue. When you look at the permissions from the AWS console rather than checking with s3cmd, are the permissions are not granted there too ? There are many moving parts, sorry for the burden.

k0ste commented 1 year ago

Hi, sorry for late reply. I have looked at the documentation a bit and couldn't find what might be causing the issue. When you look at the permissions from the AWS console rather than checking with s3cmd, are the permissions are not granted there too ? There are many moving parts, sorry for the burden.

This is S3 compatible storage, not the AWS But yes. When you try to download object with http client the answer will be 403 Forbidded, and 200 when upload performed via s3cmd

k0ste commented 1 year ago

The putObject trace from Darwin

-----------------------------------------------------
DEBUG: Request s3/PutObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
PUT /k0ste/111.txt HTTP/1.1
Host: <host>
User-Agent: aws-sdk-go/1.44.256 (go1.20.7; darwin; arm64) S3Manager
Content-Length: 0
Authorization: AWS4-HMAC-SHA256 Credential=<snip>/20230902/us-east-1/s3/aws4_request, SignedHeaders=content-md5;content-type;host;x-amz-content-sha256;x-amz-date, Signature=6afd65fea9260014f3718783ad0a13cdcd18eebb39721762b4168b1c0a1412f6
Content-Md5: 1B2M2Y8AsgTpgAmY7PhCfg==
Content-Type: text/plain; charset=utf-8
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230902T134558Z
Accept-Encoding: gzip

And from Linux x86_64

-----------------------------------------------------
DEBUG: Request s3/PutObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
PUT /k0ste/222.txt HTTP/1.1
Host: <host>
User-Agent: aws-sdk-go/1.44.256 (go1.21.0; linux; amd64) S3Manager
Content-Length: 0
Authorization: AWS4-HMAC-SHA256 Credential=<snip>/20230902/us-east-1/s3/aws4_request, SignedHeaders=content-md5;content-type;host;x-amz-content-sha256;x-amz-date, Signature=90327aed756a31c39c906e97680c313c3d434651a59730954254ff2cec02192d
Content-Md5: 1B2M2Y8AsgTpgAmY7PhCfg==
Content-Type: text/plain; charset=utf-8
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230902T134734Z
Accept-Encoding: gzip

So, the x-amz-acl header is omitted

s3cmd requst:

DEBUG: Canonical Request:
PUT
/k0ste/111.txt

content-length:0
content-type:inode/x-empty
host:<host>
x-amz-acl:public-read
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20230902T135333Z
x-amz-meta-s3cmd-attrs:atime:1693662336/ctime:1693662336/gid:0/gname:wheel/md5:d41d8cd98f00b204e9800998ecf8427e/mode:33188/mtime:1693662336/uid:501/uname:k0ste
x-amz-storage-class:STANDARD

content-length;content-type;host;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
denizsurmeli commented 1 year ago

You are right, s5cmd omits the header in version v2.2.1-be63977. Thanks for your contribution, a fix will be prepared ASAP.

k0ste commented 1 year ago

Please tag for new bugfix release

denizsurmeli commented 1 year ago

https://github.com/peak/s5cmd/releases/tag/v2.2.2, thanks for your contribution again.