search

minio / mc

Unix like utilities for object store
https://min.io/download
GNU Affero General Public License v3.0
2.87k stars 549 forks source link

mc cp adds filesystem attributes in X-Amz-Meta-Mc-Attrs when any --attr is specified #3579

Closed kevinlul closed 3 years ago

kevinlul commented 3 years ago

Expected behavior

According to documentation, preserving filesystem attributes is opt-in with the --preserve flag or -a. Therefore when I do a regular mc cp LOCALFILE MINIO_HOST/BUCKET, I don't expect my local uname, uid, gname, and gid to be leaked when I just want to specify a Cache-Control header.

Actual behavior

mc adds filesystem metadata into X-Amz-Meta-Mc-Attrs when --attr Cache-Control=no-cache and similar are used. This can be overwritten by explicitly blanking the header with --attr 'Cache-Control=no-cache;X-Amz-Meta-Mc-Attrs=, but this still adds a blank X-Amz-Meta-Mc-Attrs header.

Steps to reproduce the behavior

$ touch foobar
$ mc mb play/zzz
Bucket created successfully `play/zzz`.
$ mc cp --attr Cache-Control=no-cache foobar play/zzz
$ mc stat play/zzz/foobar
Name      : foobar
Date      : 2021-01-20 16:20:02 EST
Size      : 0 B
ETag      : d41d8cd98f00b204e9800998ecf8427e
Type      : file
Metadata  :
  X-Amz-Meta-Mc-Attrs: atime:1611177602#786120200/gid:1000/gname:debian/mode:33188/mtime:1611177602#786120200/uid:1000/uname:debian
  Cache-Control      : no-cache
  Content-Type       : application/octet-stream

Compare with

$ mc cp foobar play/zzz
$ mc stat play/zzz/foobar
Name      : foobar
Date      : 2021-01-20 16:21:47 EST
Size      : 0 B
ETag      : d41d8cd98f00b204e9800998ecf8427e
Type      : file
Metadata  :
  Content-Type: application/octet-stream

With the override workaround

$ mc cp --attr 'Cache-Control=no-cache;X-Amz-Meta-Mc-Attrs=' foobar play/zzz
$ mc stat play/zzz/foobar
Name      : foobar
Date      : 2021-01-20 16:26:47 EST
Size      : 0 B
ETag      : d41d8cd98f00b204e9800998ecf8427e
Type      : file
Metadata  :
  X-Amz-Meta-Mc-Attrs:
  Cache-Control      : no-cache
  Content-Type       : application/octet-stream

mc --version

mc version RELEASE.2021-01-16T02-45-34Z

System information

Reproduced on macOS 10.14.6 and Debian 10 in WSL1

Jean-Baptiste-Lasselle commented 1 year ago

Just dropping an info (i think of the github reader digging with his error message...) : 

^Cjbl@pokus1:~/reciproque$ mc mirror -w ${PESTO_MINIO_PROFILE}/testjb $(pwd)/optique
.../rep_sur_win/README.pesto.md: 144 B / 144 B ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.01 KiB/smc: <ERROR> Failed to copy `http://minio.pokus.io:9000/testjb/README.prusses.md`. Error in parsing file system attribute
mc: <ERROR> Failed to copy `http://minio.pokus.io:9000/testjb/C:/Users/Utilisateur/transit_pesto/rep_sur_win/README.pesto.md`. Error in parsing file system attribute
mc: <ERROR> Failed to copy `http://minio.pokus.io:9000/testjb/README.pesto.md`. Error in parsing file system attribute
...:9000/testjb/README.pesto.md: 144 B / 144 B ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8 B/s^Cjbl@pokus1:~/reciproque$ rm -fr $(pwd)/optique

So there is what you are searching for, dear reader :

Utilisateur@Utilisateur-PC MINGW64 ~/transit_pesto $ export MC_ATTRS="atime:$(date +%s)#323091789/gid:1000/gname:jbl/mode:33188/mtime:$(date +%s)#323091789/uid:1000/uname:jbl"

Utilisateur@Utilisateur-PC MINGW64 ~/transit_pesto $ export META_PESTO="Pesto1=greenIsTheTradition;Pesto2=redTastesDifferent;Mc-Attrs=${MC_ATTRS}"

Utilisateur@Utilisateur-PC MINGW64 ~/transit_pesto $ mc mirror --debug --attr "${META_PESTO}" --watch $(pwd)/rep_sur_win/ pesto/testjb/


* copy-paster to run it : 

```bash
export MC_ATTRS="atime:$(date +%s)#323091789/gid:1000/gname:jbl/mode:33188/mtime:$(date +%s)#323091789/uid:1000/uname:jbl"
export META_PESTO="Pesto1=greenIsTheTradition;Pesto2=redTastesDifferent;Mc-Attrs=${MC_ATTRS}"
# mc mirror --debug --attr "${META_PESTO}" --watch $(pwd)/rep_sur_win/ pesto/testjb/

My output (check out the HTTP Request headers): 

$ mc mirror --remove --debug --attr "${META_PESTO}" --watch $(pwd)/rep_sur_win/ pesto/testjb/
mc: <DEBUG> GET /testjb/?location= HTTP/1.1
Host: minio.pokus.io:9000
User-Agent: MinIO (windows; amd64) minio-go/v7.0.40 mc/RELEASE.2022-10-06T01-20-06Z
Authorization: AWS4-HMAC-SHA256 Credential=minioadmin/20230617/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230617T000738Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 128
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Sat, 17 Jun 2023 00:07:38 GMT
Server: nginx/1.19.2
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: e0c385c033c4356721cc9121d3109c9b9bfdefb22fd2747078acd22328799e36
X-Amz-Request-Id: 1769499979FF585F
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

mc: <DEBUG> Response Time:  7.9689ms

mc: <DEBUG> GET /testjb/?delimiter=&encoding-type=url&fetch-owner=true&list-type=2&metadata=true&prefix=%2F HTTP/1.1
Host: minio.pokus.io:9000
User-Agent: MinIO (windows; amd64) minio-go/v7.0.40 mc/RELEASE.2022-10-06T01-20-06Z
Authorization: AWS4-HMAC-SHA256 Credential=minioadmin/20230617/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230617T000738Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 270
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Sat, 17 Jun 2023 00:07:38 GMT
Server: nginx/1.19.2
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: 4de9f6f45bfb3a31ea395f3d063545f58e6a1995d2b05dee6b579f19079a6455
X-Amz-Request-Id: 176949997A30D446
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

mc: <DEBUG> Response Time:  3.6221ms

`C:/Users/Utilisateur/transit_pesto/rep_sur_win/README.pesto.md` -> `pesto/testjb/C:/Users/Utilisateur/transit_pesto/rep_sur_win/README.pesto.md`
`C:/Users/Utilisateur/transit_pesto/rep_sur_win/README.prusses.md` -> `pesto/testjb/C:/Users/Utilisateur/transit_pesto/rep_sur_win/README.prusses.md`
mc: <DEBUG> PUT /testjb//C%3A/Users/Utilisateur/transit_pesto/rep_sur_win/README.prusses.md HTTP/1.1
Host: minio.pokus.io:9000
User-Agent: MinIO (windows; amd64) minio-go/v7.0.40 mc/RELEASE.2022-10-06T01-20-06Z
Content-Length: 251
Authorization: AWS4-HMAC-SHA256 Credential=minioadmin/20230617/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-meta-mc-attrs;x-amz-meta-mm-source-mtime,Signature=**REDACTED**
Content-Type: text/markdown
X-Amz-Content-Sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
X-Amz-Date: 20230617T000738Z
X-Amz-Decoded-Content-Length: 78
X-Amz-Meta-Mc-Attrs: atime:1686959271#323091789/gid:1000/gname:jbl/mode:33188/mtime:1686959271#323091789/uid:1000/uname:jbl
X-Amz-Meta-Mm-Source-Mtime: 2023-06-17T01:30:06.8362712+02:00
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 0
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Date: Sat, 17 Jun 2023 00:07:39 GMT
Etag: "b50dec8f0d005df11730509ccc4975d0"
Server: nginx/1.19.2
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: 46efbbb7efbd81c7d995bde03cc6fabf60c12f80d4e074c1c972dbc4d583c3d4
X-Amz-Request-Id: 176949997C68587A
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

mc: <DEBUG> Response Time:  39.1612ms

mc: <DEBUG> PUT /testjb//C%3A/Users/Utilisateur/transit_pesto/rep_sur_win/README.pesto.md HTTP/1.1
Host: minio.pokus.io:9000
User-Agent: MinIO (windows; amd64) minio-go/v7.0.40 mc/RELEASE.2022-10-06T01-20-06Z
Content-Length: 245
Authorization: AWS4-HMAC-SHA256 Credential=minioadmin/20230617/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-meta-mc-attrs;x-amz-meta-mm-source-mtime,Signature=**REDACTED**
Content-Type: text/markdown
X-Amz-Content-Sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
X-Amz-Date: 20230617T000738Z
X-Amz-Decoded-Content-Length: 72
X-Amz-Meta-Mc-Attrs: atime:1686959271#323091789/gid:1000/gname:jbl/mode:33188/mtime:1686959271#323091789/uid:1000/uname:jbl
X-Amz-Meta-Mm-Source-Mtime: 2023-06-16T23:44:16.2449642+02:00
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 0
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Date: Sat, 17 Jun 2023 00:07:39 GMT
Etag: "fab56f462375dcb7dcd151b03aab0e84"
Server: nginx/1.19.2
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: 87ef3db1a67a60b69bcfee10c0874f8a253fc9bc1bf4e58d7314742c975c4eb6
X-Amz-Request-Id: 176949997C969B57
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

mc: <DEBUG> Response Time:  40.5294ms