search

kahing / goofys

a high-performance, POSIX-ish Amazon S3 file system written in Go
Apache License 2.0
5.19k stars 521 forks source link

can't remove file from S3 (localstack) #467

Open ykudashov opened 5 years ago

ykudashov commented 5 years ago

'Permission denied' for root user when removing file from localstack's s3 implementation.

How to reproduce: docker run -d -p 4572:4572 -p --rm -e HOSTNAME_EXTERNAL=localstack -e DATA_DIR=/tmp/localstack/data -e SERVICES=s3 localstack/localstack aws --endpoint-url=http://localhost:4572 s3api create-bucket --bucket aws-bucket --region us-east-1 sudo su - root mkdir -p /s3/test /usr/bin/goofys --endpoint http://localhost:4572 aws-bucket /s3/test /usr/bin/goofys -v goofys version 0.21.0-42c567cafc27c7f68086f9709a19f03b56e2e80e echo '???' > /tmp/test.txt cp /tmp/test.txt /s3/test rm /s3/test/test.txt rm: cannot remove '/s3/test/test.txt' : Permission denied

kahing commented 5 years ago

could you paste the output to --debug_s3?

bs-thomas commented 4 years ago

Hello @kahing

Unfortunately, I am facing the same issue with 0.21.0-42c567cafc27c7f68086f9709a19f03b56e2e80e

I can read files, I create files, I can replace files, but I cannot remove. It seems to be returning a 403 Forbbiden as response to me when the DELETE command is issued over to S3.

I have an ECS container running this.

This is my command to run goofys: goofys --debug_s3 -f -o allow_other --region ap-southeast-1 --stat-cache-ttl 0m1s --type-cache-ttl 0m1s --dir-mode 0777 --file-mode 0777 beamstyle-web-01:clients/www.polyauction.com.hk-staging/erp-import/xml/uploaded/ /srv/s3

When I try to delete, this comes out:

` 2019/10/27 06:58:42.174485 s3.DEBUG DEBUG: Request s3/ListObjects Details: ---[ REQUEST POST-SIGN ]----------------------------- GET /beamstyle-web-01?delimiter=%2F&max-keys=1&prefix=clients%2Fwww.polyauction.com.hk-staging%2Ferp-import%2Fxml%2Fuploaded%2Ftest02.txt%2F HTTP/1.1 Host: s3.ap-southeast-1.amazonaws.com User-Agent: aws-sdk-go/1.17.13 (go1.10.4; linux; amd64) exec-env/AWS_ECS_EC2 Accept-Encoding: identity Authorization: AWS4-HMAC-SHA256 Credential=ASIAU3XUKH4GFLX435XU/20191027/ap-southeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=837d7872c0e079bc4909dd2e0fb01c07ceadc0a396399d37cc0a6f03c6dff38c X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 X-Amz-Date: 20191027T065842Z X-Amz-Security-Token: AgoJb3JpZ2luX2VjEGcaDmFwLXNvdXRoZWFzdC0xIkgwRgIhAO+Wk2+c8iIQzGFiPPJaCqXO5Zo/r2yJ+L+9hgC4BhSuAiEAilq1e54gFGcUj0FyeU72ntj78cLC3G0FF0j/gyODUrMq5AMIcBAAGgwzMzQ0NDYwODM4NTIiDO/HPdTpvnqUk8vyvirBAxSdeOnDwzcXplSbZryVltrcotJczq18JcqOGVs736uXX4BKibivK49PMx7reY3ZsqiJDshqwjMjRuotBkE1T7mhlow9SH+CIPHTfzmixGmS/0CbBztu0Ys1zxajnIiSJUcUfe5MTCg1wUCqeW7Ri9L6pGySBUhvuPXti4ByKevJ0jgnD/Vi3cAgn39SHFqNmKFO3Wafpl4rJf9OjKaaCH+eLcqEvcHyV/6/uEgdo4bpANwAmSQxkn/QpXM9umtK9Z608KvcCmJ/v5tGpJzYLpVqORstK6f5v039mX5UVt4qJr/Kr4YsA/LRJwbZ48AzPjD9yPxl/cXJkM8TXOgQCdx2AdxHvizk+Fz4q/1KDU0ApRZXb8Nik/i7cjHocNnLIMM1pO9KkQHJ78dMHsffLfqq1Od8vgVVZXTqhz/tYqj7VdW1gYhdhXJAz9ubQEgY9G+qhOFobh+W6Xght9P0ECjk4gqgl+1Tr6uiweiKCSLHBf45X1RSaiBW5qTFcH1oMaxgH6DdCXAatgzi1nd1Thv+MlHXFoMCA71U9Xt/pfAgfNvT926nraT7ikhap1XzoNA5paD1bWSHlDVCwtY/C7S1MJXy1O0FOrMBKZHHorUSz/eBV4LeMxxC/b1QCSmJCul8J3v8lhB3emOVJ1PRNd2FQbyBUVXI0V691whe+1zL21dZoOYzZkIDq+Zod5eGiGFgny+BU56+MNE3leWadn9ywk4Ukoqx/YL0YBzrUu/44wS7fFlYwAAWT6Qj7LeuWtUiNd5K5WkagWEoXR2pNlH2uN93pO+ieW3ftbZebMnMEoKoYNOZ+vlEb9QhuIKVEYruLc8TXzVWG7oixoc=

2019/10/27 06:58:42.175064 s3.DEBUG DEBUG: Request s3/HeadObject Details: ---[ REQUEST POST-SIGN ]----------------------------- HEAD /beamstyle-web-01/clients/www.polyauction.com.hk-staging/erp-import/xml/uploaded/test02.txt/ HTTP/1.1 Host: s3.ap-southeast-1.amazonaws.com User-Agent: aws-sdk-go/1.17.13 (go1.10.4; linux; amd64) exec-env/AWS_ECS_EC2 Authorization: AWS4-HMAC-SHA256 Credential=ASIAU3XUKH4GFLX435XU/20191027/ap-southeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=ea7d17c58d4b696bea34153569eb1006f43d46366d7c71fc6469b88dc8ce85af X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 X-Amz-Date: 20191027T065842Z X-Amz-Security-Token: AgoJb3JpZ2luX2VjEGcaDmFwLXNvdXRoZWFzdC0xIkgwRgIhAO+Wk2+c8iIQzGFiPPJaCqXO5Zo/r2yJ+L+9hgC4BhSuAiEAilq1e54gFGcUj0FyeU72ntj78cLC3G0FF0j/gyODUrMq5AMIcBAAGgwzMzQ0NDYwODM4NTIiDO/HPdTpvnqUk8vyvirBAxSdeOnDwzcXplSbZryVltrcotJczq18JcqOGVs736uXX4BKibivK49PMx7reY3ZsqiJDshqwjMjRuotBkE1T7mhlow9SH+CIPHTfzmixGmS/0CbBztu0Ys1zxajnIiSJUcUfe5MTCg1wUCqeW7Ri9L6pGySBUhvuPXti4ByKevJ0jgnD/Vi3cAgn39SHFqNmKFO3Wafpl4rJf9OjKaaCH+eLcqEvcHyV/6/uEgdo4bpANwAmSQxkn/QpXM9umtK9Z608KvcCmJ/v5tGpJzYLpVqORstK6f5v039mX5UVt4qJr/Kr4YsA/LRJwbZ48AzPjD9yPxl/cXJkM8TXOgQCdx2AdxHvizk+Fz4q/1KDU0ApRZXb8Nik/i7cjHocNnLIMM1pO9KkQHJ78dMHsffLfqq1Od8vgVVZXTqhz/tYqj7VdW1gYhdhXJAz9ubQEgY9G+qhOFobh+W6Xght9P0ECjk4gqgl+1Tr6uiweiKCSLHBf45X1RSaiBW5qTFcH1oMaxgH6DdCXAatgzi1nd1Thv+MlHXFoMCA71U9Xt/pfAgfNvT926nraT7ikhap1XzoNA5paD1bWSHlDVCwtY/C7S1MJXy1O0FOrMBKZHHorUSz/eBV4LeMxxC/b1QCSmJCul8J3v8lhB3emOVJ1PRNd2FQbyBUVXI0V691whe+1zL21dZoOYzZkIDq+Zod5eGiGFgny+BU56+MNE3leWadn9ywk4Ukoqx/YL0YBzrUu/44wS7fFlYwAAWT6Qj7LeuWtUiNd5K5WkagWEoXR2pNlH2uN93pO+ieW3ftbZebMnMEoKoYNOZ+vlEb9QhuIKVEYruLc8TXzVWG7oixoc=

2019/10/27 06:58:42.174810 s3.DEBUG DEBUG: Request s3/HeadObject Details: ---[ REQUEST POST-SIGN ]----------------------------- HEAD /beamstyle-web-01/clients/www.polyauction.com.hk-staging/erp-import/xml/uploaded/test02.txt HTTP/1.1 Host: s3.ap-southeast-1.amazonaws.com User-Agent: aws-sdk-go/1.17.13 (go1.10.4; linux; amd64) exec-env/AWS_ECS_EC2 Authorization: AWS4-HMAC-SHA256 Credential=ASIAU3XUKH4GFLX435XU/20191027/ap-southeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=a2c8bf90724e9cd06c6fdafc2a49c3bab516c6ebefcd0d0064d2c29dd257b3e8 X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 X-Amz-Date: 20191027T065842Z X-Amz-Security-Token: AgoJb3JpZ2luX2VjEGcaDmFwLXNvdXRoZWFzdC0xIkgwRgIhAO+Wk2+c8iIQzGFiPPJaCqXO5Zo/r2yJ+L+9hgC4BhSuAiEAilq1e54gFGcUj0FyeU72ntj78cLC3G0FF0j/gyODUrMq5AMIcBAAGgwzMzQ0NDYwODM4NTIiDO/HPdTpvnqUk8vyvirBAxSdeOnDwzcXplSbZryVltrcotJczq18JcqOGVs736uXX4BKibivK49PMx7reY3ZsqiJDshqwjMjRuotBkE1T7mhlow9SH+CIPHTfzmixGmS/0CbBztu0Ys1zxajnIiSJUcUfe5MTCg1wUCqeW7Ri9L6pGySBUhvuPXti4ByKevJ0jgnD/Vi3cAgn39SHFqNmKFO3Wafpl4rJf9OjKaaCH+eLcqEvcHyV/6/uEgdo4bpANwAmSQxkn/QpXM9umtK9Z608KvcCmJ/v5tGpJzYLpVqORstK6f5v039mX5UVt4qJr/Kr4YsA/LRJwbZ48AzPjD9yPxl/cXJkM8TXOgQCdx2AdxHvizk+Fz4q/1KDU0ApRZXb8Nik/i7cjHocNnLIMM1pO9KkQHJ78dMHsffLfqq1Od8vgVVZXTqhz/tYqj7VdW1gYhdhXJAz9ubQEgY9G+qhOFobh+W6Xght9P0ECjk4gqgl+1Tr6uiweiKCSLHBf45X1RSaiBW5qTFcH1oMaxgH6DdCXAatgzi1nd1Thv+MlHXFoMCA71U9Xt/pfAgfNvT926nraT7ikhap1XzoNA5paD1bWSHlDVCwtY/C7S1MJXy1O0FOrMBKZHHorUSz/eBV4LeMxxC/b1QCSmJCul8J3v8lhB3emOVJ1PRNd2FQbyBUVXI0V691whe+1zL21dZoOYzZkIDq+Zod5eGiGFgny+BU56+MNE3leWadn9ywk4Ukoqx/YL0YBzrUu/44wS7fFlYwAAWT6Qj7LeuWtUiNd5K5WkagWEoXR2pNlH2uN93pO+ieW3ftbZebMnMEoKoYNOZ+vlEb9QhuIKVEYruLc8TXzVWG7oixoc=

2019/10/27 06:58:42.206502 s3.DEBUG DEBUG: Response s3/HeadObject Details: ---[ RESPONSE ]-------------------------------------- HTTP/1.1 200 OK Content-Length: 7 Accept-Ranges: bytes Content-Type: binary/octet-stream Date: Sun, 27 Oct 2019 06:58:43 GMT Etag: "efd4020238e5bd860e727400b3ba050f" Last-Modified: Sun, 27 Oct 2019 06:55:52 GMT Server: AmazonS3 X-Amz-Id-2: Awk+Cp8fOtuQlJWcmQZPyui09gRzqArIounAt7+DNMzi2kg64OsWtdpUcJK0KCWt7kRl8T+YE98= X-Amz-Request-Id: 607CB74E7EEEEA6F

2019/10/27 06:58:42.206576 s3.DEBUG clients/www.polyauction.com.hk-staging/erp-import/xml/uploaded/test02.txt: 7 2019/10/27 06:58:42.207898 s3.DEBUG DEBUG: Request s3/DeleteObject Details: ---[ REQUEST POST-SIGN ]----------------------------- DELETE /beamstyle-web-01/clients/www.polyauction.com.hk-staging/erp-import/xml/uploaded/test02.txt HTTP/1.1 Host: s3.ap-southeast-1.amazonaws.com User-Agent: aws-sdk-go/1.17.13 (go1.10.4; linux; amd64) exec-env/AWS_ECS_EC2 Authorization: AWS4-HMAC-SHA256 Credential=ASIAU3XUKH4GFLX435XU/20191027/ap-southeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=4c3d879a03a5c2b8ccbb0011092b5c39a5809c8d36924966b3a57e4f795bd0f2 X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 X-Amz-Date: 20191027T065842Z X-Amz-Security-Token: AgoJb3JpZ2luX2VjEGcaDmFwLXNvdXRoZWFzdC0xIkgwRgIhAO+Wk2+c8iIQzGFiPPJaCqXO5Zo/r2yJ+L+9hgC4BhSuAiEAilq1e54gFGcUj0FyeU72ntj78cLC3G0FF0j/gyODUrMq5AMIcBAAGgwzMzQ0NDYwODM4NTIiDO/HPdTpvnqUk8vyvirBAxSdeOnDwzcXplSbZryVltrcotJczq18JcqOGVs736uXX4BKibivK49PMx7reY3ZsqiJDshqwjMjRuotBkE1T7mhlow9SH+CIPHTfzmixGmS/0CbBztu0Ys1zxajnIiSJUcUfe5MTCg1wUCqeW7Ri9L6pGySBUhvuPXti4ByKevJ0jgnD/Vi3cAgn39SHFqNmKFO3Wafpl4rJf9OjKaaCH+eLcqEvcHyV/6/uEgdo4bpANwAmSQxkn/QpXM9umtK9Z608KvcCmJ/v5tGpJzYLpVqORstK6f5v039mX5UVt4qJr/Kr4YsA/LRJwbZ48AzPjD9yPxl/cXJkM8TXOgQCdx2AdxHvizk+Fz4q/1KDU0ApRZXb8Nik/i7cjHocNnLIMM1pO9KkQHJ78dMHsffLfqq1Od8vgVVZXTqhz/tYqj7VdW1gYhdhXJAz9ubQEgY9G+qhOFobh+W6Xght9P0ECjk4gqgl+1Tr6uiweiKCSLHBf45X1RSaiBW5qTFcH1oMaxgH6DdCXAatgzi1nd1Thv+MlHXFoMCA71U9Xt/pfAgfNvT926nraT7ikhap1XzoNA5paD1bWSHlDVCwtY/C7S1MJXy1O0FOrMBKZHHorUSz/eBV4LeMxxC/b1QCSmJCul8J3v8lhB3emOVJ1PRNd2FQbyBUVXI0V691whe+1zL21dZoOYzZkIDq+Zod5eGiGFgny+BU56+MNE3leWadn9ywk4Ukoqx/YL0YBzrUu/44wS7fFlYwAAWT6Qj7LeuWtUiNd5K5WkagWEoXR2pNlH2uN93pO+ieW3ftbZebMnMEoKoYNOZ+vlEb9QhuIKVEYruLc8TXzVWG7oixoc= Accept-Encoding: gzip

2019/10/27 06:58:42.208094 s3.DEBUG DEBUG: Response s3/ListObjects Details: ---[ RESPONSE ]-------------------------------------- HTTP/1.1 200 OK Transfer-Encoding: chunked Content-Type: application/xml Date: Sun, 27 Oct 2019 06:58:43 GMT Server: AmazonS3 X-Amz-Bucket-Region: ap-southeast-1 X-Amz-Id-2: h9fEVDLT62BGvJjAQN/jD3KX+42FY/1Jz3JbaW15aXmtL1B1OpXKUIQhi4Ej5I9f391D7v9pB3o= X-Amz-Request-Id: 12A7CD24227AD332

2019/10/27 06:58:42.208133 s3.DEBUG &{ [] [] false } 2019/10/27 06:58:42.212031 s3.DEBUG DEBUG: Response s3/DeleteObject Details: ---[ RESPONSE ]-------------------------------------- HTTP/1.1 403 Forbidden Transfer-Encoding: chunked Content-Type: application/xml Date: Sun, 27 Oct 2019 06:58:41 GMT Server: AmazonS3 X-Amz-Id-2: MY+oKiOkv8E4mFudkHVP+3WlrqckbXJ6B+1xz0FuUmpj/OSqiFe4Yye92BKihMH8o01OIbcJ/Bc= X-Amz-Request-Id: 58A31508AE6E84B4

2019/10/27 06:58:42.212126 s3.DEBUG DEBUG: Validate Response s3/DeleteObject failed, not retrying, error AccessDenied: Access Denied status code: 403, request id: 58A31508AE6E84B4, host id: MY+oKiOkv8E4mFudkHVP+3WlrqckbXJ6B+1xz0FuUmpj/OSqiFe4Yye92BKihMH8o01OIbcJ/Bc= 2019/10/27 06:58:42.212679 s3.DEBUG DEBUG: Response s3/HeadObject Details: ---[ RESPONSE ]-------------------------------------- HTTP/1.1 404 Not Found Transfer-Encoding: chunked Content-Type: application/xml Date: Sun, 27 Oct 2019 06:58:41 GMT Server: AmazonS3 X-Amz-Id-2: jJofkYTaMVtdABuR23ra2scNWU119BzawIGnlJ1Uz7jDW51WL7SrxL4NYthhjfiPfknJ6KNyBpk= X-Amz-Request-Id: 479FE05DC99266D2

2019/10/27 06:58:42.212708 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found status code: 404, request id: 479FE05DC99266D2, host id: jJofkYTaMVtdABuR23ra2scNWU119BzawIGnlJ1Uz7jDW51WL7SrxL4NYthhjfiPfknJ6KNyBpk= `

====

PS: Side question: for some reason I never have to pass in credentials, I have no .git, and no environment variables, but it reads, would you happen to know why?

====

Thanks Kahing!

bs-thomas commented 4 years ago

@kahing Btw, I'm not quite sure where those "404 Not Found"s pop up from. Though, I did trigger the deletion through FTP (via an installed FTP server "vsftpd" inside the container)