search

axboe / fio

Flexible I/O Tester
GNU General Public License v2.0
5.01k stars 1.23k forks source link

Problem with authentication when http ioengine communicates with AWS S3 #1231

Open mbukatov opened 3 years ago

mbukatov commented 3 years ago

Description of the bug:

When I try to use http ioengine in s3 mode (http_mode=s3) so that fio will write data on given AWS S3 bucket, the operation fails on HTTP/1.1 403 Forbidden error.

Environment: Fedora 33

fio version: fio-3.21 (from rpm package), fio-3.26-59-gb54e0 (build from source)

Reproduction steps

  1. Create AWS S3 bucket
  2. Store AWS credentials in env file (so that we don't have to edit fio job file later), which looks like this:
export BUCKET_HOST=s3.us-east-2.amazonaws.com
export BUCKET_PORT=443
export AWS_REGION=us-east-2
export AWS_SECRET_ACCESS_KEY=...
export AWS_ACCESS_KEY_ID=...
export BUCKET_NAME=mbukatov-fio
  1. Source the env file to make the env variables available.
  2. Create minimal s3 fio job file to reproduce the problem: 
    [simple-write]                                                                  
readwrite=write                                                                 
ioengine=http                                                                   
filename=/${BUCKET_NAME}/object                                                 
https=insecure                                                                  
http_mode=s3                                                                    
http_host=${BUCKET_HOST}:${BUCKET_PORT}                                         
http_s3_keyid=${AWS_ACCESS_KEY_ID}                                              
http_s3_key=${AWS_SECRET_ACCESS_KEY}                                            
http_s3_region=${AWS_REGION}                                                    
size=1M
  3. Run the fio job: ./fio -f workload.fio. 
    
simple-write: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=http, iodepth=1
fio-3.26-59-gb54e0
Starting 1 process
DDIR_WRITE failed with HTTP status code 403
fio: pid=40126, err=-1/file:engines/http.c:570, func=transfer, error=Unknown error -1
3;fio-3.26-59-gb54e0;simple-write;0;-1;0;0;0;0;0;0;0.000000;0.000000;0;0;0.000000;0.000000;1.000000%=0;5.000000%=0;10.000000%=0;20.000000%=0;30.000000%=0;40.000000%=0;50.000000%=0;60.000000%=0;70.000000%=0;80.000000%=0;90.000000%=0;95.000000%=0;99.000000%=0;99.500000%=0;99.900000%=0;99.950000%=0;99.990000%=0;0%=0;0%=0;0%=0;0;0;0.000000;0.000000;0;0;0.000000%;0.000000;0.000000;0;0;0;0;0;0;0.000000;0.000000;0;0;0.000000;0.000000;1.000000%=0;5.000000%=0;10.000000%=0;20.000000%=0;30.000000%=0;40.000000%=0;50.000000%=0;60.000000%=0;70.000000%=0;80.000000%=0;90.000000%=0;95.000000%=0;99.000000%=0;99.500000%=0;99.900000%=0;99.950000%=0;99.990000%=0;0%=0;0%=0;0%=0;0;0;0.000000;0.000000;0;0;0.000000%;0.000000;0.000000;2.097902%;0.174825%;26;0;242;100.0%;0.0%;0.0%;0.0%;0.0%;0.0%;0.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%

simple-write: (groupid=0, jobs=1): err=-1 (file:engines/http.c:570, func=transfer, error=Unknown error -1): pid=40126: Fri May 21 17:36:45 2021 cpu : usr=2.10%, sys=0.17%, ctx=26, majf=0, minf=242 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):



**Additional information:**

When I drop `https=insecure` from the job file, fio gets stuck instead, but the end result is the same: no objects are written in the bucket.

I'm able to write data to the bucket via simple python script (using boto module) reusing the same env variables with AWS credentials (so there should be no problem with credentials or access rights itself).
sitsofe commented 3 years ago

@l-mb any ideas on this one?

nfonseca commented 4 months ago

I am having exactly the same issue when running FIO against a S3 compatible ObjectStorage. Keys are fine as I can connect to the ObjectStorage with awscli

create: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=http, iodepth=1
fio-3.36-88-g9af4af
Starting 1 process
== Info:   Trying 172.20.46.10:443...
== Info: Connected to 172.20.46.10 (172.20.46.10) port 443 (#0)
== Info: ALPN, offering h2
== Info: ALPN, offering http/1.1
== Info: TLSv1.0 (OUT), TLS header, Certificate Status (22):
== Info: TLSv1.3 (OUT), TLS handshake, Client hello (1):
== Info: TLSv1.2 (IN), TLS header, Certificate Status (22):
== Info: TLSv1.3 (IN), TLS handshake, Server hello (2):
== Info: TLSv1.2 (IN), TLS handshake, Certificate (11):
== Info: TLSv1.2 (IN), TLS handshake, Server finished (14):
== Info: TLSv1.2 (OUT), TLS header, Certificate Status (22):
== Info: TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
== Info: TLSv1.2 (OUT), TLS header, Finished (20):
== Info: TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
== Info: TLSv1.2 (OUT), TLS header, Certificate Status (22):
== Info: TLSv1.2 (OUT), TLS handshake, Finished (20):
== Info: TLSv1.2 (IN), TLS header, Finished (20):
== Info: TLSv1.2 (IN), TLS header, Certificate Status (22):
== Info: TLSv1.2 (IN), TLS handshake, Finished (20):
== Info: SSL connection using TLSv1.2 / AES256-GCM-SHA384
== Info: ALPN, server did not agree to a protocol
== Info: Server certificate:
== Info:  subject: CN=DataService
== Info:  start date: Feb  8 10:13:50 2024 GMT
== Info:  expire date: Feb  5 10:13:50 2034 GMT
== Info:  issuer: CN=DataService
== Info:  SSL certificate verify result: self-signed certificate (18), continuing anyway.
== Info: TLSv1.2 (OUT), TLS header, Supplemental data (23):
=> Send header: GET /fio/test_0_4096 HTTP/1.1
Host: 172.20.46.10
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20240228T101805Z
x-amz-storage-class: STANDARD
Authorization: AWS4-HMAC-SHA256 Credential=OKIACA62B0E3AE4A8557/20240228/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-storage-class,Signature=4f9b4eb6fc6d6f16fdfda9ec0045ac635a726152cf74af980ba0b399054bf39d

== Info: TLSv1.2 (IN), TLS header, Supplemental data (23):
== Info: Mark bundle as not supporting multiuse
<= Recv header: HTTP/1.1 403 Forbidden
<= Recv header: Date: Wed, 28 Feb 2024 10:18:05 GMT
<= Recv header: x-amz-id-2:
<= Recv header: Server: ViPR/1.0
<= Recv header: x-amz-request-id: 0a8100b7:18d88386fa9:2658:15d
<= Recv header: Content-Type: application/xml
<= Recv header: Content-Length: 2285
<= Recv header:
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your Secret Access Key and signing method. For more information, see REST Authentication and SOAP Authentication for details.</Message><RequestId>0a8100b7:18d88386fa9:2658:15d</RequestId><AWSAccessKeyId>okiaca62b0e3ae4a8557</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256
20240228T101805Z
20240228/us-east-1/s3/aws4_request
de43c9b5d311a9936309003d330ab82ea76ea4e6433faf1d2de4acb6a24db737</StringToSign><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 34 30 32 32 38 54 31 30 31 38 30 35 5a 0a 32 30 32 34 30 32 32 38 2f 75 73 2d 65 61 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 64 65 34 33 63 39 62 35 64 33 31 31 61 39 39 33 36 33 30 39 30 30 33 64 33 33 30 61 62 38 32 65 61 37 36 65 61 34 65 36 34 33 33 66 61 66 31 64 32 64 65 34 61 63 62 36 61 32 34 64 62 37 33 37</StringToSignBytes><CanonicalRequest>GET
/fio/test_0_4096

host:172.20.46.10
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20240228T101805Z
x-amz-storage-class:STANDARD

host;x-amz-content-sha256;x-amz-date;x-amz-storage-class
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</CanonicalRequest><CanonicalRequestBytes>47 45 54 0a 2f 66 69 6f 2f 74 65 73 74 5f 30 5f 34 30 39 36 0a 0a 68 6f 73 74 3a 31 37 32 2e 32 30 2e 34 36 2e 31 30 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 32 34 30 32 32 38 54 31 30 31 38 30 35 5a 0a 78 2d 61 6d 7a 2d 73 74 6f 72 61 67 65 2d 63 6c 61 73 73 3a 53 54 41 4e 44 41 52 44 0a 0a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 3b 78 2d 61 6d 7a 2d 73 74 6f 72 61 67 65 2d 63 6c 61 73 73 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35</CanonicalRequestBytes></Error><= Recv data: <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your Secret Access Key and signing method. For more information, see REST Authentication and SOAP Authentication for details.</Message><RequestId>0a8100b7:18d88386fa9:2658:15d</RequestId><AWSAccessKeyId>okiaca62b0e3ae4a8557</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256
20240228T101805Z
20240228/us-east-1/s3/aws4_request
de43c9b5d311a9936309003d330ab82ea76ea4e6433faf1d2de4acb6a24db737</StringToSign><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 34 30 32 32 38 54 31 30 31 38 30 35 5a 0a 32 30 32 34 30 32 32 38 2f 75 73 2d 65 61 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 64 65 34 33 63 39 62 35 64 33 31 31 61 39 39 33 36 33 30 39 30 30 33 64 33 33 30 61 62 38 32 65 61 37 36 65 61 34 65 36 34 33 33 66 61 66 31 64 32 64 65 34 61 63 62 36 61 32 34 64 62 37 33 37</StringToSignBytes><CanonicalRequest>GET
/fio/test_0_4096

host:172.20.46.10
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20240228T101805Z
x-amz-storage-class:STANDARD

host;x-amz-content-sha256;x-amz-date;x-amz-storage-class
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</CanonicalRequest><CanonicalRequestBytes>47 45 54 0a 2f 66 69 6f 2f 74 65 73 74 5f 30 5f 34 30 39 36 0a 0a 68 6f 73 74 3a 31 37 32 2e 32 30 2e 34 36 2e 31 30 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 32 34 30 32 32 38 54 31== Info: Connection #0 to host 172.20.46.10 left intact
DDIR_READ failed with HTTP status code 403
 30 31 38 30 35 5a 0a 78 2d 61 6d 7a 2d 73 74 6f 72 61 67 65 2d 63 6c 61 73 73 3a 53 54 41 4e 44 41 52 44 0a 0a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 3b 78 2d 61 6d 7a 2d 73 74 6f 72 61 67 65 2d 63 6c 61 73 73 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35</CanonicalRequestBytes></Error>fio: first I/O failed. If /fio/test is a zoned block device, consider --zonemode=zbd
fio: pid=248, err=5/file:engines/http.c:715, func=transfer, error=Input/output error

create: (groupid=0, jobs=1): err= 5 (file:engines/http.c:715, func=transfer, error=Input/output error): pid=248: Wed Feb 28 10:18:05 2024
  cpu          : usr=15.38%, sys=0.00%, ctx=4, majf=0, minf=247
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):

aws cli output

~/fio-objectscale$ aws s3api --no-verify-ssl  --profile default list-buckets --endpoint-url https://172.20.46.10:443

urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.20.46.10'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
{
    "Buckets": [
        {
            "Name": "fio",
            "CreationDate": "2024-02-27T17:40:30.774000+00:00"
        },
        {
            "Name": "test",
            "CreationDate": "2024-02-20T11:15:04.006000+00:00"
        }
    ],
    "Owner": {
        "DisplayName": "urn:osc:iam::osai23255076cce06025:root",
        "ID": "urn:osc:iam::osai23255076cce06025:root"
    }
}
nfonseca commented 4 months ago

Forgot to add my fio config file

[global]
ioengine=http
name=test
direct=1
filename=/fio/test
http_verbose=2
https=insecure
http_mode=s3
http_s3_key=Wu0gMuv3nKn0AVSp2AaqGkLg5zeKMENCNu6kJXM0
http_s3_keyid=OKIACA62B0E3AE4A8557
http_host=172.20.46.10:443
http_s3_region=us-east-1
group_reporting

# With verify, this both writes and reads the object
[create]
rw=read
bs=4k
size=64k
io_size=4k
verify=sha256
vincentkfu commented 4 months ago

Should the keys you shared be secret?

nfonseca commented 4 months ago

It's a Lab on air gapped network. I am no fool but thanks for the heads up !

hjk068 commented 3 months ago

I could resolve this issue by manually adding the aws session token as 'x-amz-security-token' in the request header for aws authorization (_add_aws_auth_header function in http.c).

https://stackoverflow.com/a/61248929