Does fio-3.26 support minio test? Hit "DDIR_WRITE failed with HTTP status code 404" during CREATE test

[garyguo]

Please acknowledge you have done the following before creating a ticket

• [ ] I have read the GitHub issues section of REPORTING-BUGS.

Description of the bug:

Not sure if fio-3.26 supports minio test? Hit "DDIR_WRITE failed with HTTP status code 404" during CREATE test with fio-3.26 building on CentOS 8.0 from src The details can be found in the following.

Environment: CentOS, fio version, minio version

[root@bluesky-centroot80 fio]# cat /etc/redhat-release 
CentOS Linux release 8.0.1905 (Core) 
[root@bluesky-centroot80 fio]# fio --version
fio-3.26
[root@bluesky-centroot80 fio]# /opt/minio --version
minio version RELEASE.2021-04-22T15-44-28Z
[root@bluesky-centroot80 fio]# 

++++++ Simply starting minio after downloading binary files on CentOS 8.0 ++++++

[root@bluesky-centroot80 opt]# wget https://dl.minio.io/server/minio/release/linux-amd64/minio
[root@bluesky-centroot80 opt]# wget https://dl.min.io/client/mc/release/linux-amd64/mc
[root@bluesky-centroot80 opt]# 
[root@bluesky-centroot80 opt]# chmod +x minio mc
[root@bluesky-centroot80 opt]# ls -l
total 69984
-rwxr-xr-x 1 root root 20529152 Apr 22 11:42 mc
-rwxr-xr-x 1 root root 51134464 Apr 22 09:54 minio
[root@bluesky-centroot80 opt]# 

[root@bluesky-centroot80 ~]# /opt/minio server /data &
[1] 26580
[root@bluesky-centroot80 ~]# Endpoint: http://127.0.0.1:9000         
RootUser: minioadmin 
RootPass: minioadmin 

Browser Access:
   http://127.0.0.1:9000        

Command-line Access: https://docs.min.io/docs/minio-client-quickstart-guide
   $ mc alias set myminio http://127.0.0.1:9000 minioadmin minioadmin

Object API (Amazon S3 compatible):
   Go:         https://docs.min.io/docs/golang-client-quickstart-guide
   Java:       https://docs.min.io/docs/java-client-quickstart-guide
   Python:     https://docs.min.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.min.io/docs/dotnet-client-quickstart-guide
Detected default credentials 'minioadmin:minioadmin', please change the credentials immediately using 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD'
IAM initialization complete
++++++ Verify minio server works fine via mc (minio client) on the same node ++++++

[root@bluesky-centroot80 fio]# /opt/mc alias list myminio
myminio
  URL       : http://127.0.0.1:9000
  AccessKey : minioadmin
  SecretKey : minioadmin
  API       : s3v4
  Path      : auto
[root@bluesky-centroot80 fio]# /opt/mc ls myminio/
[2021-05-10 21:59:39 MDT]     0B histest/
[2021-05-10 21:05:24 MDT]     0B mytest/
[2021-05-10 21:16:15 MDT]     0B test/
[root@bluesky-centroot80 fio]# 
[root@bluesky-centroot80 fio]# /opt/mc ls myminio/test
[root@bluesky-centroot80 fio]# 
[root@bluesky-centroot80 fio]# /opt/mc cp myobject myminio/test/
myobject:                            18 B / 18 B ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 2.79 KiB/s 0s[root@bluesky-centroot80 fio]# 
[root@bluesky-centroot80 fio]# /opt/mc ls myminio/test
[2021-05-11 12:20:42 MDT]    18B myobject
[root@bluesky-centroot80 fio]# 

++++++ Start fio-3.26 via s3_object.config and hit the issue ++++++

[root@bluesky-centroot80 fio]# cat s3_object.config
# Example test for the HTTP engine's S3 support against Amazon AWS.
# Obviously, you have to adjust the S3 credentials; for this example,
# they're passed in via the environment.
#

[global]
ioengine=http
name=test
direct=1
#filename=/opt/fio/hpe.pnp
filename=/opt/fio/myobject
http_verbose=0
https=off
http_mode=s3
#http_s3_key=${S3_KEY}
#http_s3_keyid=${S3_ID}
http_s3_keyid=minioadmin
http_s3_key=minioadmin
http_host=http://127.0.0.1:9000
#http_s3_region=eu-central-1
#http_s3_region=""
group_reporting
loops=2

# With verify, this both writes and reads the object
[create]
rw=write
bs=4k
size=64k
io_size=4k
verify=sha256

[trim]
stonewall
#trim is translated object deletion
rw=trim
bs=4k
size=64k
io_size=4k

[root@bluesky-centroot80 fio]# fio s3_object.config
create: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=http, iodepth=1
trim: (g=1): rw=trim, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=http, iodepth=1
fio-3.26
Starting 2 processes
DDIR_WRITE failed with HTTP status code 404                                             <<<<<<<< the issue
fio: pid=31747, err=-1/file:engines/http.c:570, func=transfer, error=Unknown error -1       <<<<<<<< the issue

create: (groupid=0, jobs=1): err=-1 (file:engines/http.c:570, func=transfer, error=Unknown error -1): pid=31747: Tue May 11 12:22:05 2021
  cpu          : usr=0.00%, sys=2.42%, ctx=8, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
trim: (groupid=1, jobs=1): err= 0: pid=31749: Tue May 11 12:22:05 2021
  trim: IOPS=12, BW=49.7KiB/s (50.9kB/s)(8192B/161msec)
    clat (usec): min=76808, max=83457, avg=80133.03, stdev=4701.81
     lat (usec): min=76808, max=83458, avg=80133.41, stdev=4701.71
    clat percentiles (usec):
     |  1.00th=[77071],  5.00th=[77071], 10.00th=[77071], 20.00th=[77071],
     | 30.00th=[77071], 40.00th=[77071], 50.00th=[77071], 60.00th=[83362],
     | 70.00th=[83362], 80.00th=[83362], 90.00th=[83362], 95.00th=[83362],
     | 99.00th=[83362], 99.50th=[83362], 99.90th=[83362], 99.95th=[83362],
     | 99.99th=[83362]
  lat (msec)   : 100=100.00%
  cpu          : usr=1.25%, sys=0.00%, ctx=5, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,0,2,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):

Run status group 1 (all jobs):
   TRIM: bw=49.7KiB/s (50.9kB/s), 49.7KiB/s-49.7KiB/s (50.9kB/s-50.9kB/s), io=8192B (8192B), run=161-161msec

[sitsofe]

@garyguo We're not a minio users but are you happy to debug this one. Does moving stonewall to the global section make a difference?

[sitsofe]

Mentioning @l-mb for comment.

[rlemmers]

Try this @garyguo: http_s3_region=us-east-1

I have been fiddling with the same. MinIO does have a region, so http_s3_region should have a value (it's empty in @garyguo's configuration). The MinIO's default region is us-east-1. You can configure a different value (see https://docs.gitlab.com/charts/charts/minio/). In my case, using us-east-1 as region resolved the 404.

I'm currently stuck with a "The request signature we calculated does not match the signature you provided. Check your key and signing method.", but that's out of scope for this ticket.

[enrico2828]

I encountered this issue when trying to configure fio for Minio S3. I found a working configuration. I think the problem with the original example in this thread could be that the bucket is not correctly specified. It seems you need to put it in the filename so that it ends up correctly in the http request.

      [global]
      ioengine=http
      name=test
      direct=1
      filename=/<bucket>/testobject
      http_verbose=0
      https=insecure
      http_mode=s3
      http_s3_key=minio123
      http_s3_keyid=minio
      http_host=na-minio-tenant-1-hl.na-minio.svc.cluster.local:9000
      http_s3_region=us-east-1
      group_reporting

      # With verify, this both writes and reads the object
      [create]
      rw=write
      bs=32M
      size=50G
      verify=sha256

      [trim]
      stonewall
      rw=trim
      bs=32M
      size=50G