search

vitalif / vitastor

Simplified distributed block and file storage with strong consistency, like in Ceph (repository mirror)
https://vitastor.io
Other
136 stars 22 forks source link

[vitastor] When vdbench rewrites 8000000 small files, it reports an error #47

Closed lnsyyj closed 2 years ago

lnsyyj commented 2 years ago

Hi @vitalif ,

When I format the xfs file system with vitastor nbd and use the test tool vdbench to read and write, the following problems occur occasionally. The performance of vitastor is very good now, and I think it will be perfect if I increase the stability.

vdbench log

Mar 15, 2022 ..Interval.. .ReqstdOps... ...cpu%...  read ....read..... ....write.... ..mb/sec... mb/sec .xfer.. ...mkdir.... ...rmdir.... ...create... ....open.... ...close.... ...delete... ..getattr... ..setattr... ...a[502/1854]
                            rate   resp total  sys   pct   rate   resp   rate   resp  read write  total    size  rate   resp  rate   resp  rate   resp  rate   resp  rate   resp  rate   resp  rate   resp  rate   resp  rate   resp  
14:57:08.012            1    0.0  0.000  10.9 5.12   0.0    0.0  0.000    0.0  0.000  0.00  0.00   0.00       0   0.0  0.000   0.0  0.000   0.0  0.000   0.0  0.000   0.0  0.000   0.0  0.000   0.0  0.000   0.0  0.000   0.0  0.000  
14:57:08.021      avg_2-1 NaN  0.000 NaN NaN   0.0 NaN  0.000 NaN  0.000 NaN NaN NaN       0 NaN  0.000 NaN  0.000 NaN  0.000 NaN  0.000 NaN  0.000 NaN  0.000 NaN  0.000 NaN  0.000 NaN  0.000                                       
14:57:08.021      std_2-1                                                                                                                                                                                                             
14:57:08.021      max_2-1                                                                                                                                                                                                             
14:57:08.130                                                                                                                                                                                                                          
14:57:08.130 Miscellaneous statistics:                                                                                                                                                                                                
14:57:08.130 (These statistics do not include activity between the last reported interval and shutdown.)                                                                                                                              
14:57:08.130 DIR_EXISTS          Directory may not exist (yet):                   14          4/sec                                                                                                                                   
14:57:08.130 FILE_MAY_NOT_EXIST  File may not exist (yet):                        16          5/sec                                                                                                                                   
14:57:08.130                                                                                                                                                                                                                          
14:57:38.173 Waiting for slave synchronization: hd1-0. Building and validating file structure(s).                                                                                                                                     
14:58:08.063 hd1-0: Completing the creation of internal file structure for anchor=/root/performance/fg-mp1/verilog_1: 4,096,000 files.                                                                                                
14:58:08.202 Waiting for slave synchronization: hd1-0. Building and validating file structure(s).                                                                                                                                     
14:58:38.232 Waiting for slave synchronization: hd1-0. Building and validating file structure(s).                                                                                                                                     
14:59:08.262 Waiting for slave synchronization: hd1-0. Building and validating file structure(s).                                                                                                                                     
14:59:08.411 hd1-0: Completing the creation of internal file structure for anchor=/root/performance/fg-mp2/verilog_2: 4,096,000 files.                                                                                                
14:59:09.000 Starting RD=rd2; elapsed=300; fwdrate=max. For loops: xfersize=64k threads=128                                                                                                                                           
14:59:09.461 hd1-0: 14:59:09.460 op: write  lun: /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f0064.file lba:            0 0x00000000 xfer:     1024 err
no: EINVAL: 'Invalid argument'           
14:59:09.475 hd1-0: 14:59:09.470 Write error using file /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f0064.file                                         
Error:         EINVAL: 'Invalid argument'                                                                          
lba:           0                                                                                                                                                                                                                      
xfersize:      1024                                                                                                
blocks_done:   0                                                                                                                                                                                                                      
bytes_done:    0                                                                                                   
open_for_read: false                                                                                                                                                                                                                  
fhandle:       17                                                                                                  
14:59:09.506 hd1-0: 14:59:09.472 op: read   lun: /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f0008.file lba:            0 0x00000000 xfer:     2048 err
no: EINVAL: 'Invalid argument'                                                                                     
14:59:09.508 hd1-0: 14:59:09.507 Read error using file /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f0008.file                                          
Error:         EINVAL: 'Invalid argument'
lba:           0                                                                                                                                                                                                                      
xfersize:      2048                                                                                                
blocks_done:   0                                                                                                                                                                                                                      
bytes_done:    0
open_for_read: true
fhandle:       13
14:59:09.633 hd1-0: 14:59:09.633 op: read   lun: /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_2.dir/vdb.7_1.dir/vdb_f0474.file lba:            0 0x00000000 xfer:     1024 err
no: EINVAL: 'Invalid argument' 
14:59:09.635 hd1-0: 14:59:09.634 op: read   lun: /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_2.dir/vdb.7_1.dir/vdb_f0473.file lba:            0 0x00000000 xfer:     1024 err
no: EINVAL: 'Invalid argument' 
14:59:09.635 hd1-0: 14:59:09.635 Read error using file /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_2.dir/vdb.7_1.dir/vdb_f0474.file
Error:         EINVAL: 'Invalid argument'
lba:           0
xfersize:      1024
blocks_done:   0
bytes_done:    0
open_for_read: true
fhandle:       97
14:59:09.636 hd1-0: 14:59:09.636 op: read   lun: /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_2.dir/vdb.7_1.dir/vdb_f0472.file lba:            0 0x00000000 xfer:     1024 err
no: EINVAL: 'Invalid argument'

file size

root@vita-1:~/output# ls -l /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f0064.file
-rw-r--r-- 1 root root 1024 Mar 15 14:36 /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f0064.file
root@vita-1:~/output# ls -l /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f0008.file
-rw-r--r-- 1 root root 2048 Mar 15 14:49 /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_1.dir/vdb.7_1.dir/vdb_f0008.file
root@vita-1:~/output# ls -l  /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_2.dir/vdb.7_1.dir/vdb_f0474.file
-rw-r--r-- 1 root root 1024 Mar 15 15:18 /root/performance/fg-mp1/verilog_1/vdb.1_1.dir/vdb.2_1.dir/vdb.3_1.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_2.dir/vdb.7_1.dir/vdb_f0474.file
vitalif commented 2 years ago

I only see these here:

Error:         EINVAL: 'Invalid argument'                                                                          
lba:           0                                                                                                                                                                                                                      
xfersize:      1024

These are correct, minimal writable unit / sector size in Vitastor is 4096 bytes.

lnsyyj commented 2 years ago

Oh, does this mean that overwriting a 1KB file will have an error?

vdbench added openflag=o_direct parameter, it is no problem not to use this parameter

root@vita-1:~# xfs_info /root/performance/
meta-data=/dev/nbd0              isize=512    agcount=11, agsize=268435455 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=0
data     =                       bsize=4096   blocks=2952790005, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
vitalif commented 2 years ago

Oh, does this mean that overwriting a 1KB file will have an error?

As I understand all file systems in Linux use 4 KB block size. So when you overwrite 1 KB you actually overwrite 4 KB. :-) But if a tool tries to do O_DIRECT reads or writes smaller than 4 KB - it gets an error, yes. It's basically the same as HDD/SSD switched to 4 KB sectors.

lnsyyj commented 2 years ago

Oh oh, got it. I guess this can be summed up as users use QA.