search

gluster / gluster-block

A framework for gluster block storage
GNU General Public License v2.0
74 stars 32 forks source link

tcmu-runner: add write same support which will improve the perf of formating the device. #160

Closed lxbsz closed 5 years ago

lxbsz commented 5 years ago

https://github.com/open-iscsi/tcmu-runner/pull/494

pkalever commented 5 years ago

Yes this is helpful :-) Thanks @lxbsz.

lxbsz commented 5 years ago 
]# sg_write_same 
missing device name!
Usage: sg_write_same [--10] [--16] [--32] [--anchor] [--grpnum=GN] [--help]
                     [--in=IF] [--lba=LBA] [--lbdata] [--ndob] [--num=NUM]
                     [--pbdata] [--timeout=TO] [--unmap] [--verbose]
                     [--version] [--wrprotect=WRP] [xferlen=LEN]
                     DEVICE
  where:
    --10|-R              do WRITE SAME(10) (even if '--unmap' is given)
    --16|-S              do WRITE SAME(16) (def: 10 unless '--unmap' given
                         or LBA+NUM needs more than 32 bits)
    --32|-T              do WRITE SAME(32) (def: 10 or 16)
    --anchor|-a          set anchor field in cdb
    --grpnum=GN|-g GN    GN is group number field (def: 0)
    --help|-h            print out usage message
    --in=IF|-i IF        IF is file to fetch one block of data from (use LEN
                         bytes or whole file). Block written to DEVICE
    --lba=LBA|-l LBA     LBA is the logical block address to start (def: 0)
    --lbdata|-L          set LBDATA bit
    --ndob|-N            set 'no data-out buffer' bit
    --num=NUM|-n NUM     NUM is number of logical blocks to write (def: 1)
                         [Beware NUM==0 means rest of device]
    --pbdata|-P          set PBDATA bit
    --timeout=TO|-t TO    command timeout (unit: seconds) (def: 60)
    --unmap|-U           set UNMAP bit
    --verbose|-v         increase verbosity
    --version|-V         print version string then exit
    --wrprotect=WPR|-w WPR    WPR is the WRPROTECT field value (def: 0)
    --xferlen=LEN|-x LEN    LEN is number of bytes from IF to send to
                            DEVICE (def: IF file length)

Performs a SCSI WRITE SAME (10, 16 or 32) command

First, enable the DEBUG_SCSI_CMD in /etc/tcmu/tcmu.conf, and then execut:

]# tailf /var/log/tcmu-runner.log

Test Case1: with UNMAP bit being set and use WRITE SAME(10)

]# sg_write_same -R -U --lba=13 --num=100999 /dev/sdg --verbose
Default data-out buffer set to 512 zeros
do_write_same: use WRITE SAME(16) instead of 10 byte cdb
]#

We will see the tcmu-runner debug logs as:

[...]
2018-11-13 13:21:21.814 2424 [DEBUG_SCSI_CMD] tcmu_print_cdb_info:1195 glfs/block0: 9e 10 0 0 0 0 0 0 0 0 0 0 0 20 0 0 
2018-11-13 13:21:21.817 2424 [DEBUG_SCSI_CMD] tcmu_print_cdb_info:1195 glfs/block0: 93 8 0 0 0 0 0 0 0 d 0 1 8a 87 0 0 
2018-11-13 13:21:21.817 2424 [DEBUG] handle_writesame_check:673 glfs/block0: Start lba: 13, number of lba: 100999, last lba: 101011
2018-11-13 13:21:21.817 2424 [DEBUG] handle_unmap_in_writesame:688 glfs/block0: Do UNMAP in WRITE_SAME cmd!
2018-11-13 13:21:21.817 2424 [DEBUG] align_and_split_unmap:316 glfs/block0: OPTIMAL UNMAP GRANULARITY: 128, UNMAP GRANULARITY ALIGNMENT mask: 18446744073709551615, lbas 115
2018-11-13 13:21:21.817 2424 [DEBUG] align_and_split_unmap:340 glfs/block0: The first split: start lba: 13, end lba: 127, lbas: 115
2018-11-13 13:21:21.866 2424 [DEBUG] align_and_split_unmap:345 glfs/block0: The last split: start lba: 100992, end lba: 101011, lbas: 20
2018-11-13 13:21:21.866 2424 [DEBUG] align_and_split_unmap:346 glfs/block0: There are totally 790 splits
[...]

Test Case2: with UNMAP bit being set and use WRITE SAME(16)

]# sg_write_same -S -U --lba=13 --num=100999 /dev/sdg --verbose
Default data-out buffer set to 512 zeros
]#

The tcmu-runner debug logs as:

2018-11-13 13:26:04.702 2424 [DEBUG] handle_writesame_check:673 glfs/block0: Start lba: 13, number of lba: 100999, last lba: 101011
2018-11-13 13:26:04.702 2424 [DEBUG] handle_unmap_in_writesame:688 glfs/block0: Do UNMAP in WRITE_SAME cmd!
2018-11-13 13:26:04.702 2424 [DEBUG] align_and_split_unmap:316 glfs/block0: OPTIMAL UNMAP GRANULARITY: 128, UNMAP GRANULARITY ALIGNMENT mask: 18446744073709551615, lbas 115
2018-11-13 13:26:04.702 2424 [DEBUG] align_and_split_unmap:340 glfs/block0: The first split: start lba: 13, end lba: 127, lbas: 115
2018-11-13 13:26:04.725 2424 [DEBUG] align_and_split_unmap:345 glfs/block0: The last split: start lba: 100992, end lba: 101011, lbas: 20
2018-11-13 13:26:04.725 2424 [DEBUG] align_and_split_unmap:346 glfs/block0: There are totally 790 splits

Test Case3: without UNMAP bit being set and use WRITE SAME(10)

]# sg_write_same -R --lba=13 --num=100999 /dev/sdg --verbose
Default data-out buffer set to 512 zeros
do_write_same: use WRITE SAME(16) instead of 10 byte cdb
]#

Tcmu-runner logs:

2018-11-13 13:27:39.258 2424 [DEBUG_SCSI_CMD] tcmu_print_cdb_info:1195 glfs/block0: 9e 10 0 0 0 0 0 0 0 0 0 0 0 20 0 0 
2018-11-13 13:27:39.262 2424 [DEBUG_SCSI_CMD] tcmu_print_cdb_info:1195 glfs/block0: 93 0 0 0 0 0 0 0 0 d 0 1 8a 87 0 0 
2018-11-13 13:27:39.262 2424 [DEBUG] handle_writesame_check:673 glfs/block0: Start lba: 13, number of lba: 100999, last lba: 101011

Test Case4: without UNMAP bit being set and use WRITE SAME(16)

]# sg_write_same -S --lba=13 --num=100999 /dev/sdg --verbose
Default data-out buffer set to 512 zeros
]# 
2018-11-13 13:29:15.599 2424 [DEBUG_SCSI_CMD] tcmu_print_cdb_info:1195 glfs/block0: 9e 10 0 0 0 0 0 0 0 0 0 0 0 20 0 0 
2018-11-13 13:29:15.600 2424 [DEBUG_SCSI_CMD] tcmu_print_cdb_info:1195 glfs/block0: 93 0 0 0 0 0 0 0 0 d 0 1 8a 87 0 0 
2018-11-13 13:29:15.600 2424 [DEBUG] handle_writesame_check:673 glfs/block0: Start lba: 13, number of lba: 100999, last lba: 101011
2018-11-13 13:29:15.615 2424 [DEBUG_SCSI_CMD] tcmu_print_cdb_info:1195 glfs/block0: 0 0 0 0 0 0

Test Case5: without UNMAP bit being set and use WRITE SAME(16), set the num larger large enough which will exceed the actual target block size:

]# sg_write_same -S --lba=13 --num=100999999 /dev/sdg --verbose
Default data-out buffer set to 512 zeros
Write same:  Fixed format, current;  Sense key: Illegal Request
 Additional sense: Logical block address out of range
bad field in Write same(16) cdb, option probably not supported
]#

There will be errors in tcmu-runner too:

2018-11-13 13:30:31.502 2424 [DEBUG_SCSI_CMD] tcmu_print_cdb_info:1195 glfs/block0: 9e 10 0 0 0 0 0 0 0 0 0 0 0 20 0 0 
2018-11-13 13:30:31.504 2424 [DEBUG_SCSI_CMD] tcmu_print_cdb_info:1195 glfs/block0: 93 0 0 0 0 0 0 0 0 d 6 5 23 3f 0 0 
2018-11-13 13:30:31.504 2424 [ERROR] check_lbas:119 glfs/block0: cmd exceeds last lba 2097152 (lba 13, xfer len 100999999)
lxbsz commented 5 years ago

And also when the target device is formating will also trigger this:

]# mkfs.xfs /dev/sdg
meta-data=/dev/sdg               isize=512    agcount=4, agsize=65536 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
]#

Tcmu-runner logs:


2018-11-13 13:42:41.945 2424 [DEBUG_SCSI_CMD] tcmu_print_cdb_info:1195 glfs/block0: 28 0 0 0 10 0 0 0 8 0 
2018-11-13 13:42:41.950 2424 [DEBUG_SCSI_CMD] tcmu_print_cdb_info:1195 glfs/block0: 93 8 0 0 0 0 0 0 0 0 0 20 0 0 0 0 
2018-11-13 13:42:41.950 2424 [DEBUG] handle_writesame_check:673 glfs/block0: Start lba: 0, number of lba: 2097152, last lba: 2097151
2018-11-13 13:42:41.950 2424 [DEBUG] handle_unmap_in_writesame:688 glfs/block0: Do UNMAP in WRITE_SAME cmd!
2018-11-13 13:42:41.950 2424 [DEBUG] align_and_split_unmap:316 glfs/block0: OPTIMAL UNMAP GRANULARITY: 128, UNMAP GRANULARITY ALIGNMENT mask: 18446744073709551615, lbas 128
2018-11-13 13:42:41.950 2424 [DEBUG] align_and_split_unmap:340 glfs/block0: The first split: start lba: 0, end lba: 127, lbas: 128
2018-11-13 13:42:49.277 2424 [DEBUG] align_and_split_unmap:345 glfs/block0: The last split: start lba: 2097024, end lba: 2097151, lbas: 128
2018-11-13 13:42:49.277 2424 [DEBUG] align_and_split_unmap:346 glfs/block0: There are totally 16384 splits
2018-11-13 13:42:49.277 2424 [DEBUG_SCSI_CMD] tcmu_print_cdb_info:1195 glfs/block0: 0 0 0 0 0 0
pkalever commented 5 years ago

WRITE SAME is already added to tcmu-runner.

Thanks @lxbsz for your work.