In zfs 0.7 and 0.8, the same data directory has the diff performance with the same application.
If you want to get the best read performance, I have to copy the dir with cp command(single process).
eg: dir A and dir B has the same data. B was copied from A, B has the best read performance, A was generate by the application. you could run the application with dir A, the read throughput only 500MB/s, if you switch the application to access dir B. the read throughput could be reached 1GB/s.
The test did not pass the network, just read and write local zpool.
I check a lot of the iostat 2 and zpool iostat -lv 2, all loading is balance in each disk.
Describe how to reproduce the problem
Here is the simulate bash script. it could show the issue.
Create the zpool script, I switch the hardware platform from Dell R740 to MD3060e, the issue is the same.
for i in {b..k}; do parted -s /dev/sd$i mklabel gpt; parted -s /dev/sd$i mkpart p1 2048s 200G; done
zpool create tank raidz2 /dev/sd{b..k}1
[0:0:0:0] enclosu DELL MD3060e 039F -
[0:0:1:0] disk SEAGATE ST6000NM0034 MS2A /dev/sdb
[0:0:2:0] disk SEAGATE ST6000NM0034 MS2A /dev/sdc
[0:0:3:0] disk SEAGATE ST6000NM0034 MS2A /dev/sdd
[0:0:4:0] disk SEAGATE ST6000NM0034 MS2A /dev/sde
[0:0:5:0] disk SEAGATE ST6000NM0034 MS2A /dev/sdf
[0:0:6:0] disk SEAGATE ST6000NM0034 MS2A /dev/sdg
[0:0:7:0] disk SEAGATE ST6000NM0095 DS23 /dev/sdh
[0:0:8:0] disk SEAGATE ST6000NM0034 MS2A /dev/sdi
[0:0:9:0] disk SEAGATE ST6000NM0034 MS2A /dev/sdj
[0:0:10:0] disk SEAGATE ST6000NM0034 MS2A /dev/sdk
Generate data to the zpool
cat gen.sh
start_time=$(date +%s)
test_path=/tank/test4/gen
[[ ! -f /dev/shm/1G.file ]] && openssl rand -out /dev/shm/1G.file $(( 1024*1024*1024 ))
[[ ! -d $test_path ]] && mkdir -p $test_path
for j in {0..4}
do
for i in {0..19}
do
cat /dev/shm/1G.file >> ${test_path}/rw.${j} &
done
done
wait
end_time=$(date +%s)
echo "gen files total time(secs):"$((end_time-start_time))
read generate data from this zpool
cat read-gen.sh
echo 3 > /proc/sys/vm/drop_caches
start_time=$(date +%s)
test_path=/tank/test4/gen
[[ ! -d $test_path ]] && mkdir -p $test_path
for j in {0..4}
do
for i in {0..1}
do
dd if=${test_path}/rw.${j} of=/dev/null bs=1M &
done
done
wait
end_time=$(date +%s)
echo "read gen dir total time(secs):"$((end_time-start_time))
copy the generate data
cp -a /tank/test4/gen /tank/test4/gen-copy
Test read performance from the copied dir
cat read-cp.sh
echo 3 > /proc/sys/vm/drop_caches
start_time=$(date +%s)
test_path=/tank/test4/gen-copy
[[ ! -d $test_path ]] && mkdir -p $test_path
for j in {0..4}
do
for i in {0..1}
do
dd if=${test_path}/rw.${j} of=/dev/null bs=1M &
done
done
wait
end_time=$(date +%s)
echo "read copy dir total time(secs):"$((end_time-start_time))
The run process
$ cd /tank/test4
$ rm -rf gen gen-copy; sh ./gen.sh ; cp -a gen gen-copy ; sh ./read-gen.sh ; sh ./read-cp.sh
# switch the read order
$ rm -rf gen gen-copy; sh ./gen.sh ; cp -a gen gen-copy ; sh ./read-cp.sh; sh ./read-gen.sh
Include any warning/errors/backtraces from the system logs
The simulate test result(The lower the better):
test env (CentOS7)
read from generating dir(secs)
read from cp dir(secs)
MD3060E+ 0.8.4(new)
84~99
60~61
MD3060E+ 0.8.4(FRAG 13%)
113~119
67~68
MD3060E+ 0.7.13(new)
94~100
64~67
MD3060E+ 0.7.13(FRAG 51%)
133~141
75~77
MD3060E+ 0.6.5.11(new)
99~100
101~103
Here is the original in my production env, I can't re-create the zpool, I test 20 times. 10x dirA, 10x dirB, fio 3.7, I use script parallel to generate the fio test file.
This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.
System information
Describe the problem you're observing
In zfs 0.7 and 0.8, the same data directory has the diff performance with the same application.
If you want to get the best read performance, I have to copy the dir with cp command(single process).
eg: dir A and dir B has the same data. B was copied from A, B has the best read performance, A was generate by the application. you could run the application with dir A, the read throughput only 500MB/s, if you switch the application to access dir B. the read throughput could be reached 1GB/s.
The test did not pass the network, just read and write local zpool.
I check a lot of the iostat 2 and zpool iostat -lv 2, all loading is balance in each disk.
Describe how to reproduce the problem
Here is the simulate bash script. it could show the issue.
Create the zpool script, I switch the hardware platform from Dell R740 to MD3060e, the issue is the same.
Generate data to the zpool
read generate data from this zpool
copy the generate data
Test read performance from the copied dir
The run process
Include any warning/errors/backtraces from the system logs
The simulate test result(The lower the better):
Here is the original in my production env, I can't re-create the zpool, I test 20 times. 10x dirA, 10x dirB, fio 3.7, I use script parallel to generate the fio test file.
Thanks : )
Here is the zdb -dddddd show the same file in diff dirs. after cp parallel write
Test parameters