openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.6k stars 1.75k forks source link

Different writing impact the read performance in zfs 0.7 and 0.8, 0.6.5 is OK #10677

Closed homerl closed 2 years ago

homerl commented 4 years ago

System information

Type Version/Name
Distribution Name CenOS
Distribution Version 7.7/8.2
Linux Kernel 3.10.0-1062.18.1.el7.x86_64/4.18.0-193.6.3.el8_2.x86_64
Architecture x86_64
ZFS Version 0.8.3/0.8.4/0.7.9/0.7.13
SPL Version 0.8.3/0.8.4/0.7.9/0.7.13

Describe the problem you're observing

In zfs 0.7 and 0.8, the same data directory has the diff performance with the same application.
If you want to get the best read performance, I have to copy the dir with cp command(single process).
eg: dir A and dir B has the same data. B was copied from A, B has the best read performance, A was generate by the application. you could run the application with dir A, the read throughput only 500MB/s, if you switch the application to access dir B. the read throughput could be reached 1GB/s.
The test did not pass the network, just read and write local zpool.
I check a lot of the iostat 2 and zpool iostat -lv 2, all loading is balance in each disk.

Describe how to reproduce the problem

Here is the simulate bash script. it could show the issue.

Create the zpool script, I switch the hardware platform from Dell R740 to MD3060e, the issue is the same.

for i in {b..k}; do parted -s /dev/sd$i mklabel gpt; parted -s /dev/sd$i mkpart p1 2048s 200G; done  
zpool create tank raidz2 /dev/sd{b..k}1

[0:0:0:0]    enclosu DELL     MD3060e          039F  -
[0:0:1:0]    disk    SEAGATE  ST6000NM0034     MS2A  /dev/sdb
[0:0:2:0]    disk    SEAGATE  ST6000NM0034     MS2A  /dev/sdc
[0:0:3:0]    disk    SEAGATE  ST6000NM0034     MS2A  /dev/sdd
[0:0:4:0]    disk    SEAGATE  ST6000NM0034     MS2A  /dev/sde
[0:0:5:0]    disk    SEAGATE  ST6000NM0034     MS2A  /dev/sdf
[0:0:6:0]    disk    SEAGATE  ST6000NM0034     MS2A  /dev/sdg
[0:0:7:0]    disk    SEAGATE  ST6000NM0095     DS23  /dev/sdh
[0:0:8:0]    disk    SEAGATE  ST6000NM0034     MS2A  /dev/sdi
[0:0:9:0]    disk    SEAGATE  ST6000NM0034     MS2A  /dev/sdj
[0:0:10:0]   disk    SEAGATE  ST6000NM0034     MS2A  /dev/sdk

Generate data to the zpool

cat gen.sh

start_time=$(date +%s)
test_path=/tank/test4/gen

[[ ! -f /dev/shm/1G.file ]] && openssl rand -out /dev/shm/1G.file $(( 1024*1024*1024 ))
[[ ! -d $test_path ]] && mkdir -p $test_path
for j in {0..4}
do
  for i in {0..19}
  do
    cat /dev/shm/1G.file >> ${test_path}/rw.${j} &
  done
done
wait
end_time=$(date +%s)
echo "gen files total time(secs):"$((end_time-start_time))

read generate data from this zpool

cat read-gen.sh

echo 3 > /proc/sys/vm/drop_caches
start_time=$(date +%s)
test_path=/tank/test4/gen
[[ ! -d $test_path ]] && mkdir -p $test_path
for j in {0..4}
do
  for i in {0..1}
  do
    dd if=${test_path}/rw.${j} of=/dev/null bs=1M &
  done
done
wait
end_time=$(date +%s)
echo "read gen dir total time(secs):"$((end_time-start_time))

copy the generate data

cp -a /tank/test4/gen /tank/test4/gen-copy

Test read performance from the copied dir

cat read-cp.sh

echo 3 > /proc/sys/vm/drop_caches
start_time=$(date +%s)
test_path=/tank/test4/gen-copy
[[ ! -d $test_path ]] && mkdir -p $test_path
for j in {0..4}
do
  for i in {0..1}
  do
    dd if=${test_path}/rw.${j} of=/dev/null bs=1M &
  done
done
wait
end_time=$(date +%s)
echo "read copy dir total time(secs):"$((end_time-start_time))

The run process

$ cd /tank/test4
$ rm -rf gen gen-copy; sh ./gen.sh ; cp -a gen gen-copy ; sh ./read-gen.sh ; sh ./read-cp.sh

# switch the read order
$ rm -rf gen gen-copy; sh ./gen.sh ; cp -a gen gen-copy ; sh ./read-cp.sh; sh ./read-gen.sh  

Include any warning/errors/backtraces from the system logs

The simulate test result(The lower the better):

test env (CentOS7) read from generating dir(secs) read from cp dir(secs)
MD3060E+ 0.8.4(new) 84~99 60~61
MD3060E+ 0.8.4(FRAG 13%) 113~119 67~68
MD3060E+ 0.7.13(new) 94~100 64~67
MD3060E+ 0.7.13(FRAG 51%) 133~141 75~77
MD3060E+ 0.6.5.11(new) 99~100 101~103

Here is the original in my production env, I can't re-create the zpool, I test 20 times. 10x dirA, 10x dirB, fio 3.7, I use script parallel to generate the fio test file.

test env (CentOS8) read from generating dir(MB/s) read from cp dir(MB/s)
R740 + 0.8.4(FRAG 5%) 200~400 600~800

Thanks : )

Here is the zdb -dddddd show the same file in diff dirs. after cp parallel write

Test parameters

/sys/module/zfs/parameters/dbuf_cache_hiwater_pct 10
/sys/module/zfs/parameters/dbuf_cache_hiwater_pct 10
/sys/module/zfs/parameters/dbuf_cache_lowater_pct 10
/sys/module/zfs/parameters/dbuf_cache_lowater_pct 10
/sys/module/zfs/parameters/dbuf_cache_max_bytes 1051399104
/sys/module/zfs/parameters/dbuf_cache_max_bytes 1051399104
/sys/module/zfs/parameters/dbuf_cache_shift 5
/sys/module/zfs/parameters/dbuf_cache_shift 5
/sys/module/zfs/parameters/dbuf_metadata_cache_max_bytes 525699552
/sys/module/zfs/parameters/dbuf_metadata_cache_max_bytes 525699552
/sys/module/zfs/parameters/dbuf_metadata_cache_shift 6
/sys/module/zfs/parameters/dbuf_metadata_cache_shift 6
/sys/module/zfs/parameters/dmu_object_alloc_chunk_shift 7
/sys/module/zfs/parameters/dmu_object_alloc_chunk_shift 7
/sys/module/zfs/parameters/dmu_prefetch_max 134217728
/sys/module/zfs/parameters/dmu_prefetch_max 134217728
/sys/module/zfs/parameters/ignore_hole_birth 1
/sys/module/zfs/parameters/ignore_hole_birth 1
/sys/module/zfs/parameters/l2arc_feed_again 1
/sys/module/zfs/parameters/l2arc_feed_again 1
/sys/module/zfs/parameters/l2arc_feed_min_ms 200
/sys/module/zfs/parameters/l2arc_feed_min_ms 200
/sys/module/zfs/parameters/l2arc_feed_secs 1
/sys/module/zfs/parameters/l2arc_feed_secs 1
/sys/module/zfs/parameters/l2arc_headroom 2
/sys/module/zfs/parameters/l2arc_headroom 2
/sys/module/zfs/parameters/l2arc_headroom_boost 200
/sys/module/zfs/parameters/l2arc_headroom_boost 200
/sys/module/zfs/parameters/l2arc_noprefetch 1
/sys/module/zfs/parameters/l2arc_noprefetch 1
/sys/module/zfs/parameters/l2arc_norw 0
/sys/module/zfs/parameters/l2arc_norw 0
/sys/module/zfs/parameters/l2arc_write_boost 8388608
/sys/module/zfs/parameters/l2arc_write_boost 8388608
/sys/module/zfs/parameters/l2arc_write_max 8388608
/sys/module/zfs/parameters/l2arc_write_max 8388608
/sys/module/zfs/parameters/metaslab_aliquot 524288
/sys/module/zfs/parameters/metaslab_aliquot 524288
/sys/module/zfs/parameters/metaslab_bias_enabled 1
/sys/module/zfs/parameters/metaslab_bias_enabled 1
/sys/module/zfs/parameters/metaslab_debug_load 0
/sys/module/zfs/parameters/metaslab_debug_load 0
/sys/module/zfs/parameters/metaslab_debug_unload 1
/sys/module/zfs/parameters/metaslab_debug_unload 1
/sys/module/zfs/parameters/metaslab_df_max_search 16777216
/sys/module/zfs/parameters/metaslab_df_max_search 16777216
/sys/module/zfs/parameters/metaslab_df_use_largest_segment 0
/sys/module/zfs/parameters/metaslab_df_use_largest_segment 0
/sys/module/zfs/parameters/metaslab_force_ganging 16777217
/sys/module/zfs/parameters/metaslab_force_ganging 16777217
/sys/module/zfs/parameters/metaslab_fragmentation_factor_enabled 1
/sys/module/zfs/parameters/metaslab_fragmentation_factor_enabled 1
/sys/module/zfs/parameters/metaslab_lba_weighting_enabled 1
/sys/module/zfs/parameters/metaslab_lba_weighting_enabled 1
/sys/module/zfs/parameters/metaslab_preload_enabled 1
/sys/module/zfs/parameters/metaslab_preload_enabled 1
/sys/module/zfs/parameters/send_holes_without_birth_time 1
/sys/module/zfs/parameters/send_holes_without_birth_time 1
/sys/module/zfs/parameters/spa_asize_inflation 24
/sys/module/zfs/parameters/spa_asize_inflation 24
/sys/module/zfs/parameters/spa_config_path /etc/zfs/zpool.cache
/sys/module/zfs/parameters/spa_config_path /etc/zfs/zpool.cache
/sys/module/zfs/parameters/spa_load_print_vdev_tree 0
/sys/module/zfs/parameters/spa_load_print_vdev_tree 0
/sys/module/zfs/parameters/spa_load_verify_data 1
/sys/module/zfs/parameters/spa_load_verify_data 1
/sys/module/zfs/parameters/spa_load_verify_metadata 1
/sys/module/zfs/parameters/spa_load_verify_metadata 1
/sys/module/zfs/parameters/spa_load_verify_shift 4
/sys/module/zfs/parameters/spa_load_verify_shift 4
/sys/module/zfs/parameters/spa_slop_shift 5
/sys/module/zfs/parameters/spa_slop_shift 5
/sys/module/zfs/parameters/vdev_removal_max_span 32768
/sys/module/zfs/parameters/vdev_removal_max_span 32768
/sys/module/zfs/parameters/vdev_validate_skip 0
/sys/module/zfs/parameters/vdev_validate_skip 0
/sys/module/zfs/parameters/zap_iterate_prefetch 1
/sys/module/zfs/parameters/zap_iterate_prefetch 1
/sys/module/zfs/parameters/zfetch_array_rd_sz 1048576
/sys/module/zfs/parameters/zfetch_array_rd_sz 1048576
/sys/module/zfs/parameters/zfetch_max_distance 8388608
/sys/module/zfs/parameters/zfetch_max_distance 8388608
/sys/module/zfs/parameters/zfetch_max_streams 8
/sys/module/zfs/parameters/zfetch_max_streams 8
/sys/module/zfs/parameters/zfetch_min_sec_reap 2
/sys/module/zfs/parameters/zfetch_min_sec_reap 2
/sys/module/zfs/parameters/zfs_abd_scatter_enabled 1
/sys/module/zfs/parameters/zfs_abd_scatter_enabled 1
/sys/module/zfs/parameters/zfs_abd_scatter_max_order 10
/sys/module/zfs/parameters/zfs_abd_scatter_max_order 10
/sys/module/zfs/parameters/zfs_abd_scatter_min_size 1536
/sys/module/zfs/parameters/zfs_abd_scatter_min_size 1536
/sys/module/zfs/parameters/zfs_admin_snapshot 0
/sys/module/zfs/parameters/zfs_admin_snapshot 0
/sys/module/zfs/parameters/zfs_arc_average_blocksize 8192
/sys/module/zfs/parameters/zfs_arc_average_blocksize 8192
/sys/module/zfs/parameters/zfs_arc_dnode_limit 0
/sys/module/zfs/parameters/zfs_arc_dnode_limit 0
/sys/module/zfs/parameters/zfs_arc_dnode_limit_percent 10
/sys/module/zfs/parameters/zfs_arc_dnode_limit_percent 10
/sys/module/zfs/parameters/zfs_arc_dnode_reduce_percent 10
/sys/module/zfs/parameters/zfs_arc_dnode_reduce_percent 10
/sys/module/zfs/parameters/zfs_arc_grow_retry 0
/sys/module/zfs/parameters/zfs_arc_grow_retry 0
/sys/module/zfs/parameters/zfs_arc_lotsfree_percent 10
/sys/module/zfs/parameters/zfs_arc_lotsfree_percent 10
/sys/module/zfs/parameters/zfs_arc_max 0
/sys/module/zfs/parameters/zfs_arc_max 0
/sys/module/zfs/parameters/zfs_arc_meta_adjust_restarts 4096
/sys/module/zfs/parameters/zfs_arc_meta_adjust_restarts 4096
/sys/module/zfs/parameters/zfs_arc_meta_limit 0
/sys/module/zfs/parameters/zfs_arc_meta_limit 0
/sys/module/zfs/parameters/zfs_arc_meta_limit_percent 75
/sys/module/zfs/parameters/zfs_arc_meta_limit_percent 75
/sys/module/zfs/parameters/zfs_arc_meta_min 0
/sys/module/zfs/parameters/zfs_arc_meta_min 0
/sys/module/zfs/parameters/zfs_arc_meta_prune 10000
/sys/module/zfs/parameters/zfs_arc_meta_prune 10000
/sys/module/zfs/parameters/zfs_arc_meta_strategy 1
/sys/module/zfs/parameters/zfs_arc_meta_strategy 1
/sys/module/zfs/parameters/zfs_arc_min 0
/sys/module/zfs/parameters/zfs_arc_min 0
/sys/module/zfs/parameters/zfs_arc_min_prefetch_ms 0
/sys/module/zfs/parameters/zfs_arc_min_prefetch_ms 0
/sys/module/zfs/parameters/zfs_arc_min_prescient_prefetch_ms 0
/sys/module/zfs/parameters/zfs_arc_min_prescient_prefetch_ms 0
/sys/module/zfs/parameters/zfs_arc_pc_percent 0
/sys/module/zfs/parameters/zfs_arc_pc_percent 0
/sys/module/zfs/parameters/zfs_arc_p_dampener_disable 1
/sys/module/zfs/parameters/zfs_arc_p_dampener_disable 1
/sys/module/zfs/parameters/zfs_arc_p_min_shift 0
/sys/module/zfs/parameters/zfs_arc_p_min_shift 0
/sys/module/zfs/parameters/zfs_arc_shrink_shift 0
/sys/module/zfs/parameters/zfs_arc_shrink_shift 0
/sys/module/zfs/parameters/zfs_arc_sys_free 0
/sys/module/zfs/parameters/zfs_arc_sys_free 0
/sys/module/zfs/parameters/zfs_async_block_max_blocks 100000
/sys/module/zfs/parameters/zfs_async_block_max_blocks 100000
/sys/module/zfs/parameters/zfs_autoimport_disable 1
/sys/module/zfs/parameters/zfs_autoimport_disable 1
/sys/module/zfs/parameters/zfs_checksum_events_per_second 20
/sys/module/zfs/parameters/zfs_checksum_events_per_second 20
/sys/module/zfs/parameters/zfs_commit_timeout_pct 5
/sys/module/zfs/parameters/zfs_commit_timeout_pct 5
/sys/module/zfs/parameters/zfs_compressed_arc_enabled 1
/sys/module/zfs/parameters/zfs_compressed_arc_enabled 1
/sys/module/zfs/parameters/zfs_condense_indirect_commit_entry_delay_ms 0
/sys/module/zfs/parameters/zfs_condense_indirect_commit_entry_delay_ms 0
/sys/module/zfs/parameters/zfs_condense_indirect_vdevs_enable 1
/sys/module/zfs/parameters/zfs_condense_indirect_vdevs_enable 1
/sys/module/zfs/parameters/zfs_condense_max_obsolete_bytes 1073741824
/sys/module/zfs/parameters/zfs_condense_max_obsolete_bytes 1073741824
/sys/module/zfs/parameters/zfs_condense_min_mapping_bytes 131072
/sys/module/zfs/parameters/zfs_condense_min_mapping_bytes 131072
/sys/module/zfs/parameters/zfs_dbgmsg_enable 1
/sys/module/zfs/parameters/zfs_dbgmsg_enable 1
/sys/module/zfs/parameters/zfs_dbgmsg_maxsize 4194304
/sys/module/zfs/parameters/zfs_dbgmsg_maxsize 4194304
/sys/module/zfs/parameters/zfs_dbuf_state_index 0
/sys/module/zfs/parameters/zfs_dbuf_state_index 0
/sys/module/zfs/parameters/zfs_ddt_data_is_special 1
/sys/module/zfs/parameters/zfs_ddt_data_is_special 1
/sys/module/zfs/parameters/zfs_deadman_checktime_ms 60000
/sys/module/zfs/parameters/zfs_deadman_checktime_ms 60000
/sys/module/zfs/parameters/zfs_deadman_enabled 1
/sys/module/zfs/parameters/zfs_deadman_enabled 1
/sys/module/zfs/parameters/zfs_deadman_failmode wait
/sys/module/zfs/parameters/zfs_deadman_failmode wait
/sys/module/zfs/parameters/zfs_deadman_synctime_ms 600000
/sys/module/zfs/parameters/zfs_deadman_synctime_ms 600000
/sys/module/zfs/parameters/zfs_deadman_ziotime_ms 300000
/sys/module/zfs/parameters/zfs_deadman_ziotime_ms 300000
/sys/module/zfs/parameters/zfs_dedup_prefetch 0
/sys/module/zfs/parameters/zfs_dedup_prefetch 0
/sys/module/zfs/parameters/zfs_delay_min_dirty_percent 60
/sys/module/zfs/parameters/zfs_delay_min_dirty_percent 60
/sys/module/zfs/parameters/zfs_delay_scale 500000
/sys/module/zfs/parameters/zfs_delay_scale 500000
/sys/module/zfs/parameters/zfs_delete_blocks 20480
/sys/module/zfs/parameters/zfs_delete_blocks 20480
/sys/module/zfs/parameters/zfs_dirty_data_max 4294967296
/sys/module/zfs/parameters/zfs_dirty_data_max 4294967296
/sys/module/zfs/parameters/zfs_dirty_data_max_max 4294967296
/sys/module/zfs/parameters/zfs_dirty_data_max_max 4294967296
/sys/module/zfs/parameters/zfs_dirty_data_max_max_percent 25
/sys/module/zfs/parameters/zfs_dirty_data_max_max_percent 25
/sys/module/zfs/parameters/zfs_dirty_data_max_percent 10
/sys/module/zfs/parameters/zfs_dirty_data_max_percent 10
/sys/module/zfs/parameters/zfs_dirty_data_sync_percent 20
/sys/module/zfs/parameters/zfs_dirty_data_sync_percent 20
/sys/module/zfs/parameters/zfs_disable_ivset_guid_check 0
/sys/module/zfs/parameters/zfs_disable_ivset_guid_check 0
/sys/module/zfs/parameters/zfs_dmu_offset_next_sync 0
/sys/module/zfs/parameters/zfs_dmu_offset_next_sync 0
/sys/module/zfs/parameters/zfs_expire_snapshot 300
/sys/module/zfs/parameters/zfs_expire_snapshot 300
/sys/module/zfs/parameters/zfs_flags 0
/sys/module/zfs/parameters/zfs_flags 0
/sys/module/zfs/parameters/zfs_free_bpobj_enabled 1
/sys/module/zfs/parameters/zfs_free_bpobj_enabled 1
/sys/module/zfs/parameters/zfs_free_leak_on_eio 0
/sys/module/zfs/parameters/zfs_free_leak_on_eio 0
/sys/module/zfs/parameters/zfs_free_min_time_ms 1000
/sys/module/zfs/parameters/zfs_free_min_time_ms 1000
/sys/module/zfs/parameters/zfs_immediate_write_sz 32768
/sys/module/zfs/parameters/zfs_immediate_write_sz 32768
/sys/module/zfs/parameters/zfs_initialize_value 16045690984833335022
/sys/module/zfs/parameters/zfs_initialize_value 16045690984833335022
/sys/module/zfs/parameters/zfs_key_max_salt_uses 400000000
/sys/module/zfs/parameters/zfs_key_max_salt_uses 400000000
/sys/module/zfs/parameters/zfs_lua_max_instrlimit 100000000
/sys/module/zfs/parameters/zfs_lua_max_instrlimit 100000000
/sys/module/zfs/parameters/zfs_lua_max_memlimit 104857600
/sys/module/zfs/parameters/zfs_lua_max_memlimit 104857600
/sys/module/zfs/parameters/zfs_max_missing_tvds 0
/sys/module/zfs/parameters/zfs_max_missing_tvds 0
/sys/module/zfs/parameters/zfs_max_recordsize 1048576
/sys/module/zfs/parameters/zfs_max_recordsize 1048576
/sys/module/zfs/parameters/zfs_metaslab_fragmentation_threshold 70
/sys/module/zfs/parameters/zfs_metaslab_fragmentation_threshold 70
/sys/module/zfs/parameters/zfs_metaslab_segment_weight_enabled 1
/sys/module/zfs/parameters/zfs_metaslab_segment_weight_enabled 1
/sys/module/zfs/parameters/zfs_metaslab_switch_threshold 2
/sys/module/zfs/parameters/zfs_metaslab_switch_threshold 2
/sys/module/zfs/parameters/zfs_mg_fragmentation_threshold 95
/sys/module/zfs/parameters/zfs_mg_fragmentation_threshold 95
/sys/module/zfs/parameters/zfs_mg_noalloc_threshold 0
/sys/module/zfs/parameters/zfs_mg_noalloc_threshold 0
/sys/module/zfs/parameters/zfs_multihost_fail_intervals 10
/sys/module/zfs/parameters/zfs_multihost_fail_intervals 10
/sys/module/zfs/parameters/zfs_multihost_history 0
/sys/module/zfs/parameters/zfs_multihost_history 0
/sys/module/zfs/parameters/zfs_multihost_import_intervals 20
/sys/module/zfs/parameters/zfs_multihost_import_intervals 20
/sys/module/zfs/parameters/zfs_multihost_interval 1000
/sys/module/zfs/parameters/zfs_multihost_interval 1000
/sys/module/zfs/parameters/zfs_multilist_num_sublists 0
/sys/module/zfs/parameters/zfs_multilist_num_sublists 0
/sys/module/zfs/parameters/zfs_nocacheflush 0
/sys/module/zfs/parameters/zfs_nocacheflush 0
/sys/module/zfs/parameters/zfs_nopwrite_enabled 1
/sys/module/zfs/parameters/zfs_nopwrite_enabled 1
/sys/module/zfs/parameters/zfs_no_scrub_io 0
/sys/module/zfs/parameters/zfs_no_scrub_io 0
/sys/module/zfs/parameters/zfs_no_scrub_prefetch 0
/sys/module/zfs/parameters/zfs_no_scrub_prefetch 0
/sys/module/zfs/parameters/zfs_object_mutex_size 64
/sys/module/zfs/parameters/zfs_object_mutex_size 64
/sys/module/zfs/parameters/zfs_obsolete_min_time_ms 500
/sys/module/zfs/parameters/zfs_obsolete_min_time_ms 500
/sys/module/zfs/parameters/zfs_override_estimate_recordsize 0
/sys/module/zfs/parameters/zfs_override_estimate_recordsize 0
/sys/module/zfs/parameters/zfs_pd_bytes_max 52428800
/sys/module/zfs/parameters/zfs_pd_bytes_max 52428800
/sys/module/zfs/parameters/zfs_per_txg_dirty_frees_percent 5
/sys/module/zfs/parameters/zfs_per_txg_dirty_frees_percent 5
/sys/module/zfs/parameters/zfs_prefetch_disable 0
/sys/module/zfs/parameters/zfs_prefetch_disable 0
/sys/module/zfs/parameters/zfs_read_chunk_size 1048576
/sys/module/zfs/parameters/zfs_read_chunk_size 1048576
/sys/module/zfs/parameters/zfs_read_history 0
/sys/module/zfs/parameters/zfs_read_history 0
/sys/module/zfs/parameters/zfs_read_history_hits 0
/sys/module/zfs/parameters/zfs_read_history_hits 0
/sys/module/zfs/parameters/zfs_reconstruct_indirect_combinations_max 4096
/sys/module/zfs/parameters/zfs_reconstruct_indirect_combinations_max 4096
/sys/module/zfs/parameters/zfs_recover 0
/sys/module/zfs/parameters/zfs_recover 0
/sys/module/zfs/parameters/zfs_recv_queue_length 16777216
/sys/module/zfs/parameters/zfs_recv_queue_length 16777216
/sys/module/zfs/parameters/zfs_removal_ignore_errors 0
/sys/module/zfs/parameters/zfs_removal_ignore_errors 0
/sys/module/zfs/parameters/zfs_removal_suspend_progress 0
/sys/module/zfs/parameters/zfs_removal_suspend_progress 0
/sys/module/zfs/parameters/zfs_remove_max_segment 16777216
/sys/module/zfs/parameters/zfs_remove_max_segment 16777216
/sys/module/zfs/parameters/zfs_resilver_disable_defer 0
/sys/module/zfs/parameters/zfs_resilver_disable_defer 0
/sys/module/zfs/parameters/zfs_resilver_min_time_ms 3000
/sys/module/zfs/parameters/zfs_resilver_min_time_ms 3000
/sys/module/zfs/parameters/zfs_scan_checkpoint_intval 7200
/sys/module/zfs/parameters/zfs_scan_checkpoint_intval 7200
/sys/module/zfs/parameters/zfs_scan_fill_weight 3
/sys/module/zfs/parameters/zfs_scan_fill_weight 3
/sys/module/zfs/parameters/zfs_scan_ignore_errors 0
/sys/module/zfs/parameters/zfs_scan_ignore_errors 0
/sys/module/zfs/parameters/zfs_scan_issue_strategy 0
/sys/module/zfs/parameters/zfs_scan_issue_strategy 0
/sys/module/zfs/parameters/zfs_scan_legacy 0
/sys/module/zfs/parameters/zfs_scan_legacy 0
/sys/module/zfs/parameters/zfs_scan_max_ext_gap 2097152
/sys/module/zfs/parameters/zfs_scan_max_ext_gap 2097152
/sys/module/zfs/parameters/zfs_scan_mem_lim_fact 20
/sys/module/zfs/parameters/zfs_scan_mem_lim_fact 20
/sys/module/zfs/parameters/zfs_scan_mem_lim_soft_fact 20
/sys/module/zfs/parameters/zfs_scan_mem_lim_soft_fact 20
/sys/module/zfs/parameters/zfs_scan_strict_mem_lim 0
/sys/module/zfs/parameters/zfs_scan_strict_mem_lim 0
/sys/module/zfs/parameters/zfs_scan_suspend_progress 0
/sys/module/zfs/parameters/zfs_scan_suspend_progress 0
/sys/module/zfs/parameters/zfs_scan_vdev_limit 4194304
/sys/module/zfs/parameters/zfs_scan_vdev_limit 4194304
/sys/module/zfs/parameters/zfs_scrub_min_time_ms 1000
/sys/module/zfs/parameters/zfs_scrub_min_time_ms 1000
/sys/module/zfs/parameters/zfs_send_corrupt_data 0
/sys/module/zfs/parameters/zfs_send_corrupt_data 0
/sys/module/zfs/parameters/zfs_send_queue_length 16777216
/sys/module/zfs/parameters/zfs_send_queue_length 16777216
/sys/module/zfs/parameters/zfs_send_unmodified_spill_blocks 1
/sys/module/zfs/parameters/zfs_send_unmodified_spill_blocks 1
/sys/module/zfs/parameters/zfs_slow_io_events_per_second 20
/sys/module/zfs/parameters/zfs_slow_io_events_per_second 20
/sys/module/zfs/parameters/zfs_spa_discard_memory_limit 16777216
/sys/module/zfs/parameters/zfs_spa_discard_memory_limit 16777216
/sys/module/zfs/parameters/zfs_special_class_metadata_reserve_pct 25
/sys/module/zfs/parameters/zfs_special_class_metadata_reserve_pct 25
/sys/module/zfs/parameters/zfs_sync_pass_deferred_free 2
/sys/module/zfs/parameters/zfs_sync_pass_deferred_free 2
/sys/module/zfs/parameters/zfs_sync_pass_dont_compress 8
/sys/module/zfs/parameters/zfs_sync_pass_dont_compress 8
/sys/module/zfs/parameters/zfs_sync_pass_rewrite 2
/sys/module/zfs/parameters/zfs_sync_pass_rewrite 2
/sys/module/zfs/parameters/zfs_sync_taskq_batch_pct 75
/sys/module/zfs/parameters/zfs_sync_taskq_batch_pct 75
/sys/module/zfs/parameters/zfs_trim_extent_bytes_max 134217728
/sys/module/zfs/parameters/zfs_trim_extent_bytes_max 134217728
/sys/module/zfs/parameters/zfs_trim_extent_bytes_min 32768
/sys/module/zfs/parameters/zfs_trim_extent_bytes_min 32768
/sys/module/zfs/parameters/zfs_trim_metaslab_skip 0
/sys/module/zfs/parameters/zfs_trim_metaslab_skip 0
/sys/module/zfs/parameters/zfs_trim_queue_limit 10
/sys/module/zfs/parameters/zfs_trim_queue_limit 10
/sys/module/zfs/parameters/zfs_trim_txg_batch 32
/sys/module/zfs/parameters/zfs_trim_txg_batch 32
/sys/module/zfs/parameters/zfs_txg_history 100
/sys/module/zfs/parameters/zfs_txg_history 100
/sys/module/zfs/parameters/zfs_txg_timeout 5
/sys/module/zfs/parameters/zfs_txg_timeout 5
/sys/module/zfs/parameters/zfs_unlink_suspend_progress 0
/sys/module/zfs/parameters/zfs_unlink_suspend_progress 0
/sys/module/zfs/parameters/zfs_user_indirect_is_special 1
/sys/module/zfs/parameters/zfs_user_indirect_is_special 1
/sys/module/zfs/parameters/zfs_vdev_aggregate_trim 0
/sys/module/zfs/parameters/zfs_vdev_aggregate_trim 0
/sys/module/zfs/parameters/zfs_vdev_aggregation_limit 1048576
/sys/module/zfs/parameters/zfs_vdev_aggregation_limit 1048576
/sys/module/zfs/parameters/zfs_vdev_aggregation_limit_non_rotating 131072
/sys/module/zfs/parameters/zfs_vdev_aggregation_limit_non_rotating 131072
/sys/module/zfs/parameters/zfs_vdev_async_read_max_active 3
/sys/module/zfs/parameters/zfs_vdev_async_read_max_active 3
/sys/module/zfs/parameters/zfs_vdev_async_read_min_active 1
/sys/module/zfs/parameters/zfs_vdev_async_read_min_active 1
/sys/module/zfs/parameters/zfs_vdev_async_write_active_max_dirty_percent 60
/sys/module/zfs/parameters/zfs_vdev_async_write_active_max_dirty_percent 60
/sys/module/zfs/parameters/zfs_vdev_async_write_active_min_dirty_percent 30
/sys/module/zfs/parameters/zfs_vdev_async_write_active_min_dirty_percent 30
/sys/module/zfs/parameters/zfs_vdev_async_write_max_active 10
/sys/module/zfs/parameters/zfs_vdev_async_write_max_active 10
/sys/module/zfs/parameters/zfs_vdev_async_write_min_active 2
/sys/module/zfs/parameters/zfs_vdev_async_write_min_active 2
/sys/module/zfs/parameters/zfs_vdev_cache_bshift 16
/sys/module/zfs/parameters/zfs_vdev_cache_bshift 16
/sys/module/zfs/parameters/zfs_vdev_cache_max 16384
/sys/module/zfs/parameters/zfs_vdev_cache_max 16384
/sys/module/zfs/parameters/zfs_vdev_cache_size 0
/sys/module/zfs/parameters/zfs_vdev_cache_size 0
/sys/module/zfs/parameters/zfs_vdev_default_ms_count 200
/sys/module/zfs/parameters/zfs_vdev_default_ms_count 200
/sys/module/zfs/parameters/zfs_vdev_initializing_max_active 1
/sys/module/zfs/parameters/zfs_vdev_initializing_max_active 1
/sys/module/zfs/parameters/zfs_vdev_initializing_min_active 1
/sys/module/zfs/parameters/zfs_vdev_initializing_min_active 1
/sys/module/zfs/parameters/zfs_vdev_max_active 1000
/sys/module/zfs/parameters/zfs_vdev_max_active 1000
/sys/module/zfs/parameters/zfs_vdev_min_ms_count 16
/sys/module/zfs/parameters/zfs_vdev_min_ms_count 16
/sys/module/zfs/parameters/zfs_vdev_mirror_non_rotating_inc 0
/sys/module/zfs/parameters/zfs_vdev_mirror_non_rotating_inc 0
/sys/module/zfs/parameters/zfs_vdev_mirror_non_rotating_seek_inc 1
/sys/module/zfs/parameters/zfs_vdev_mirror_non_rotating_seek_inc 1
/sys/module/zfs/parameters/zfs_vdev_mirror_rotating_inc 0
/sys/module/zfs/parameters/zfs_vdev_mirror_rotating_inc 0
/sys/module/zfs/parameters/zfs_vdev_mirror_rotating_seek_inc 5
/sys/module/zfs/parameters/zfs_vdev_mirror_rotating_seek_inc 5
/sys/module/zfs/parameters/zfs_vdev_mirror_rotating_seek_offset 1048576
/sys/module/zfs/parameters/zfs_vdev_mirror_rotating_seek_offset 1048576
/sys/module/zfs/parameters/zfs_vdev_ms_count_limit 131072
/sys/module/zfs/parameters/zfs_vdev_ms_count_limit 131072
/sys/module/zfs/parameters/zfs_vdev_queue_depth_pct 1000
/sys/module/zfs/parameters/zfs_vdev_queue_depth_pct 1000
/sys/module/zfs/parameters/zfs_vdev_raidz_impl cycle [fastest] original scalar sse2 ssse3 avx2
/sys/module/zfs/parameters/zfs_vdev_raidz_impl cycle [fastest] original scalar sse2 ssse3 avx2
/sys/module/zfs/parameters/zfs_vdev_read_gap_limit 32768
/sys/module/zfs/parameters/zfs_vdev_read_gap_limit 32768
/sys/module/zfs/parameters/zfs_vdev_removal_max_active 2
/sys/module/zfs/parameters/zfs_vdev_removal_max_active 2
/sys/module/zfs/parameters/zfs_vdev_removal_min_active 1
/sys/module/zfs/parameters/zfs_vdev_removal_min_active 1
/sys/module/zfs/parameters/zfs_vdev_scheduler deadline
/sys/module/zfs/parameters/zfs_vdev_scheduler deadline
/sys/module/zfs/parameters/zfs_vdev_scrub_max_active 2
/sys/module/zfs/parameters/zfs_vdev_scrub_max_active 2
/sys/module/zfs/parameters/zfs_vdev_scrub_min_active 1
/sys/module/zfs/parameters/zfs_vdev_scrub_min_active 1
/sys/module/zfs/parameters/zfs_vdev_sync_read_max_active 10
/sys/module/zfs/parameters/zfs_vdev_sync_read_max_active 10
/sys/module/zfs/parameters/zfs_vdev_sync_read_min_active 10
/sys/module/zfs/parameters/zfs_vdev_sync_read_min_active 10
/sys/module/zfs/parameters/zfs_vdev_sync_write_max_active 10
/sys/module/zfs/parameters/zfs_vdev_sync_write_max_active 10
/sys/module/zfs/parameters/zfs_vdev_sync_write_min_active 10
/sys/module/zfs/parameters/zfs_vdev_sync_write_min_active 10
/sys/module/zfs/parameters/zfs_vdev_trim_max_active 2
/sys/module/zfs/parameters/zfs_vdev_trim_max_active 2
/sys/module/zfs/parameters/zfs_vdev_trim_min_active 1
/sys/module/zfs/parameters/zfs_vdev_trim_min_active 1
/sys/module/zfs/parameters/zfs_vdev_write_gap_limit 4096
/sys/module/zfs/parameters/zfs_vdev_write_gap_limit 4096
/sys/module/zfs/parameters/zfs_zevent_cols 80
/sys/module/zfs/parameters/zfs_zevent_cols 80
/sys/module/zfs/parameters/zfs_zevent_console 0
/sys/module/zfs/parameters/zfs_zevent_console 0
/sys/module/zfs/parameters/zfs_zevent_len_max 128
/sys/module/zfs/parameters/zfs_zevent_len_max 128
/sys/module/zfs/parameters/zfs_zil_clean_taskq_maxalloc 1048576
/sys/module/zfs/parameters/zfs_zil_clean_taskq_maxalloc 1048576
/sys/module/zfs/parameters/zfs_zil_clean_taskq_minalloc 1024
/sys/module/zfs/parameters/zfs_zil_clean_taskq_minalloc 1024
/sys/module/zfs/parameters/zfs_zil_clean_taskq_nthr_pct 100
/sys/module/zfs/parameters/zfs_zil_clean_taskq_nthr_pct 100
/sys/module/zfs/parameters/zil_maxblocksize 131072
/sys/module/zfs/parameters/zil_maxblocksize 131072
/sys/module/zfs/parameters/zil_nocacheflush 0
/sys/module/zfs/parameters/zil_nocacheflush 0
/sys/module/zfs/parameters/zil_replay_disable 0
/sys/module/zfs/parameters/zil_replay_disable 0
/sys/module/zfs/parameters/zil_slog_bulk 786432
/sys/module/zfs/parameters/zil_slog_bulk 786432
/sys/module/zfs/parameters/zio_deadman_log_all 0
/sys/module/zfs/parameters/zio_deadman_log_all 0
/sys/module/zfs/parameters/zio_dva_throttle_enabled 0
/sys/module/zfs/parameters/zio_dva_throttle_enabled 0
/sys/module/zfs/parameters/zio_requeue_io_start_cut_in_line 1
/sys/module/zfs/parameters/zio_requeue_io_start_cut_in_line 1
/sys/module/zfs/parameters/zio_slow_io_ms 30000
/sys/module/zfs/parameters/zio_slow_io_ms 30000
/sys/module/zfs/parameters/zio_taskq_batch_pct 75
/sys/module/zfs/parameters/zio_taskq_batch_pct 75
/sys/module/zfs/parameters/zvol_inhibit_dev 0
/sys/module/zfs/parameters/zvol_inhibit_dev 0
/sys/module/zfs/parameters/zvol_major 230
/sys/module/zfs/parameters/zvol_major 230
/sys/module/zfs/parameters/zvol_max_discard_blocks 16384
/sys/module/zfs/parameters/zvol_max_discard_blocks 16384
/sys/module/zfs/parameters/zvol_prefetch_bytes 131072
/sys/module/zfs/parameters/zvol_prefetch_bytes 131072
/sys/module/zfs/parameters/zvol_request_sync 0
/sys/module/zfs/parameters/zvol_request_sync 0
/sys/module/zfs/parameters/zvol_threads 32
/sys/module/zfs/parameters/zvol_threads 32
/sys/module/zfs/parameters/zvol_volmode 1
/sys/module/zfs/parameters/zvol_volmode 1
stale[bot] commented 3 years ago

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.