mongo zfs pre-reads

[alja]

Notes on online documentation

Understanding ZFS: Prefetch by Ben Rockwood http://www.cuddletech.com/blog/pivot/entry.php?id=1040

• Documentation for Solaris.
• Dtrace seem to be a very useful tool and the script to monitor pre reads too. But, dtrace is not available in mongo. ??? maybe I shoud try harder. In the /usr/include code I can see source of the functions dmu_zfetch_fetch() and zfs:vdev_cache_read(). As I read drace is not fully supported on linux, an alternative is systap tool. That is another project...
• mentioning parameter zfs_prefetch_disable in the /etc/system file, which does not exist in Centos6. I guess this is a kernel parameter???

[alja]

zfs tuning on FreeBSD https://wiki.freebsd.org/ZFSTuningGuide

• has description of ARC and L2ARC cache. Here I'm confused becuse there is no diect mentioning how l2arc is related to prefetch.
• mentioning parameter vfs.zfs.l2arc_noprefetch, which is not a centos6 kerenel or zfs parameter, at least what what I could grep with
  • sysctl -a
  • zfs get -all

[alja]

does zfs-read ahead, rc6 google group discuss https://groups.google.com/a/zfsonlinux.org/forum/#!topic/zfs-discuss/CmqaV0-WZaQ

• Can look into memory usage in files:
  • cat /proc/spl/kmem/slab
  • cat /proc/spl/kstat/zfs/arcstats
  • .... in the same directory noticed also /proc/spl/kstat/zfs/zfetchstats
• Mentioning editing /etc/modprobe.d/zfs.conf where zfs_arc_max is changed, ... cant see the modprobe file neither can I get zfs_arc_max, but I couls see lots of files in /sys/module/zfs/parameters ??? I don't know where are this set, they are not in the list of paramters in 'zfs get -all'
• Found this link which says the module paramters setting can work only under this conditions http://serverfault.com/questions/581669/why-isnt-the-arc-max-setting-honoured-on-zfs-on-linux

[alja]

arcstat.py

http://fibrevillage.com/storage/169-zfs-arc-on-linux-how-to-set-and-monitor-on-linux

example getting prefetch statistic every 10 s , 10 times [root@mongo parameters]# arcstat.py -f time,ph%,pm%,pread,pmis,phit 10 10 14:27:06 0 0 0 0 0 0 0 0 0 31G 31G

ph% : Prefetch hits percentage pm% : Prefetch miss percentage pread : Prefetch accesses per second pmis : Prefetch misses per second phit : Prefetch hits per second

[alja]

my confusion on linux

missing /etc/modprobe.d/zfs.conf or should I edit files??? what this has to do whit zfs command and its available options????

[root@mongo parameters]# for i in /sys/module/zfs/parameters/fetch; do echo "$i >>>> " ; cat $i; done /sys/module/zfs/parameters/l2arc_noprefetch >>>> 1 /sys/module/zfs/parameters/zfetch_array_rd_sz >>>> 1048576 /sys/module/zfs/parameters/zfetch_block_cap >>>> 256 /sys/module/zfs/parameters/zfetch_max_streams >>>> 8 /sys/module/zfs/parameters/zfetch_min_sec_reap >>>> 2 /sys/module/zfs/parameters/zfs_arc_min_prefetch_lifespan >>>> 1000 /sys/module/zfs/parameters/zfs_dedup_prefetch >>>> 0 /sys/module/zfs/parameters/zfs_no_scrub_prefetch >>>> 0 /sys/module

[alja]

[root@mongo ~]# arcstat.py 60 time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c 15:24:28 0 0 9 0 25 0 0 0 0 31G 31G 15:25:28 8.6K 1.4K 15 73 1 1.3K 28 47 1 31G 31G 15:26:28 8.4K 1.4K 16 73 1 1.3K 30 46 1 31G 31G 15:27:28 10K 1.6K 14 89 1 1.5K 25 57 1 31G 31G 15:28:28 10K 1.6K 15 84 1 1.5K 27 53 1 31G 31G 15:29:28 8.3K 1.4K 17 72 1 1.4K 32 45 1 31G 31G 15:30:28 9.2K 1.5K 16 76 1 1.4K 29 47 1 31G 31G 15:31:28 9.5K 1.4K 15 79 1 1.4K 27 49 1 31G 31G 15:32:29 8.6K 1.4K 15 70 1 1.3K 28 44 1 31G 31G 15:33:29 9.7K 1.4K 13 77 1 1.3K 24 49 1 31G 31G 15:34:29 8.7K 1.4K 16 70 1 1.3K 28 44 1 31G 31G 15:35:29 9.8K 1.5K 15 79 1 1.5K 28 49 1 31G 31G 15:36:29 8.1K 1.4K 17 67 1 1.3K 31 42 1 31G 31G 15:37:29 10.0K 1.5K 14 80 1 1.4K 26 50 1 31G 31G 15:38:29 7.7K 1.3K 16 66 1 1.2K 31 42 1 31G 31G 15:39:29 9.8K 1.4K 13 78 1 1.3K 24 49 1 31G 31G 15:40:29 9.6K 1.4K 14 77 1 1.4K 26 49 1 31G 31G 15:41:29 10K 1.4K 13 86 1 1.3K 24 52 1 31G 31G 15:42:29 10.0K 1.4K 13 88 1 1.3K 24 51 1 31G 31G 15:43:29 10K 1.3K 13 87 1 1.2K 23 50 1 31G 31G 15:44:29 10K 1.4K 12 87 1 1.3K 22 53 1 31G 31G 15:45:29 9.6K 1.4K 14 79 1 1.4K 26 50 1 31G 31G 15:46:29 12K 1.5K 12 100 1 1.4K 21 62 1 31G 31G 15:47:30 13K 1.4K 10 104 1 1.3K 19 66 1 31G 31G 15:48:30 12K 884 7 86 1 797 11 60 1 31G 31G 15:49:30 11K 675 5 77 1 598 9 56 1 31G 31G 15:50:30 12K 613 4 81 1 531 8 55 1 31G 31G 15:51:30 9.4K 572 6 34 0 537 10 16 0 31G 31G

[alja]

arcstat.py continued

• hope this one show something meaningful .... arcstat.py -f time,hits,mread,pread,dread mread : Metadata accesses per second pread : Prefetch accesses per second dread : Demand accesses per second
• ARC reads seem to be negligable
• seems like L2ARC does not exists ... looking at the source code of the arcstat l2bytes : bytes read per second from the L2ARC l2read : Total L2ARC accesses per second l2size : Size of the L2ARC

[alja]

don't see high enough values

[alja@mongo ~]$ arcstat.py -f time,hits,mread,mm%,pread,pm%,dread,dm%,miss% 10 10 time hits mread mm% pread pm% dread dm% miss% 09:43:41 40 18 0 25 18 19 0 10 09:43:51 9.9K 4.6K 0 6.2K 22 5.1K 1 12 09:44:01 9.2K 4.5K 1 6.0K 26 4.9K 1 15 09:44:11 10K 5.0K 1 6.6K 20 5.5K 1 11 09:44:21 10K 5.0K 1 6.5K 21 5.4K 1 12 09:44:31 8.6K 4.5K 1 5.6K 29 4.8K 1 16 09:44:41 7.8K 4.1K 1 4.8K 26 4.4K 1 14 09:44:51 5.7K 3.0K 1 3.8K 34 3.3K 2 19 09:45:01 5.6K 2.9K 1 3.6K 30 3.2K 1 17