search

akiradeveloper / dm-writeboost

Log-structured Caching for Linux
GNU General Public License v2.0
123 stars 19 forks source link

writeboost: Failed to allocate data #202

Closed nhocchi007 closed 5 years ago

nhocchi007 commented 5 years ago

Dmesg dump:

[21861.412135] Out of memory: Kill process 17490 (gnome-terminal-) score 2 or sacrifice child [21861.412138] Killed process 840 (bash) total-vm:115568kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [21861.486416] vmalloc: allocation failure, allocated 7794429952 of 7801806848 bytes [21861.486420] dmsetup: page allocation failure: order:0, mode:0xd2 [21861.486423] CPU: 3 PID: 13856 Comm: dmsetup Kdump: loaded Tainted: G OE ------------ T 3.10.0-957.1.3.el7.x86_64 #1 [21861.486424] Hardware name: ASUS All Series/H81M-K, BIOS 1104 02/22/2016 [21861.486425] Call Trace: [21861.486431] [] dump_stack+0x19/0x1b [21861.486435] [] warn_alloc_failed+0x110/0x180 [21861.486438] [] ? alloc_pages_current+0x98/0x110 [21861.486442] [] vmalloc_node_range+0x23b/0x280 [21861.486446] [] ? large_array_alloc+0x45/0x90 [dm_writeboost] [21861.486449] [] vmalloc+0x52/0x60 [21861.486452] [] ? large_array_alloc+0x45/0x90 [dm_writeboost] [21861.486454] [] large_array_alloc+0x45/0x90 [dm_writeboost] [21861.486457] [] resume_cache+0x1ef/0xfe0 [dm_writeboost] [21861.486460] [] writeboost_ctr+0x4b3/0x840 [dm_writeboost] [21861.486483] [] dm_table_add_target+0x17d/0x440 [dm_mod] [21861.486495] [] table_load+0x157/0x390 [dm_mod] [21861.486505] [] ctl_ioctl+0x24e/0x550 [dm_mod] [21861.486514] [] ? retrieve_status+0x1c0/0x1c0 [dm_mod] [21861.486523] [] dm_ctl_ioctl+0xe/0x20 [dm_mod] [21861.486527] [] do_vfs_ioctl+0x3a0/0x5a0 [21861.486529] [] ? do_page_fault+0x228/0x500 [21861.486531] [] SyS_ioctl+0xa1/0xc0 [21861.486534] [] system_call_fastpath+0x22/0x27 [21861.486536] Mem-Info: [21861.486540] active_anon:285 inactive_anon:4107 isolated_anon:0 active_file:13 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:1 unstable:0 slab_reclaimable:10736 slab_unreclaimable:12467 mapped:1 shmem:4194 pagetables:9617 bounce:0 free:0 free_pcp:11 free_cma:0 [21861.486544] Node 0 DMA free:4kB min:136kB low:168kB high:204kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [21861.486550] lowmem_reserve[]: 0 2941 7674 7674 [21861.486554] Node 0 DMA32 free:0kB min:25852kB low:32312kB high:38776kB active_anon:332kB inactive_anon:6372kB active_file:0kB inactive_file:4kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3261892kB managed:3015408kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:6516kB slab_reclaimable:13408kB slab_unreclaimable:15612kB kernel_stack:2736kB pagetables:16400kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:15437 all_unreclaimable? yes [21861.486559] lowmem_reserve[]: 0 0 4732 4732 [21861.486562] Node 0 Normal free:0kB min:41592kB low:51988kB high:62388kB active_anon:808kB inactive_anon:10056kB active_file:52kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:4978688kB managed:4846092kB mlocked:0kB dirty:0kB writeback:4kB mapped:0kB shmem:10260kB slab_reclaimable:29536kB slab_unreclaimable:34256kB kernel_stack:6608kB pagetables:22068kB unstable:0kB bounce:0kB free_pcp:40kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:31190 all_unreclaimable? yes [21861.486567] lowmem_reserve[]: 0 0 0 0 [21861.486570] Node 0 DMA: 04kB 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 04096kB = 0kB [21861.486577] Node 0 DMA32: 04kB 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 04096kB = 0kB [21861.486584] Node 0 Normal: 04kB 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 0kB [21861.486592] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [21861.486594] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [21861.486595] 4353 total pagecache pages [21861.486597] 142 pages in swap cache [21861.486599] Swap cache stats: add 3564452, delete 3564241, find 401623/486974 [21861.486600] Free swap = 7357908kB [21861.486601] Total swap = 8126460kB [21861.486603] 2064142 pages RAM [21861.486604] 0 pages HighMem/MovableOnly [21861.486605] 94793 pages reserved [21861.674096] device-mapper: writeboost: Failed to allocate data [21861.674100] device-mapper: writeboost: Failed to allocate segment_header_array [21861.674101] device-mapper: writeboost: init_segment_header_array failed [21861.674451] device-mapper: table: 253:5: writeboost: resume_cache failed [21861.674454] device-mapper: ioctl: error adding target to table

With: lsblk

sdb 8:16 0 931.5G 0 disk └─sdb1 8:17 0 931.5G 0 part └─hdddisk-hdd1 253:3 0 931.5G 0 lvm
sdc 8:32 0 745.2G 0 disk └─sdc1 8:33 0 745.2G 0 part └─ssddisk-ssd1 253:2 0 745.2G 0 lvm

Command:

wbcreate wbdev1 /dev/mapper/hdddisk-hdd1 /dev/mapper/ssddisk-ssd1 --writeback_threshold=70 --reformat

Thank you for your time!

akiradeveloper commented 5 years ago

The title is incorrect. That is just because your SSD is too big and allocation for the metadata has failed.

https://github.com/akiradeveloper/dm-writeboost/wiki/Frequently-asked-Q&A

Writeboost allocates metadata in memory and the mount is roughly 1% of the SSD you use. So in this case, roughly 8GB memory is used. Using 750GB cache for 930GB backing store is ridiculous but aside this, you should choose either increasing the memory or use smaller SSD where I recommend the latter for this case.

nhocchi007 commented 5 years ago

Oh, i understand now, i just resize my disk to 245G( which fit to my free 4G of memory) and it work just fine! In my situation, it 's just a fast test and i'm sorry for didn't read Q&A before using it. For more research purpose, i just want know why 1%? And what do you think about 1T2 SSD cache for 60TB of HDD disk? Is it violate any convention about memory and cache? Thank you for sharing your knowledge

akiradeveloper commented 5 years ago

What do you mean by convention? It is just you must have 10GB memory to use 1TB SSD.

The number 1% comes from calculation and the strategy to putting all metadata in memory for performance optimization sounds sane to me because it is just about how you can leverage the memory consumption: If there is no SSD caching you only cache 10GB data and others go into HDD but you can get 1TB SSD caching by only paying 10GB memory. That sounds reasonable to me.