search

openzfsonwindows / openzfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
480 stars 18 forks source link

Bugcheck: "ABD chunk allocation failed" when system in low memory condition. #265

Open datacore-rm opened 1 year ago

datacore-rm commented 1 year ago

When the VM is in low memory condition, the new ABD chunk allocation failed and it then bug checked while writing to the null pointer in memcpy().

SYSTEM_THREAD_EXCEPTION_NOT_HANDLED (7e)
EXCEPTION_RECORD:  ffffdf066a4d6fc8 -- (.exr 0xffffdf066a4d6fc8)
ExceptionAddress: fffff8043b8f78d3 (ZFSin!memcpy+0x0000000000000113)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 0000000000000001
   Parameter[1]: 0000000000000000
Attempt to write to address 0000000000000000

2: kd> k
 # Child-SP          RetAddr           Call Site
00 ffffdf06`6a4d5f78 fffff804`2ea3e981 nt!KeBugCheckEx
01 ffffdf06`6a4d5f80 fffff804`2e9ddb6f nt!PspSystemThreadStartup$filt$0+0x44
02 ffffdf06`6a4d5fc0 fffff804`2ea2a41f nt!_C_specific_handler+0x9f
03 ffffdf06`6a4d6030 fffff804`2e8708e1 nt!RtlpExecuteHandlerForException+0xf
04 ffffdf06`6a4d6060 fffff804`2e874ee4 nt!RtlDispatchException+0x301
05 ffffdf06`6a4d67b0 fffff804`2ea3458e nt!KiDispatchException+0x304
06 ffffdf06`6a4d6e90 fffff804`2ea2fc26 nt!KiExceptionDispatch+0x10e
07 ffffdf06`6a4d7070 fffff804`3b8f78d3 nt!KiPageFault+0x426
08 ffffdf06`6a4d7208 fffff804`3b706c79 ZFSin!memcpy+0x113 [minkernel\crts\crtw32\string\amd64\memcpy.asm @ 255] 
09 ffffdf06`6a4d7210 fffff804`3b706b21 ZFSin!abd_copy_off_cb+0x9 [C:\BuildAgent\work\9f5ca3432854b81\module\zfs\abd.c @ 975] 
0a ffffdf06`6a4d7240 fffff804`3b706c68 ZFSin!abd_iterate_func2+0x1f1 [C:\BuildAgent\work\9f5ca3432854b81\module\zfs\abd.c @ 951] 
0b ffffdf06`6a4d7350 fffff804`3b6e0386 ZFSin!abd_copy_off+0x28 [C:\BuildAgent\work\9f5ca3432854b81\module\zfs\abd.c @ 986] 
0c ffffdf06`6a4d7390 fffff804`3b6a9e45 ZFSin!arc_write_ready+0x476 [C:\BuildAgent\work\9f5ca3432854b81\module\zfs\arc.c @ 6920] 
0d ffffdf06`6a4d7410 fffff804`3b6a6dbc ZFSin!zio_ready+0x75 [C:\BuildAgent\work\9f5ca3432854b81\module\zfs\zio.c @ 4373] 
0e ffffdf06`6a4d7490 fffff804`3b6033d3 ZFSin!__zio_execute+0x9c [C:\BuildAgent\work\9f5ca3432854b81\module\zfs\zio.c @ 2198] 
0f ffffdf06`6a4d74d0 fffff804`2e95f775 ZFSin!taskq_thread+0x233 [C:\BuildAgent\work\9f5ca3432854b81\module\os\windows\spl\spl-taskq.c @ 2098] 
10 ffffdf06`6a4d75b0 fffff804`2ea240c8 nt!PspSystemThreadStartup+0x55
11 ffffdf06`6a4d7600 00000000`00000000 nt!KiStartSystemThread+0x28
2: kd> .frame 0n12;dv /t /v
0c ffffdf06`6a4d7390 fffff804`3b6a9e45 ZFSin!arc_write_ready+0x476 [C:\BuildAgent\work\9f5ca3432854b81\module\zfs\arc.c @ 6920] 
<unavailable>     struct zio * zio = <value unavailable>
@r15              struct blkptr * bp = 0xffff8008`7bf73610
<unavailable>     struct arc_write_callback * callback = <value unavailable>
@rbx              struct arc_buf_hdr * hdr = 0xffff8007`a192d190
<unavailable>     struct arc_buf * buf = <value unavailable>
<unavailable>     int cookie = <value unavailable>
@r12              unsigned int64 psize = 0xe000
<unavailable>     zio_compress compress = <value unavailable>

2: kd> dx -id 0,0,ffff80076609c040 -r1 ((ZFSin!arc_buf_hdr *)0xffff8007a192d190)
((ZFSin!arc_buf_hdr *)0xffff8007a192d190)                 : 0xffff8007a192d190 [Type: arc_buf_hdr *]
    [+0x000] b_dva            [Type: dva]
    [+0x010] b_birth          : 0x0 [Type: unsigned __int64]
    [+0x018] b_type           : ARC_BUFC_METADATA (2) [Type: arc_buf_contents]
    [+0x01c] b_complevel      : 0x0 [Type: unsigned char]
    [+0x01d] b_reserved1      : 0x0 [Type: unsigned char]
    [+0x01e] b_reserved2      : 0x0 [Type: unsigned short]
    [+0x020] b_hash_next      : 0x0 [Type: arc_buf_hdr *]
    [+0x028] b_flags          : ARC_FLAG_L2CACHE | ARC_FLAG_IO_IN_PROGRESS | ARC_FLAG_INDIRECT | ARC_FLAG_BUFC_METADATA | ARC_FLAG_HAS_L1HDR | ARC_FLAG_COMPRESSED_ARC | ARC_FLAG_COMPRESS_0 | ARC_FLAG_COMPRESS_1 | ARC_FLAG_COMPRESS_2 | ARC_FLAG_COMPRESS_3 (253101328) [Type: arc_flags]
    [+0x02c] b_psize          : 0x70 [Type: unsigned short]
    [+0x02e] b_lsize          : 0x100 [Type: unsigned short]
    [+0x030] b_spa            : 0x56950c432ecbb08a [Type: unsigned __int64]
    [+0x038] b_l2hdr          [Type: l2arc_buf_hdr]
    [+0x060] b_l1hdr          [Type: l1arc_buf_hdr]
    [+0x138] b_crypt_hdr      [Type: arc_buf_hdr_crypt]

2: kd> dx -id 0,0,ffff80076609c040 -r1 (*((ZFSin!l1arc_buf_hdr *)0xffff8007a192d1f0))
(*((ZFSin!l1arc_buf_hdr *)0xffff8007a192d1f0))                 [Type: l1arc_buf_hdr]
    [+0x000] b_freeze_lock    [Type: kmutex]
    [+0x028] b_freeze_cksum   : 0x0 [Type: zio_cksum *]
    [+0x030] b_buf            : 0xffff800809b14ea8 [Type: arc_buf *]
    [+0x038] b_bufcnt         : 0x1 [Type: unsigned int]
    [+0x040] b_cv             [Type: cv]
    [+0x080] b_byteswap       : 0xa [Type: unsigned char]
    [+0x088] b_state          : 0xfffff8043ccf4c80 [Type: arc_state *]
    [+0x090] b_arc_node       [Type: list_node]
    [+0x0a0] b_arc_access     : 0x0 [Type: unsigned __int64]
    [+0x0a8] b_mru_hits       : 0x0 [Type: unsigned int]
    [+0x0ac] b_mru_ghost_hits : 0x0 [Type: unsigned int]
    [+0x0b0] b_mfu_hits       : 0x0 [Type: unsigned int]
    [+0x0b4] b_mfu_ghost_hits : 0x0 [Type: unsigned int]
    [+0x0b8] b_l2_hits        : 0x0 [Type: unsigned int]
    [+0x0c0] b_refcnt         [Type: refcount]
    [+0x0c8] b_acb            : 0x0 [Type: arc_callback *]
    [+0x0d0] b_pabd           : 0xffff8007bd152a00 [Type: abd *]

2: kd> dx -id 0,0,ffff80076609c040 -r1 ((ZFSin!abd *)0xffff8007bd152a00)
((ZFSin!abd *)0xffff8007bd152a00)                 : 0xffff8007bd152a00 [Type: abd *]
    [+0x000] abd_flags        : ABD_FLAG_OWNER | ABD_FLAG_META | ABD_FLAG_ALLOCD (518) [Type: abd_flags]
    [+0x004] abd_size         : 0xe000 [Type: unsigned int]
    [+0x008] abd_gang_link    [Type: list_node]
    [+0x018] abd_mtx          [Type: kmutex]
    [+0x040] abd_u            [Type: abd::<unnamed-tag>]

2: kd> dx -id 0,0,ffff80076609c040 -r1 (*((ZFSin!abd::<unnamed-tag> *)0xffff8007bd152a40))
(*((ZFSin!abd::<unnamed-tag> *)0xffff8007bd152a40))                 [Type: abd::<unnamed-tag>]
    [+0x000] abd_scatter      [Type: abd_scatter]
    [+0x000] abd_linear       [Type: abd_linear]
    [+0x000] abd_gang         [Type: abd_gang]

2: kd> dx -id 0,0,ffff80076609c040 -r1 (*((ZFSin!abd_scatter *)0xffff8007bd152a40))
(*((ZFSin!abd_scatter *)0xffff8007bd152a40))                 [Type: abd_scatter]
    [+0x000] abd_offset       : 0x0 [Type: unsigned int]
    [+0x004] abd_chunk_size   : 0x1000 [Type: unsigned int]
    [+0x008] abd_chunks       [Type: void * [1]]

2: kd> dx -id 0,0,ffff80076609c040 -r1 (*((ZFSin!void * (*)[1])0xffff8007bd152a48))
(*((ZFSin!void * (*)[1])0xffff8007bd152a48))                 [Type: void * [1]]
    [0]              : 0x0 [Type: void *]
datacore-rm commented 1 year ago 
2: kd> !vm
Paging File Name paged out
  Current:  18874368 Kb  Free Space:  17733468 Kb
  Minimum:  18874368 Kb  Maximum:     52851256 Kb

Physical Memory:          4718349 (   18873396 Kb)
Available Pages:              279 (       1116 Kb)
ResAvail Pages:                -3 (        -12 Kb)

********** Running out of physical memory **********

zfs:0:tunable:zfs_total_memory_limit 3,865,470,566 zfs:0:tunable:zfs_arc_max 3,478,923,509

kd> dt zfsin!segkmem_total_mem_allocated 0xb83f7000=3,091,165,184

arc_write_ready() => arc_hdr_alloc_abd() => arc_get_data_abd()=>abd_alloc()=> abd_alloc_chunks() { ABD_SCATTER(abd).abd_chunks[i][=lookasidelist_cache_alloc(abd_chunk_cache) }

lookasidelist cache returned null. It does not have option like 'KM_SLEEP' to wait till new allocation success. lookasidelist_cache_free() returns the chunk back to the OS using ZFS callback osif_free()

datacore-rm commented 1 year ago

Reverting the ABD chunk cache to kmem_cache(KM_SLEEP) should help to avoid bugcheck in such scenario, correct?

lundman commented 1 year ago

OK great, thanks for letter me know. We can probably just rollback for breathing room and land it again if other options are considered..