rincebrain
commented
2 weeks ago Dupe of #16322, if I had to guess.
Open Dmitrius7 opened 2 weeks ago
rincebrain
commented
2 weeks ago Dupe of #16322, if I had to guess.
Dmitrius7
commented
2 weeks ago @robn Thank you for your help!
you wrote at https://github.com/openzfs/zfs/issues/16322
Could you capture cat /proc/spl/kmem/slab before the import, and at the time of the OOM, and post them?
I did it.
Full log /proc/slabinfo and /proc/meminfo memory_info_2024-07-08_17-03-53.txt
Important note! If I import the pool as read-only, everything works fine, and the RAM is not consumed.
zpool import -o readonly=on -d /dev/disk/by-partlabel/ zp-erp1-hdd
Before import pool zp-erp1-hdd
root@erpband-pve1:~# cat /proc/spl/kmem/slab
--------------------- cache ------------------------------------------------------- ----- slab ------ ---- object ----- --- emergency ---
name flags size alloc slabsize objsize total alloc max total alloc max dlock alloc max
spl_zlib_workspace_cache 0x00080 0 0 2145216 268104 0 0 0 0 0 0 0 0 0
kcf_context_cache 0x00100 - 0 - 40 - - - - 0 - - - -
zfs_btree_leaf_cache 0x00100 - 135168 - 4096 - - - - 33 - - - -
metaslab_alloc_trace_cache 0x00100 - 0 - 72 - - - - 0 - - - -
brt_entry_cache 0x00100 - 0 - 40 - - - - 0 - - - -
brt_pending_entry_cache 0x00100 - 0 - 160 - - - - 0 - - - -
ddt_cache 0x00080 995520 794880 199104 24840 5 4 4 40 32 32 0 0 0
ddt_entry_cache 0x00100 - 0 - 448 - - - - 0 - - - -
zio_cache 0x00100 - 46080 - 1280 - - - - 36 - - - -
zio_link_cache 0x00100 - 0 - 48 - - - - 0 - - - -
zio_buf_comb_512 0x00102 - 4096 - 512 - - - - 8 - - - -
zio_buf_comb_1024 0x00102 - 1024 - 1024 - - - - 1 - - - -
zio_buf_comb_1536 0x00102 - 3072 - 1536 - - - - 2 - - - -
zio_buf_comb_2048 0x00102 - 0 - 2048 - - - - 0 - - - -
zio_buf_comb_2560 0x00102 - 2560 - 2560 - - - - 1 - - - -
zio_buf_comb_3072 0x00102 - 0 - 3072 - - - - 0 - - - -
zio_buf_comb_3584 0x00102 - 0 - 3584 - - - - 0 - - - -
zio_buf_comb_4096 0x00102 - 12288 - 4096 - - - - 3 - - - -
zio_buf_comb_5120 0x00102 - 0 - 5120 - - - - 0 - - - -
zio_buf_comb_6144 0x00102 - 0 - 6144 - - - - 0 - - - -
zio_buf_comb_7168 0x00102 - 0 - 7168 - - - - 0 - - - -
zio_buf_comb_8192 0x00102 - 0 - 8192 - - - - 0 - - - -
zio_buf_comb_10240 0x00102 - 0 - 10240 - - - - 0 - - - -
zio_buf_comb_12288 0x00102 - 0 - 12288 - - - - 0 - - - -
zio_buf_comb_14336 0x00102 - 14336 - 14336 - - - - 1 - - - -
zio_buf_comb_16384 0x00102 - 671744 - 16384 - - - - 41 - - - -
zio_buf_comb_20480 0x00082 401408 163840 200704 20480 2 1 1 16 8 8 0 0 0
zio_buf_comb_24576 0x00082 2334720 1769472 233472 24576 10 9 9 80 72 72 0 0 0
zio_buf_comb_28672 0x00082 532480 229376 266240 28672 2 1 1 16 8 8 0 0 0
zio_buf_comb_32768 0x00082 598016 262144 299008 32768 2 1 1 16 8 8 0 0 0
zio_buf_comb_40960 0x00082 4009984 3276800 364544 40960 11 10 10 88 80 80 0 0 0
zio_buf_comb_49152 0x00082 3870720 3145728 430080 49152 9 8 8 72 64 64 0 0 0
zio_buf_comb_57344 0x00082 3964928 3211264 495616 57344 8 7 7 64 56 56 0 0 0
zio_buf_comb_65536 0x00082 1122304 524288 561152 65536 2 1 1 16 8 8 0 0 0
zio_buf_comb_81920 0x00082 8998912 7864320 692224 81920 13 12 12 104 96 96 0 0 0
zio_buf_comb_98304 0x00082 10702848 9437184 823296 98304 13 12 12 104 96 96 0 0 0
zio_buf_comb_114688 0x00082 8589312 7340032 954368 114688 9 8 8 72 64 64 0 0 0
zio_buf_comb_131072 0x00082 106373120 73400320 1085440 131072 98 98 192 784 560 1536 0 0 0
zio_buf_comb_163840 0x00082 0 0 1347584 163840 0 0 0 0 0 0 0 0 0
zio_buf_comb_196608 0x00082 0 0 1609728 196608 0 0 0 0 0 0 0 0 0
zio_buf_comb_229376 0x00082 0 0 1871872 229376 0 0 0 0 0 0 0 0 0
zio_buf_comb_262144 0x00082 0 0 2134016 262144 0 0 0 0 0 0 0 0 0
zio_buf_comb_327680 0x00082 0 0 2658304 327680 0 0 0 0 0 0 0 0 0
zio_buf_comb_393216 0x00082 0 0 3182592 393216 0 0 0 0 0 0 0 0 0
zio_buf_comb_458752 0x00082 0 0 3706880 458752 0 0 0 0 0 0 0 0 0
zio_buf_comb_524288 0x00082 0 0 4231168 524288 0 0 0 0 0 0 0 0 0
zio_buf_comb_655360 0x00082 0 0 5279744 655360 0 0 0 0 0 0 0 0 0
zio_buf_comb_786432 0x00082 0 0 6328320 786432 0 0 0 0 0 0 0 0 0
zio_buf_comb_917504 0x00082 0 0 7376896 917504 0 0 0 0 0 0 0 0 0
zio_buf_comb_1048576 0x00082 8425472 4194304 8425472 1048576 1 1 1 8 4 4 0 0 0
zio_buf_comb_1310720 0x00082 0 0 10522624 1310720 0 0 0 0 0 0 0 0 0
zio_buf_comb_1572864 0x00082 0 0 12619776 1572864 0 0 0 0 0 0 0 0 0
zio_buf_comb_1835008 0x00082 0 0 14716928 1835008 0 0 0 0 0 0 0 0 0
zio_buf_comb_2097152 0x00082 0 0 16814080 2097152 0 0 0 0 0 0 0 0 0
zio_buf_comb_2621440 0x00082 0 0 21008384 2621440 0 0 0 0 0 0 0 0 0
zio_buf_comb_3145728 0x00082 0 0 25202688 3145728 0 0 0 0 0 0 0 0 0
zio_buf_comb_3670016 0x00082 0 0 29396992 3670016 0 0 0 0 0 0 0 0 0
zio_buf_comb_4194304 0x00082 0 0 29392896 4194304 0 0 0 0 0 0 0 0 0
zio_buf_comb_5242880 0x00082 0 0 31485952 5242880 0 0 0 0 0 0 0 0 0
zio_buf_comb_6291456 0x00082 0 0 31481856 6291456 0 0 0 0 0 0 0 0 0
zio_buf_comb_7340032 0x00082 0 0 29380608 7340032 0 0 0 0 0 0 0 0 0
zio_buf_comb_8388608 0x00082 0 0 25182208 8388608 0 0 0 0 0 0 0 0 0
zio_buf_comb_10485760 0x00082 0 0 31473664 10485760 0 0 0 0 0 0 0 0 0
zio_buf_comb_12582912 0x00082 0 0 25178112 12582912 0 0 0 0 0 0 0 0 0
zio_buf_comb_14680064 0x00082 0 0 29372416 14680064 0 0 0 0 0 0 0 0 0
zio_buf_comb_16777216 0x00082 0 0 16785408 16777216 0 0 0 0 0 0 0 0 0
lz4_cache 0x00100 - 0 - 16384 - - - - 0 - - - -
abd_t 0x00100 - 1531192 - 104 - - - - 14723 - - - -
sa_cache 0x00100 - 3472 - 248 - - - - 14 - - - -
dnode_t 0x00100 - 367032 - 984 - - - - 373 - - - -
arc_buf_hdr_t_full 0x00100 - 3536160 - 240 - - - - 14734 - - - -
arc_buf_hdr_t_l2only 0x00100 - 0 - 96 - - - - 0 - - - -
arc_buf_t 0x00100 - 16640 - 32 - - - - 520 - - - -
dmu_buf_impl_t 0x00100 - 299880 - 392 - - - - 765 - - - -
zil_lwb_cache 0x00100 - 0 - 408 - - - - 0 - - - -
zil_zcw_cache 0x00100 - 0 - 152 - - - - 0 - - - -
sio_cache_0 0x00100 - 0 - 136 - - - - 0 - - - -
sio_cache_1 0x00100 - 0 - 152 - - - - 0 - - - -
sio_cache_2 0x00100 - 0 - 168 - - - - 0 - - - -
zfs_znode_cache 0x00100 - 18176 - 1136 - - - - 16 - - - -
zfs_znode_hold_cache 0x00100 - 0 - 88 - - - - 0 - - - -
import pool zp-erp1-hdd
root@erpband-pve1:~# zpool import -d /dev/disk/by-partlabel/ zp-erp1-hdd
root@erpband-pve1:~# cat /proc/spl/kmem/slab
--------------------- cache ------------------------------------------------------- ----- slab ------ ---- object ----- --- emergency ---
name flags size alloc slabsize objsize total alloc max total alloc max dlock alloc max
spl_zlib_workspace_cache 0x00080 0 0 2145216 268104 0 0 0 0 0 0 0 0 0
kcf_context_cache 0x00100 - 0 - 40 - - - - 0 - - - -
zfs_btree_leaf_cache 0x00100 - 33320960 - 4096 - - - - 8135 - - - -
metaslab_alloc_trace_cache 0x00100 - 0 - 72 - - - - 0 - - - -
brt_entry_cache 0x00100 - 0 - 40 - - - - 0 - - - -
brt_pending_entry_cache 0x00100 - 0 - 160 - - - - 0 - - - -
ddt_cache 0x00080 1791936 1589760 199104 24840 9 8 8 72 64 64 0 0 0
ddt_entry_cache 0x00100 - 41959680 - 448 - - - - 93660 - - - -
zio_cache 0x00100 - 1708556800 - 1280 - - - - 1334810 - - - -
zio_link_cache 0x00100 - 64067040 - 48 - - - - 1334730 - - - -
zio_buf_comb_512 0x00102 - 28160 - 512 - - - - 55 - - - -
zio_buf_comb_1024 0x00102 - 0 - 1024 - - - - 0 - - - -
zio_buf_comb_1536 0x00102 - 6144 - 1536 - - - - 4 - - - -
zio_buf_comb_2048 0x00102 - 0 - 2048 - - - - 0 - - - -
zio_buf_comb_2560 0x00102 - 5120 - 2560 - - - - 2 - - - -
zio_buf_comb_3072 0x00102 - 0 - 3072 - - - - 0 - - - -
zio_buf_comb_3584 0x00102 - 0 - 3584 - - - - 0 - - - -
zio_buf_comb_4096 0x00102 - 32768 - 4096 - - - - 8 - - - -
zio_buf_comb_5120 0x00102 - 0 - 5120 - - - - 0 - - - -
zio_buf_comb_6144 0x00102 - 0 - 6144 - - - - 0 - - - -
zio_buf_comb_7168 0x00102 - 0 - 7168 - - - - 0 - - - -
zio_buf_comb_8192 0x00102 - 0 - 8192 - - - - 0 - - - -
zio_buf_comb_10240 0x00102 - 0 - 10240 - - - - 0 - - - -
zio_buf_comb_12288 0x00102 - 0 - 12288 - - - - 0 - - - -
zio_buf_comb_14336 0x00102 - 14336 - 14336 - - - - 1 - - - -
zio_buf_comb_16384 0x00102 - 18563072 - 16384 - - - - 1133 - - - -
zio_buf_comb_20480 0x00082 401408 163840 200704 20480 2 1 1 16 8 8 0 0 0
zio_buf_comb_24576 0x00082 3035136 2359296 233472 24576 13 12 12 104 96 96 0 0 0
zio_buf_comb_28672 0x00082 532480 229376 266240 28672 2 1 1 16 8 8 0 0 0
zio_buf_comb_32768 0x00082 118407168 99614720 299008 32768 396 396 398 3168 3040 3184 0 0 0
zio_buf_comb_40960 0x00082 4009984 3276800 364544 40960 11 10 10 88 80 80 0 0 0
zio_buf_comb_49152 0x00082 5591040 4718592 430080 49152 13 12 12 104 96 96 0 0 0
zio_buf_comb_57344 0x00082 6443008 5505024 495616 57344 13 12 12 104 96 96 0 0 0
zio_buf_comb_65536 0x00082 1122304 524288 561152 65536 2 1 1 16 8 8 0 0 0
zio_buf_comb_81920 0x00082 8998912 7864320 692224 81920 13 12 12 104 96 96 0 0 0
zio_buf_comb_98304 0x00082 10702848 9437184 823296 98304 13 12 12 104 96 96 0 0 0
zio_buf_comb_114688 0x00082 20996096 19267584 954368 114688 22 21 21 176 168 168 0 0 0
zio_buf_comb_131072 0x00082 74895360 16777216 1085440 131072 69 69 636 552 128 5088 0 0 0
zio_buf_comb_163840 0x00082 0 0 1347584 163840 0 0 0 0 0 0 0 0 0
zio_buf_comb_196608 0x00082 0 0 1609728 196608 0 0 0 0 0 0 0 0 0
zio_buf_comb_229376 0x00082 0 0 1871872 229376 0 0 0 0 0 0 0 0 0
zio_buf_comb_262144 0x00082 0 0 2134016 262144 0 0 0 0 0 0 0 0 0
zio_buf_comb_327680 0x00082 0 0 2658304 327680 0 0 0 0 0 0 0 0 0
zio_buf_comb_393216 0x00082 0 0 3182592 393216 0 0 0 0 0 0 0 0 0
zio_buf_comb_458752 0x00082 0 0 3706880 458752 0 0 0 0 0 0 0 0 0
zio_buf_comb_524288 0x00082 0 0 4231168 524288 0 0 0 0 0 0 0 0 0
zio_buf_comb_655360 0x00082 0 0 5279744 655360 0 0 0 0 0 0 0 0 0
zio_buf_comb_786432 0x00082 0 0 6328320 786432 0 0 0 0 0 0 0 0 0
zio_buf_comb_917504 0x00082 0 0 7376896 917504 0 0 0 0 0 0 0 0 0
zio_buf_comb_1048576 0x00082 8425472 4194304 8425472 1048576 1 1 1 8 4 4 0 0 0
zio_buf_comb_1310720 0x00082 0 0 10522624 1310720 0 0 0 0 0 0 0 0 0
zio_buf_comb_1572864 0x00082 0 0 12619776 1572864 0 0 0 0 0 0 0 0 0
zio_buf_comb_1835008 0x00082 0 0 14716928 1835008 0 0 0 0 0 0 0 0 0
zio_buf_comb_2097152 0x00082 0 0 16814080 2097152 0 0 0 0 0 0 0 0 0
zio_buf_comb_2621440 0x00082 0 0 21008384 2621440 0 0 0 0 0 0 0 0 0
zio_buf_comb_3145728 0x00082 0 0 25202688 3145728 0 0 0 0 0 0 0 0 0
zio_buf_comb_3670016 0x00082 0 0 29396992 3670016 0 0 0 0 0 0 0 0 0
zio_buf_comb_4194304 0x00082 0 0 29392896 4194304 0 0 0 0 0 0 0 0 0
zio_buf_comb_5242880 0x00082 0 0 31485952 5242880 0 0 0 0 0 0 0 0 0
zio_buf_comb_6291456 0x00082 0 0 31481856 6291456 0 0 0 0 0 0 0 0 0
zio_buf_comb_7340032 0x00082 0 0 29380608 7340032 0 0 0 0 0 0 0 0 0
zio_buf_comb_8388608 0x00082 0 0 25182208 8388608 0 0 0 0 0 0 0 0 0
zio_buf_comb_10485760 0x00082 0 0 31473664 10485760 0 0 0 0 0 0 0 0 0
zio_buf_comb_12582912 0x00082 0 0 25178112 12582912 0 0 0 0 0 0 0 0 0
zio_buf_comb_14680064 0x00082 0 0 29372416 14680064 0 0 0 0 0 0 0 0 0
zio_buf_comb_16777216 0x00082 0 0 16785408 16777216 0 0 0 0 0 0 0 0 0
lz4_cache 0x00100 - 0 - 16384 - - - - 0 - - - -
abd_t 0x00100 - 5090176 - 104 - - - - 48944 - - - -
sa_cache 0x00100 - 2976 - 248 - - - - 12 - - - -
dnode_t 0x00100 - 3540432 - 984 - - - - 3598 - - - -
arc_buf_hdr_t_full 0x00100 - 20352720 - 240 - - - - 84803 - - - -
arc_buf_hdr_t_l2only 0x00100 - 0 - 96 - - - - 0 - - - -
arc_buf_t 0x00100 - 131168 - 32 - - - - 4099 - - - -
dmu_buf_impl_t 0x00100 - 2934120 - 392 - - - - 7485 - - - -
zil_lwb_cache 0x00100 - 0 - 408 - - - - 0 - - - -
zil_zcw_cache 0x00100 - 0 - 152 - - - - 0 - - - -
sio_cache_0 0x00100 - 0 - 136 - - - - 0 - - - -
sio_cache_1 0x00100 - 0 - 152 - - - - 0 - - - -
sio_cache_2 0x00100 - 0 - 168 - - - - 0 - - - -
zfs_znode_cache 0x00100 - 20448 - 1136 - - - - 18 - - - -
zfs_znode_hold_cache 0x00100 - 0 - 88 - - - - 0 - - - -
3.3 GiB - ARC size (current) 3.0 GiB - Anonymous metadata size
root@erpband-pve1:~# 3gb anon
-bash: 3gb: command not found
root@erpband-pve1:~#
root@erpband-pve1:~# cat /proc/spl/kmem/slab
--------------------- cache ------------------------------------------------------- ----- slab ------ ---- object ----- --- emergency ---
name flags size alloc slabsize objsize total alloc max total alloc max dlock alloc max
spl_zlib_workspace_cache 0x00080 0 0 2145216 268104 0 0 0 0 0 0 0 0 0
kcf_context_cache 0x00100 - 0 - 40 - - - - 0 - - - -
zfs_btree_leaf_cache 0x00100 - 34217984 - 4096 - - - - 8354 - - - -
metaslab_alloc_trace_cache 0x00100 - 0 - 72 - - - - 0 - - - -
brt_entry_cache 0x00100 - 0 - 40 - - - - 0 - - - -
brt_pending_entry_cache 0x00100 - 0 - 160 - - - - 0 - - - -
ddt_cache 0x00080 1791936 1589760 199104 24840 9 8 8 72 64 64 0 0 0
ddt_entry_cache 0x00100 - 2141819904 - 448 - - - - 4780848 - - - -
zio_cache 0x00100 - 98560 - 1280 - - - - 77 - - - -
zio_link_cache 0x00100 - 144 - 48 - - - - 3 - - - -
zio_buf_comb_512 0x00102 - 28160 - 512 - - - - 55 - - - -
zio_buf_comb_1024 0x00102 - 0 - 1024 - - - - 0 - - - -
zio_buf_comb_1536 0x00102 - 6144 - 1536 - - - - 4 - - - -
zio_buf_comb_2048 0x00102 - 0 - 2048 - - - - 0 - - - -
zio_buf_comb_2560 0x00102 - 5120 - 2560 - - - - 2 - - - -
zio_buf_comb_3072 0x00102 - 0 - 3072 - - - - 0 - - - -
zio_buf_comb_3584 0x00102 - 0 - 3584 - - - - 0 - - - -
zio_buf_comb_4096 0x00102 - 32768 - 4096 - - - - 8 - - - -
zio_buf_comb_5120 0x00102 - 0 - 5120 - - - - 0 - - - -
zio_buf_comb_6144 0x00102 - 0 - 6144 - - - - 0 - - - -
zio_buf_comb_7168 0x00102 - 0 - 7168 - - - - 0 - - - -
zio_buf_comb_8192 0x00102 - 0 - 8192 - - - - 0 - - - -
zio_buf_comb_10240 0x00102 - 0 - 10240 - - - - 0 - - - -
zio_buf_comb_12288 0x00102 - 0 - 12288 - - - - 0 - - - -
zio_buf_comb_14336 0x00102 - 14336 - 14336 - - - - 1 - - - -
zio_buf_comb_16384 0x00102 - 18563072 - 16384 - - - - 1133 - - - -
zio_buf_comb_20480 0x00082 401408 163840 200704 20480 2 1 1 16 8 8 0 0 0
zio_buf_comb_24576 0x00082 3035136 2359296 233472 24576 13 12 12 104 96 96 0 0 0
zio_buf_comb_28672 0x00082 532480 229376 266240 28672 2 1 1 16 8 8 0 0 0
zio_buf_comb_32768 0x00082 3045695488 2669936640 299008 32768 10186 10185 10185 81488 81480 81480 0 0 0
zio_buf_comb_40960 0x00082 4009984 3276800 364544 40960 11 10 10 88 80 80 0 0 0
zio_buf_comb_49152 0x00082 5591040 4718592 430080 49152 13 12 12 104 96 96 0 0 0
zio_buf_comb_57344 0x00082 6443008 5505024 495616 57344 13 12 12 104 96 96 0 0 0
zio_buf_comb_65536 0x00082 1122304 524288 561152 65536 2 1 1 16 8 8 0 0 0
zio_buf_comb_81920 0x00082 8998912 7864320 692224 81920 13 12 12 104 96 96 0 0 0
zio_buf_comb_98304 0x00082 10702848 9437184 823296 98304 13 12 12 104 96 96 0 0 0
zio_buf_comb_114688 0x00082 20996096 19267584 954368 114688 22 21 21 176 168 168 0 0 0
zio_buf_comb_131072 0x00082 74895360 16777216 1085440 131072 69 69 636 552 128 5088 0 0 0
zio_buf_comb_163840 0x00082 0 0 1347584 163840 0 0 0 0 0 0 0 0 0
zio_buf_comb_196608 0x00082 0 0 1609728 196608 0 0 0 0 0 0 0 0 0
zio_buf_comb_229376 0x00082 0 0 1871872 229376 0 0 0 0 0 0 0 0 0
zio_buf_comb_262144 0x00082 0 0 2134016 262144 0 0 0 0 0 0 0 0 0
zio_buf_comb_327680 0x00082 0 0 2658304 327680 0 0 0 0 0 0 0 0 0
zio_buf_comb_393216 0x00082 0 0 3182592 393216 0 0 0 0 0 0 0 0 0
zio_buf_comb_458752 0x00082 0 0 3706880 458752 0 0 0 0 0 0 0 0 0
zio_buf_comb_524288 0x00082 0 0 4231168 524288 0 0 0 0 0 0 0 0 0
zio_buf_comb_655360 0x00082 0 0 5279744 655360 0 0 0 0 0 0 0 0 0
zio_buf_comb_786432 0x00082 0 0 6328320 786432 0 0 0 0 0 0 0 0 0
zio_buf_comb_917504 0x00082 0 0 7376896 917504 0 0 0 0 0 0 0 0 0
zio_buf_comb_1048576 0x00082 8425472 4194304 8425472 1048576 1 1 1 8 4 4 0 0 0
zio_buf_comb_1310720 0x00082 0 0 10522624 1310720 0 0 0 0 0 0 0 0 0
zio_buf_comb_1572864 0x00082 0 0 12619776 1572864 0 0 0 0 0 0 0 0 0
zio_buf_comb_1835008 0x00082 0 0 14716928 1835008 0 0 0 0 0 0 0 0 0
zio_buf_comb_2097152 0x00082 0 0 16814080 2097152 0 0 0 0 0 0 0 0 0
zio_buf_comb_2621440 0x00082 0 0 21008384 2621440 0 0 0 0 0 0 0 0 0
zio_buf_comb_3145728 0x00082 0 0 25202688 3145728 0 0 0 0 0 0 0 0 0
zio_buf_comb_3670016 0x00082 0 0 29396992 3670016 0 0 0 0 0 0 0 0 0
zio_buf_comb_4194304 0x00082 0 0 29392896 4194304 0 0 0 0 0 0 0 0 0
zio_buf_comb_5242880 0x00082 0 0 31485952 5242880 0 0 0 0 0 0 0 0 0
zio_buf_comb_6291456 0x00082 0 0 31481856 6291456 0 0 0 0 0 0 0 0 0
zio_buf_comb_7340032 0x00082 0 0 29380608 7340032 0 0 0 0 0 0 0 0 0
zio_buf_comb_8388608 0x00082 0 0 25182208 8388608 0 0 0 0 0 0 0 0 0
zio_buf_comb_10485760 0x00082 0 0 31473664 10485760 0 0 0 0 0 0 0 0 0
zio_buf_comb_12582912 0x00082 0 0 25178112 12582912 0 0 0 0 0 0 0 0 0
zio_buf_comb_14680064 0x00082 0 0 29372416 14680064 0 0 0 0 0 0 0 0 0
zio_buf_comb_16777216 0x00082 0 0 16785408 16777216 0 0 0 0 0 0 0 0 0
lz4_cache 0x00100 - 0 - 16384 - - - - 0 - - - -
abd_t 0x00100 - 8599240 - 104 - - - - 82685 - - - -
sa_cache 0x00100 - 2976 - 248 - - - - 12 - - - -
dnode_t 0x00100 - 3540432 - 984 - - - - 3598 - - - -
arc_buf_hdr_t_full 0x00100 - 29341680 - 240 - - - - 122257 - - - -
arc_buf_hdr_t_l2only 0x00100 - 0 - 96 - - - - 0 - - - -
arc_buf_t 0x00100 - 2645856 - 32 - - - - 82683 - - - -
dmu_buf_impl_t 0x00100 - 33736696 - 392 - - - - 86063 - - - -
zil_lwb_cache 0x00100 - 0 - 408 - - - - 0 - - - -
zil_zcw_cache 0x00100 - 0 - 152 - - - - 0 - - - -
sio_cache_0 0x00100 - 0 - 136 - - - - 0 - - - -
sio_cache_1 0x00100 - 0 - 152 - - - - 0 - - - -
sio_cache_2 0x00100 - 0 - 168 - - - - 0 - - - -
zfs_znode_cache 0x00100 - 20448 - 1136 - - - - 18 - - - -
zfs_znode_hold_cache 0x00100 - 0 - 88 - - - - 0 - - - -20.6 GiB - ARC size (current) 20.1 GiB - Anonymous metadata size
root@erpband-pve1:~# cat /proc/spl/kmem/slab
--------------------- cache ------------------------------------------------------- ----- slab ------ ---- object ----- --- emergency ---
name flags size alloc slabsize objsize total alloc max total alloc max dlock alloc max
spl_zlib_workspace_cache 0x00080 0 0 2145216 268104 0 0 0 0 0 0 0 0 0
kcf_context_cache 0x00100 - 0 - 40 - - - - 0 - - - -
zfs_btree_leaf_cache 0x00100 - 40005632 - 4096 - - - - 9767 - - - -
metaslab_alloc_trace_cache 0x00100 - 0 - 72 - - - - 0 - - - -
brt_entry_cache 0x00100 - 0 - 40 - - - - 0 - - - -
brt_pending_entry_cache 0x00100 - 0 - 160 - - - - 0 - - - -
ddt_cache 0x00080 1791936 1589760 199104 24840 9 8 8 72 64 64 0 0 0
ddt_entry_cache 0x00100 - 1925172480 - 448 - - - - 4297260 - - - -
zio_cache 0x00100 - 98560 - 1280 - - - - 77 - - - -
zio_link_cache 0x00100 - 144 - 48 - - - - 3 - - - -
zio_buf_comb_512 0x00102 - 28160 - 512 - - - - 55 - - - -
zio_buf_comb_1024 0x00102 - 0 - 1024 - - - - 0 - - - -
zio_buf_comb_1536 0x00102 - 6144 - 1536 - - - - 4 - - - -
zio_buf_comb_2048 0x00102 - 0 - 2048 - - - - 0 - - - -
zio_buf_comb_2560 0x00102 - 5120 - 2560 - - - - 2 - - - -
zio_buf_comb_3072 0x00102 - 0 - 3072 - - - - 0 - - - -
zio_buf_comb_3584 0x00102 - 0 - 3584 - - - - 0 - - - -
zio_buf_comb_4096 0x00102 - 32768 - 4096 - - - - 8 - - - -
zio_buf_comb_5120 0x00102 - 0 - 5120 - - - - 0 - - - -
zio_buf_comb_6144 0x00102 - 0 - 6144 - - - - 0 - - - -
zio_buf_comb_7168 0x00102 - 0 - 7168 - - - - 0 - - - -
zio_buf_comb_8192 0x00102 - 0 - 8192 - - - - 0 - - - -
zio_buf_comb_10240 0x00102 - 0 - 10240 - - - - 0 - - - -
zio_buf_comb_12288 0x00102 - 0 - 12288 - - - - 0 - - - -
zio_buf_comb_14336 0x00102 - 14336 - 14336 - - - - 1 - - - -
zio_buf_comb_16384 0x00102 - 18563072 - 16384 - - - - 1133 - - - -
zio_buf_comb_20480 0x00082 401408 163840 200704 20480 2 1 1 16 8 8 0 0 0
zio_buf_comb_24576 0x00082 3035136 2359296 233472 24576 13 12 12 104 96 96 0 0 0
zio_buf_comb_28672 0x00082 532480 229376 266240 28672 2 1 1 16 8 8 0 0 0
zio_buf_comb_32768 0x00082 16854183936 14776008704 299008 32768 56367 56366 56366 450936 450928 450928 0 0 0
zio_buf_comb_40960 0x00082 4009984 3276800 364544 40960 11 10 10 88 80 80 0 0 0
zio_buf_comb_49152 0x00082 5591040 4718592 430080 49152 13 12 12 104 96 96 0 0 0
zio_buf_comb_57344 0x00082 6443008 5505024 495616 57344 13 12 12 104 96 96 0 0 0
zio_buf_comb_65536 0x00082 1122304 524288 561152 65536 2 1 1 16 8 8 0 0 0
zio_buf_comb_81920 0x00082 8998912 7864320 692224 81920 13 12 12 104 96 96 0 0 0
zio_buf_comb_98304 0x00082 10702848 9437184 823296 98304 13 12 12 104 96 96 0 0 0
zio_buf_comb_114688 0x00082 20996096 19267584 954368 114688 22 21 21 176 168 168 0 0 0
zio_buf_comb_131072 0x00082 74895360 16777216 1085440 131072 69 69 636 552 128 5088 0 0 0
zio_buf_comb_163840 0x00082 0 0 1347584 163840 0 0 0 0 0 0 0 0 0
zio_buf_comb_196608 0x00082 0 0 1609728 196608 0 0 0 0 0 0 0 0 0
zio_buf_comb_229376 0x00082 0 0 1871872 229376 0 0 0 0 0 0 0 0 0
zio_buf_comb_262144 0x00082 0 0 2134016 262144 0 0 0 0 0 0 0 0 0
zio_buf_comb_327680 0x00082 0 0 2658304 327680 0 0 0 0 0 0 0 0 0
zio_buf_comb_393216 0x00082 0 0 3182592 393216 0 0 0 0 0 0 0 0 0
zio_buf_comb_458752 0x00082 0 0 3706880 458752 0 0 0 0 0 0 0 0 0
zio_buf_comb_524288 0x00082 0 0 4231168 524288 0 0 0 0 0 0 0 0 0
zio_buf_comb_655360 0x00082 0 0 5279744 655360 0 0 0 0 0 0 0 0 0
zio_buf_comb_786432 0x00082 0 0 6328320 786432 0 0 0 0 0 0 0 0 0
zio_buf_comb_917504 0x00082 0 0 7376896 917504 0 0 0 0 0 0 0 0 0
zio_buf_comb_1048576 0x00082 8425472 4194304 8425472 1048576 1 1 1 8 4 4 0 0 0
zio_buf_comb_1310720 0x00082 0 0 10522624 1310720 0 0 0 0 0 0 0 0 0
zio_buf_comb_1572864 0x00082 0 0 12619776 1572864 0 0 0 0 0 0 0 0 0
zio_buf_comb_1835008 0x00082 0 0 14716928 1835008 0 0 0 0 0 0 0 0 0
zio_buf_comb_2097152 0x00082 0 0 16814080 2097152 0 0 0 0 0 0 0 0 0
zio_buf_comb_2621440 0x00082 0 0 21008384 2621440 0 0 0 0 0 0 0 0 0
zio_buf_comb_3145728 0x00082 0 0 25202688 3145728 0 0 0 0 0 0 0 0 0
zio_buf_comb_3670016 0x00082 0 0 29396992 3670016 0 0 0 0 0 0 0 0 0
zio_buf_comb_4194304 0x00082 0 0 29392896 4194304 0 0 0 0 0 0 0 0 0
zio_buf_comb_5242880 0x00082 0 0 31485952 5242880 0 0 0 0 0 0 0 0 0
zio_buf_comb_6291456 0x00082 0 0 31481856 6291456 0 0 0 0 0 0 0 0 0
zio_buf_comb_7340032 0x00082 0 0 29380608 7340032 0 0 0 0 0 0 0 0 0
zio_buf_comb_8388608 0x00082 0 0 25182208 8388608 0 0 0 0 0 0 0 0 0
zio_buf_comb_10485760 0x00082 0 0 31473664 10485760 0 0 0 0 0 0 0 0 0
zio_buf_comb_12582912 0x00082 0 0 25178112 12582912 0 0 0 0 0 0 0 0 0
zio_buf_comb_14680064 0x00082 0 0 29372416 14680064 0 0 0 0 0 0 0 0 0
zio_buf_comb_16777216 0x00082 0 0 16785408 16777216 0 0 0 0 0 0 0 0 0
lz4_cache 0x00100 - 0 - 16384 - - - - 0 - - - -
abd_t 0x00100 - 47021936 - 104 - - - - 452134 - - - -
sa_cache 0x00100 - 2976 - 248 - - - - 12 - - - -
dnode_t 0x00100 - 3540432 - 984 - - - - 3598 - - - -
arc_buf_hdr_t_full 0x00100 - 116441760 - 240 - - - - 485174 - - - -
arc_buf_hdr_t_l2only 0x00100 - 0 - 96 - - - - 0 - - - -
arc_buf_t 0x00100 - 14468224 - 32 - - - - 452132 - - - -
dmu_buf_impl_t 0x00100 - 178560704 - 392 - - - - 455512 - - - -
zil_lwb_cache 0x00100 - 0 - 408 - - - - 0 - - - -
zil_zcw_cache 0x00100 - 0 - 152 - - - - 0 - - - -
sio_cache_0 0x00100 - 0 - 136 - - - - 0 - - - -
sio_cache_1 0x00100 - 0 - 152 - - - - 0 - - - -
sio_cache_2 0x00100 - 0 - 168 - - - - 0 - - - -
zfs_znode_cache 0x00100 - 20448 - 1136 - - - - 18 - - - -
zfs_znode_hold_cache 0x00100 - 0 - 88 - - - - 0 - - - -51.1 GiB - ARC size (current) 50.2 GiB - Anonymous metadata size
root@erpband-pve1:~# cat /proc/spl/kmem/slab
--------------------- cache ------------------------------------------------------- ----- slab ------ ---- object ----- --- emergency ---
name flags size alloc slabsize objsize total alloc max total alloc max dlock alloc max
spl_zlib_workspace_cache 0x00080 0 0 2145216 268104 0 0 0 0 0 0 0 0 0
kcf_context_cache 0x00100 - 0 - 40 - - - - 0 - - - -
zfs_btree_leaf_cache 0x00100 - 47239168 - 4096 - - - - 11533 - - - -
metaslab_alloc_trace_cache 0x00100 - 0 - 72 - - - - 0 - - - -
brt_entry_cache 0x00100 - 0 - 40 - - - - 0 - - - -
brt_pending_entry_cache 0x00100 - 0 - 160 - - - - 0 - - - -
ddt_cache 0x00080 1791936 1589760 199104 24840 9 8 8 72 64 64 0 0 0
ddt_entry_cache 0x00100 - 857299072 - 448 - - - - 1913614 - - - -
zio_cache 0x00100 - 93440 - 1280 - - - - 73 - - - -
zio_link_cache 0x00100 - 0 - 48 - - - - 0 - - - -
zio_buf_comb_512 0x00102 - 28160 - 512 - - - - 55 - - - -
zio_buf_comb_1024 0x00102 - 0 - 1024 - - - - 0 - - - -
zio_buf_comb_1536 0x00102 - 6144 - 1536 - - - - 4 - - - -
zio_buf_comb_2048 0x00102 - 0 - 2048 - - - - 0 - - - -
zio_buf_comb_2560 0x00102 - 5120 - 2560 - - - - 2 - - - -
zio_buf_comb_3072 0x00102 - 0 - 3072 - - - - 0 - - - -
zio_buf_comb_3584 0x00102 - 0 - 3584 - - - - 0 - - - -
zio_buf_comb_4096 0x00102 - 32768 - 4096 - - - - 8 - - - -
zio_buf_comb_5120 0x00102 - 0 - 5120 - - - - 0 - - - -
zio_buf_comb_6144 0x00102 - 0 - 6144 - - - - 0 - - - -
zio_buf_comb_7168 0x00102 - 0 - 7168 - - - - 0 - - - -
zio_buf_comb_8192 0x00102 - 0 - 8192 - - - - 0 - - - -
zio_buf_comb_10240 0x00102 - 0 - 10240 - - - - 0 - - - -
zio_buf_comb_12288 0x00102 - 0 - 12288 - - - - 0 - - - -
zio_buf_comb_14336 0x00102 - 14336 - 14336 - - - - 1 - - - -
zio_buf_comb_16384 0x00102 - 18563072 - 16384 - - - - 1133 - - - -
zio_buf_comb_20480 0x00082 200704 163840 200704 20480 1 1 1 8 8 8 0 0 0
zio_buf_comb_24576 0x00082 2568192 2162688 233472 24576 11 11 12 88 88 96 0 0 0
zio_buf_comb_28672 0x00082 266240 229376 266240 28672 1 1 1 8 8 8 0 0 0
zio_buf_comb_32768 0x00082 38581002240 33824178176 299008 32768 129030 129029 129029 1032240 1032232 1032232 0 0 0
zio_buf_comb_40960 0x00082 3280896 2908160 364544 40960 9 9 10 72 71 80 0 0 0
zio_buf_comb_49152 0x00082 4730880 4325376 430080 49152 11 11 12 88 88 96 0 0 0
zio_buf_comb_57344 0x00082 5451776 5046272 495616 57344 11 11 12 88 88 96 0 0 0
zio_buf_comb_65536 0x00082 561152 524288 561152 65536 1 1 1 8 8 8 0 0 0
zio_buf_comb_81920 0x00082 7614464 7208960 692224 81920 11 11 12 88 88 96 0 0 0
zio_buf_comb_98304 0x00082 9056256 8650752 823296 98304 11 11 12 88 88 96 0 0 0
zio_buf_comb_114688 0x00082 17178624 14106624 954368 114688 18 18 21 144 123 168 0 0 0
zio_buf_comb_131072 0x00082 66211840 15073280 1085440 131072 61 61 636 488 115 5088 0 0 0
zio_buf_comb_163840 0x00082 0 0 1347584 163840 0 0 0 0 0 0 0 0 0
zio_buf_comb_196608 0x00082 0 0 1609728 196608 0 0 0 0 0 0 0 0 0
zio_buf_comb_229376 0x00082 0 0 1871872 229376 0 0 0 0 0 0 0 0 0
zio_buf_comb_262144 0x00082 0 0 2134016 262144 0 0 0 0 0 0 0 0 0
zio_buf_comb_327680 0x00082 0 0 2658304 327680 0 0 0 0 0 0 0 0 0
zio_buf_comb_393216 0x00082 0 0 3182592 393216 0 0 0 0 0 0 0 0 0
zio_buf_comb_458752 0x00082 0 0 3706880 458752 0 0 0 0 0 0 0 0 0
zio_buf_comb_524288 0x00082 0 0 4231168 524288 0 0 0 0 0 0 0 0 0
zio_buf_comb_655360 0x00082 0 0 5279744 655360 0 0 0 0 0 0 0 0 0
zio_buf_comb_786432 0x00082 0 0 6328320 786432 0 0 0 0 0 0 0 0 0
zio_buf_comb_917504 0x00082 0 0 7376896 917504 0 0 0 0 0 0 0 0 0
zio_buf_comb_1048576 0x00082 8425472 4194304 8425472 1048576 1 1 1 8 4 4 0 0 0
zio_buf_comb_1310720 0x00082 0 0 10522624 1310720 0 0 0 0 0 0 0 0 0
zio_buf_comb_1572864 0x00082 0 0 12619776 1572864 0 0 0 0 0 0 0 0 0
zio_buf_comb_1835008 0x00082 0 0 14716928 1835008 0 0 0 0 0 0 0 0 0
zio_buf_comb_2097152 0x00082 0 0 16814080 2097152 0 0 0 0 0 0 0 0 0
zio_buf_comb_2621440 0x00082 0 0 21008384 2621440 0 0 0 0 0 0 0 0 0
zio_buf_comb_3145728 0x00082 0 0 25202688 3145728 0 0 0 0 0 0 0 0 0
zio_buf_comb_3670016 0x00082 0 0 29396992 3670016 0 0 0 0 0 0 0 0 0
zio_buf_comb_4194304 0x00082 0 0 29392896 4194304 0 0 0 0 0 0 0 0 0
zio_buf_comb_5242880 0x00082 0 0 31485952 5242880 0 0 0 0 0 0 0 0 0
zio_buf_comb_6291456 0x00082 0 0 31481856 6291456 0 0 0 0 0 0 0 0 0
zio_buf_comb_7340032 0x00082 0 0 29380608 7340032 0 0 0 0 0 0 0 0 0
zio_buf_comb_8388608 0x00082 0 0 25182208 8388608 0 0 0 0 0 0 0 0 0
zio_buf_comb_10485760 0x00082 0 0 31473664 10485760 0 0 0 0 0 0 0 0 0
zio_buf_comb_12582912 0x00082 0 0 25178112 12582912 0 0 0 0 0 0 0 0 0
zio_buf_comb_14680064 0x00082 0 0 29372416 14680064 0 0 0 0 0 0 0 0 0
zio_buf_comb_16777216 0x00082 0 0 16785408 16777216 0 0 0 0 0 0 0 0 0
lz4_cache 0x00100 - 0 - 16384 - - - - 0 - - - -
abd_t 0x00100 - 107477760 - 104 - - - - 1033440 - - - -
sa_cache 0x00100 - 2976 - 248 - - - - 12 - - - -
dnode_t 0x00100 - 3540432 - 984 - - - - 3598 - - - -
arc_buf_hdr_t_full 0x00100 - 252282720 - 240 - - - - 1051178 - - - -
arc_buf_hdr_t_l2only 0x00100 - 0 - 96 - - - - 0 - - - -
arc_buf_t 0x00100 - 33070016 - 32 - - - - 1033438 - - - -
dmu_buf_impl_t 0x00100 - 406432656 - 392 - - - - 1036818 - - - -
zil_lwb_cache 0x00100 - 0 - 408 - - - - 0 - - - -
zil_zcw_cache 0x00100 - 0 - 152 - - - - 0 - - - -
sio_cache_0 0x00100 - 0 - 136 - - - - 0 - - - -
sio_cache_1 0x00100 - 0 - 152 - - - - 0 - - - -
sio_cache_2 0x00100 - 0 - 168 - - - - 0 - - - -
zfs_znode_cache 0x00100 - 20448 - 1136 - - - - 18 - - - -
zfs_znode_hold_cache 0x00100 - 0 - 88 - - - - 0 - - - -
Today I reinstalled proxmox with previous version. It didn' help. The problem is still exist.
Now:
# zfs version
zfs-2.2.3-pve1
zfs-kmod-2.2.3-pve1was before reinstall
# zfs version
zfs-2.2.4-pve1
zfs-kmod-2.2.4-pve1
System information
Describe the problem you're observing
Immediately after importing the pool, the ARC size (current) starts growing and quickly exceeds the Max size (zfs_arc_max). It continues to grow until there is no free RAM left on the server. This causes an Out of memory: killed process. The server has a total of 64GB of RAM.
In arc_summary, I see that the Anonymous metadata size starts growing, and along with it, the ARC size (current) starts growing. When the Anonymous metadata size reaches 50.1 GiB and the Target size (adaptive) becomes 52.2 GiB, there is no free RAM left in the system.
arc_summary | grep -E 'ARC size (current)|Min size (hard limit)|Max size (high water)|Anonymous metadata size' ARC size (current): 2616.7 % 52.3 GiB Min size (hard limit): 50.0 % 1024.0 MiB Max size (high water): 2:1 2.0 GiB Anonymous metadata size: 99.5 % 50.9 GiB
Why is the ARC Max size limit (/sys/module/zfs/parameters/zfs_arc_max) set to 2GB being ignored? How is this possible?
What is Anonymous metadata size? I didn't find any information.
Describe how to reproduce the problem
A complete video demonstrating the problem. https://youtu.be/ygukivJmaKw?si=jF5vZMDQm7FyfUmw&t=436
I import the pool.
root@erpband-pve1:~# zpool import -d /dev/disk/by-partlabel/ zp-erp1-hddAfter a 3-5 minutes, the ARC size starts to grow.
The maximum ARC size is set to 2GB, but it is being ignored.
I wrote a script arc_monitor.txt that outputs the parameters ARC size (current), Anonymous metadata size, Max size (high water), and Min size (hard limit) in a table. It clearly shows how the Anonymous metadata size grows, and along with it, the ARC size (current) grows.
Additional information
Include any warning/errors/backtraces from the system logs
There are no errors or warnings in the logs. The pool imports successfully, but it consumes all the RAM. Here is the log file for the period from the pool import to the system freeze: logs_2024-07-07_18-40-36.txt .