kmalloc-tracer only triggering on kfree

scratchadams commented 2 weeks ago

I've got an issue where the kmalloc-tracer function is only triggering on kfree events. I've tested this with a few kernel versions (6.1, 6.5, and 6.7) and they all have the same results, however I also tested on an earlier version (4.19) and it triggered on both the allocs and frees. I did check to see that the malloc breakpoints were getting added correctly and tested setting a manual breakpoint on __kmalloc which triggered just fine, so not sure what is going on here.

bata24 commented 2 weeks ago

Thank you for your reporting. I have does not appear in 6.5 or 6.9, but it does appear in 6.10. I'll investigate further.

It seems that the breakpoint on about kmalloc is failing. Is this the same in your environment?

6.5

gef> kversion 
[+] Wait for memory scan
0xffffffff9ab7e9a0: Linux version 6.5.11-060511-generic (kernel@kathleen) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.2.0-6ubuntu1) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.41) #202311151304 SMP PREEMPT_DYNAMIC Thu Nov 16 08:20:12 UTC 2023
gef> kmalloc-tracer 
[+] Wait for memory scan
kmalloc_large: Breakpoint 1 at 0xffffffff999d4ad0
kmalloc_trace: Breakpoint 2 at 0xffffffff999d4750
__kmalloc: Breakpoint 3 at 0xffffffff999d5240
kmalloc_node_trace: Breakpoint 4 at 0xffffffff999d4810
__kmalloc_node: Breakpoint 5 at 0xffffffff999d50c0
krealloc: Breakpoint 6 at 0xffffffff999d5820
kfree: Breakpoint 7 at 0xffffffff999d5480
[+] Setup is complete. continuing...

6.9

gef> kversion 
[+] Wait for memory scan
0xffffffff9efa0c20: Linux version 6.9.0-060900rc4-generic (kernel@kathleen) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.2.0-23ubuntu3) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #202404142134 SMP PREEMPT_DYNAMIC Sun Apr 14 21:46:53 UTC 2024
gef> kmalloc-tracer 
[+] Wait for memory scan
kmalloc_large: Breakpoint 8 at 0xffffffff9dc6bbf0
kmalloc_trace: Breakpoint 9 at 0xffffffff9dc717c0
__kmalloc: Breakpoint 10 at 0xffffffff9dc6fb20
kmalloc_node_trace: Breakpoint 11 at 0xffffffff9dc70380
__kmalloc_node: Breakpoint 12 at 0xffffffff9dc70c20
krealloc: Breakpoint 13 at 0xffffffff9dc121a0
kfree: Breakpoint 14 at 0xffffffff9dc6e190
[+] Setup is complete. continuing...

6.10

gef> kversion 
[+] Wait for memory scan
0xffffffff919a5320: Linux version 6.10.0-061000rc2-generic (kernel@kathleen) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.2.0-23ubuntu4) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #202406022333 SMP PREEMPT_DYNAMIC Sun Jun  2 23:43:11 UTC 2024
gef> kmalloc-tracer 
[+] Wait for memory scan
kfree: Breakpoint 1 at 0xffffffff90679690 ★kmalloc not found
[+] Setup is complete. continuing...

scratchadams commented 2 weeks ago

The breakpoints appear to be getting set properly, just not hitting the path to display the task/addr info when the breakpoint is hit. I did modify the code to not continue after setting those breakpoints so I could validate them and they are all set and enabled and resolve to the same addr as when I set them manually.

6.5

gef> kmalloc-tracer 
[+] Wait for memory scan
kmalloc_large: Breakpoint 1 at 0xffffffff8163eae0: file mm/slab_common.c, line 1142.
kmalloc_trace: Breakpoint 2 at 0xffffffff8163ea30: file mm/slab_common.c, line 1075.
__kmalloc: Breakpoint 3 at 0xffffffff8163f1b0: file mm/slab_common.c, line 997.
kmalloc_node_trace: Breakpoint 4 at 0xffffffff8163e980: file mm/slab_common.c, line 1088.
__kmalloc_node: Breakpoint 5 at 0xffffffff8163f040: file mm/slab_common.c, line 991.
krealloc: Breakpoint 6 at 0xffffffff8163f710: file mm/slab_common.c, line 1398.
kfree: Breakpoint 7 at 0xffffffff8163f3e0: file mm/slab_common.c, line 1016.
[+] Setup is complete. continuing...
[task:0xffff8880078b2d00 ksoftirqd/1     ] kfree                         : 0xffff88800cb80200 (size: 0x100  name: kmalloc-192)
[task:0xffff8880078b6900 kworker/1:1     ] kfree                         : 0xffff88800cb80300 (size: 0x100  name: kmalloc-192)
[task:0xffff8880078b2d00 ksoftirqd/1     ] kfree                         : 0xffff88800cb80400 (size: 0x100  name: kmalloc-192)
[task:0xffff8880078b2d00 ksoftirqd/1     ] kfree                         : 0xffff88800cb80500 (size: 0x100  name: kmalloc-192)
[task:0xffff8880078b2d00 ksoftirqd/1     ] kfree                         : 0xffff88800cb80600 (size: 0x100  name: kmalloc-192)
[task:0xffff8880078b2d00 ksoftirqd/1     ] kfree                         : 0xffff88800cb80700 (size: 0x100  name: kmalloc-192)
[task:0xffff8880078b2d00 ksoftirqd/1     ] kfree                         : 0xffff88800cb80800 (size: 0x100  name: kmalloc-192)
[task:0xffff8880078b2d00 ksoftirqd/1     ] kfree                         : 0xffff88800cb80900 (size: 0x100  name: kmalloc-192)
[task:0xffff88800ce00000 kworker/1:2     ] kfree                         : 0xffff88800cb80a00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800b532d00 bash            ] kfree                         : 0xffff88800b75fe00 (size: 0x200  name: kmalloc-256)
[task:0xffff88800b533c00 bash            ] kfree                         : 0xffff88800a882000 (size: 0x800  name: kmalloc-1k)
[task:0xffff88800b533c00 bash            ] kfree                         : 0xffff88800b282700 (size: 0x100  name: kmalloc-192)
[task:0xffff88800a2fda00 in:imklog       ] kfree                         : 0xffff88801283b000 (size: 0x1000 name: kmalloc-2k)
[task:0xffff88800a8f8f00 systemd-journal ] kfree                         : 0xffff888015155880 (size: 0x80   name: kmalloc-rcl-64)
[task:0xffff88800b533c00 bash            ] kfree                         : 0xffff88800ccb5580 (size: 0x80   name: kmalloc-96)
[task:0xffff88800b533c00 bash            ] kfree                         : 0xffff8880080ae500 (size: 0x40   name: kmalloc-32)

bata24 commented 2 weeks ago

Thanks. I found root cause, so I'll try to fix now.

bata24 commented 2 weeks ago

I think it's fixed. Could you please use the latest gef and check if it works?

scratchadams commented 2 weeks ago

So seems to be triggering on __kmalloc_node_track_caller but not the other kmalloc variants. Also seems to have a heavy performance impact now (took about a minute for ls to complete)

gef> kmalloc-tracer 
[+] Wait for memory scan
kmem_cache_alloc: Breakpoint 1 at 0xffffffff8175c9e0: file mm/slub.c, line 3493.
__kmalloc: Breakpoint 2 at 0xffffffff8163f1b0: file mm/slab_common.c, line 997.
kmalloc_large: Breakpoint 3 at 0xffffffff8163eae0: file mm/slab_common.c, line 1142.
kmalloc_trace: Breakpoint 4 at 0xffffffff8163ea30: file mm/slab_common.c, line 1075.
kmem_cache_alloc_node: Breakpoint 5 at 0xffffffff8175c720: file mm/slub.c, line 3514.
kmem_cache_alloc: Breakpoint 6 at 0xffffffff8175c9e0: file mm/slub.c, line 3493.
__kmalloc_node: Breakpoint 7 at 0xffffffff8163f040: file mm/slab_common.c, line 991.
kmalloc_large_node: Breakpoint 8 at 0xffffffff8163eb90: file mm/slab_common.c, line 1152.
kmalloc_node_trace: Breakpoint 9 at 0xffffffff8163e980: file mm/slab_common.c, line 1088.
__kmalloc_node_track_caller: Breakpoint 10 at 0xffffffff8163eed0: file mm/slab_common.c, line 1004.
krealloc: Breakpoint 11 at 0xffffffff8163f710: file mm/slab_common.c, line 1398.
kfree: Breakpoint 12 at 0xffffffff8163f3e0: file mm/slab_common.c, line 1016.
[+] Setup is complete. continuing...
[task:0xffff8880078b2d00 ksoftirqd/1     ] kfree                         : 0xffff88800b317600 (size: 0x100  name: kmalloc-192)
[task:0xffff8880078b2d00 ksoftirqd/1     ] kfree                         : 0xffff88800b317500 (size: 0x100  name: kmalloc-192)
[task:0xffff88800b8c1e00 bash            ] kfree                         : 0xffff88800c7cec00 (size: 0x200  name: kmalloc-256)
[task:0xffff8880078b6900 kworker/1:1     ] kfree                         : 0xffff88800b317900 (size: 0x100  name: kmalloc-192)
[task:0xffff8880078b3c00 kworker/1:0     ] __kmalloc_node_track_caller   : 0xffff8880105c1700 (size: 0x20   name: kmalloc-16)
[task:0xffff8880078b3c00 kworker/1:0     ] kfree                         : 0xffff88800ad1c680 (size: 0x80   name: kmalloc-64)
[task:0xffff888008521e00 in:imklog       ] kfree                         : 0xffff88800ce13000 (size: 0x1000 name: kmalloc-2k)
[task:0xffff8880078b2d00 ksoftirqd/1     ] kfree                         : 0xffff88800b38f100 (size: 0x100  name: kmalloc-192)
[task:0xffff8880078b2d00 ksoftirqd/1     ] kfree                         : 0xffff88800b38f100 (size: 0x100  name: kmalloc-192)
[task:0xffff888007824b00 rcu_preempt     ] kfree                         : 0xffff888010b88400 (size: 0x80   name: kmalloc-rcl-64)
[task:0xffff8880078b3c00 kworker/1:0     ] kfree                         : 0xffff88800b38f000 (size: 0x100  name: kmalloc-192)
[task:0xffff88800b8c3c00 kworker/1:3     ] kfree                         : 0xffff88800b38f300 (size: 0x100  name: kmalloc-192)
[task:0xffff888010f1da00 systemd-journal ] kfree                         : 0xffff88800b8d6880 (size: 0x80   name: kmalloc-96)
[task:0xffff888007398000 systemd         ] kfree                         : 0xffff88800bc84000 (size: 0x2000 name: kmalloc-4k)
[task:0xffff888007398000 systemd         ] kfree                         : 0xffff88800bc80000 (size: 0x2000 name: kmalloc-4k)
[task:0xffff888007398000 systemd         ] kfree                         : 0xffff888007cd4e40 (size: 0x40   name: kmalloc-32)
[task:0xffff8880078b3c00 kworker/1:0     ] kfree                         : 0xffff88800b38f200 (size: 0x100  name: kmalloc-192)
[task:0xffff88800a198f00 kworker/1:1H    ] __kmalloc_node_track_caller   : 0xffff8880105c17c0 (size: 0x20   name: kmalloc-16)
[task:0xffff88800739e900 kworker/0:0H    ] __kmalloc_node_track_caller   : 0xffff8880101e66e0 (size: 0x20   name: kmalloc-16)
[task:0xffff88800a198f00 kworker/1:1H    ] kfree                         : 0xffff88800ad1c600 (size: 0x80   name: kmalloc-64)
[task:0xffff8880078b6900 kworker/1:1     ] kfree                         : 0xffff88800b38fd00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800739e900 kworker/0:0H    ] kfree                         : 0xffff88800a3e3180 (size: 0x80   name: kmalloc-64)
[task:0xffff888010e2ad00 bash            ] kfree                         : 0xffff88801506e800 (size: 0x800  name: kmalloc-1k)
[task:0xffff888010e2ad00 bash            ] kfree                         : 0xffff88800b317d00 (size: 0x100  name: kmalloc-192)
[task:0xffff888007b30f00 kworker/u4:1    ] kfree                         : 0xffff88800b38fe00 (size: 0x100  name: kmalloc-192)
[task:0xffff8880078b6900 kworker/1:1     ] kfree                         : 0xffff88800b38fc00 (size: 0x100  name: kmalloc-192)
[task:0xffff888010e2ad00 bash            ] kfree                         : 0xffff88800a9b3800 (size: 0x80   name: kmalloc-96)
[task:0xffff888008442d00 kworker/u4:2    ] kfree                         : 0xffff88800b38fb00 (size: 0x100  name: kmalloc-192)
[task:0xffff888010e2ad00 bash            ] kfree                         : 0xffff888009d1b5c0 (size: 0x40   name: kmalloc-32)
[task:0xffff8880078b3c00 kworker/1:0     ] kfree                         : 0xffff88800b38f900 (size: 0x100  name: kmalloc-192)
[task:0xffff888010e2ad00 ls              ] kfree                         : 0xffff88800a3e3780 (size: 0x80   name: kmalloc-64)
[task:0xffff888010e2ad00 ls              ] kfree                         : 0xffff888007dedc00 (size: 0x400  name: kmalloc-512)
[task:0xffff888010e2ad00 ls              ] kfree                         : 0xffff888010f23000 (size: 0x800  name: kmalloc-1k)
[task:0xffff8880078b3c00 kworker/1:0     ] kfree                         : 0xffff88800b38fa00 (size: 0x100  name: kmalloc-192)
[task:0xffff888010e2ad00 ls              ] kfree                         : 0xffff88800a996400 (size: 0x400  name: kmalloc-512)
[task:0xffff8880078b3c00 kworker/1:0     ] kfree                         : 0xffff88800b38f800 (size: 0x100  name: kmalloc-192)
[task:0xffff8880078b6900 kworker/1:1     ] kfree                         : 0xffff88800b38f600 (size: 0x100  name: kmalloc-192)

bata24 commented 2 weeks ago

Hmm, sorry, I'm sure I'm missing something, but I can't think of what it could be. Can you provide an image of the misbehaving kernel (zImage, vmlinuz, etc.)? If this is some kind of CTF challenge, a download link would be fine.

bata24 commented 2 weeks ago

I realized that I had set breakpoints on kmem_cache_alloc and kmem_cache_alloc_trace twice, so I just fixed that. However, I don't think that this is a fundamental fix to this issue.

scratchadams commented 2 weeks ago

I wonder if this is related to how these kernel images are built. I am using the following tool: https://github.com/deepseagirl/easylkb to build these images. to generate the exact image I am using I just run ./easylkb.py -k 6.5 -dcm then once the config/compile is complete ./easylkb.py -i

I do have to adjust the generated runk.sh script to remove --enable-kvm and -cpu host from the qemu cli, but otherwise I am not changing anything.

Honestly I have had better results with kCTF challenges (which have been older kernel versions) and using ksymaddr-remote-apply to apply the symbols.

I can provide the associated configs or images, however it might be easier to generate from scratch using that tool, just let me know.

bata24 commented 2 weeks ago

I created a kernel and image with easylkb.py -k 6.5 -dcm and easylkb.py -k 6.5 -i and ran the kmalloc-tracer command, but the problem did not occur again. I have pasted the log below, and it appears that it stopped correctly at __kmalloc etc. and was able to dump.

For further investigation, please provide the kernel you are using if possible, and also the output of the version command.

gef> kmalloc-tracer
[+] Wait for memory scan
kmem_cache_alloc: Breakpoint 1 at 0xffffffff8175c9e0
__kmalloc: Breakpoint 2 at 0xffffffff8163f1b0
kmalloc_large: Breakpoint 3 at 0xffffffff8163eae0
kmalloc_trace: Breakpoint 4 at 0xffffffff8163ea30
kmem_cache_alloc_node: Breakpoint 5 at 0xffffffff8175c720
__kmalloc_node: Breakpoint 6 at 0xffffffff8163f040
kmalloc_large_node: Breakpoint 7 at 0xffffffff8163eb90
kmalloc_node_trace: Breakpoint 8 at 0xffffffff8163e980
__kmalloc_node_track_caller: Breakpoint 9 at 0xffffffff8163eed0
krealloc: Breakpoint 10 at 0xffffffff8163f710
kfree: Breakpoint 11 at 0xffffffff8163f3e0
[+] Setup is complete. continuing...
[task:0xffff888011823c00 kworker/0:1     ] __kmalloc                     : 0xffff88800adb9b00 (size: 0x100  name: kmalloc-192)
[task:0xffff888007823c00 ksoftirqd/0     ] kfree                         : 0xffff88800adb9b00 (size: 0x100  name: kmalloc-192)
[task:0xffff888013968000 sshd            ] __kmalloc_node_track_caller   : 0xffff888012f81000 (size: 0x800  name: kmalloc-1k)
[task:0xffff888007872d00 ksoftirqd/1     ] kfree                         : 0xffff888012f81000 (size: 0x800  name: kmalloc-1k)
[task:0xffff88801396bc00 bash            ] kmalloc_trace                 : 0xffff88800adb9800 (size: 0x100  name: kmalloc-192)
[task:0xffff888011823c00 kworker/0:1     ] __kmalloc                     : 0xffff88800fa57f00 (size: 0x100  name: kmalloc-192)
[task:0xffff88801396bc00 bash            ] __kmalloc                     : 0xffff888014e97ec0 (size: 0x40   name: kmalloc-32)
[task:0xffff888011826900 kworker/0:3     ] kfree                         : 0xffff88800fa57f00 (size: 0x100  name: kmalloc-192)
[task:0xffff88801396bc00 bash            ] kmalloc_trace                 : 0xffff888007327580 (size: 0x80   name: kmalloc-64)
[task:0xffff88801396bc00 bash            ] __kmalloc_node                : 0xffff88800802f000 (size: 0x1000 name: kmalloc-2k)
[task:0xffff88801396bc00 bash            ] __kmalloc_node                : 0xffff888009e00800 (size: 0x80   name: kmalloc-96)
[task:0xffff888011826900 kworker/0:3     ] __kmalloc                     : 0xffff88800fa57e00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800fa57e00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800ba70f00 kworker/0:0     ] kmalloc_node_trace            : 0xffff88800fa57800 (size: 0x100  name: kmalloc-192)
[task:0xffff88800ba70f00 kworker/0:0     ] kmalloc_trace                 : 0xffff88800ac17100 (size: 0x80   name: kmalloc-64)
[task:0xffff88800ba70f00 kworker/0:0     ] __kmalloc_node_track_caller   : 0xffff88800f4a5080 (size: 0x20   name: kmalloc-16)
[task:0xffff888007398f00 kthreadd        ] __kmalloc                     : 0xffff888009e22a40 (size: 0x40   name: kmalloc-32)
[task:0xffff888007398f00 kthreadd        ] kmalloc_trace                 : 0xffff88800b356f00 (size: 0x100  name: kmalloc-128)
[task:0xffff888011826900 kworker/0:3     ] __kmalloc                     : 0xffff88800fa57400 (size: 0x100  name: kmalloc-192)
[task:0xffff888011823c00 kworker/0:1     ] kfree                         : 0xffff88800fa57400 (size: 0x100  name: kmalloc-192)
[task:0xffff88800ba70f00 kworker/0:0     ] kfree                         : 0xffff88800ac17100 (size: 0x80   name: kmalloc-64)
[task:0xffff888011823c00 kworker/0:1     ] __kmalloc                     : 0xffff88800fa57c00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800fa57c00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800fa57d00 (size: 0x100  name: kmalloc-192)
[task:0xffff888011823c00 kworker/0:1     ] kfree                         : 0xffff88800fa57d00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800fa57a00 (size: 0x100  name: kmalloc-192)
[task:0xffff888011823c00 kworker/0:1     ] kfree                         : 0xffff88800fa57a00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800fa57500 (size: 0x100  name: kmalloc-192)
[task:0xffff888011823c00 kworker/0:1     ] kfree                         : 0xffff88800fa57500 (size: 0x100  name: kmalloc-192)
[task:0xffff888011823c00 kworker/0:1     ] __kmalloc                     : 0xffff88800fa57300 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800fa57300 (size: 0x100  name: kmalloc-192)
[task:0xffff888011823c00 kworker/0:1     ] __kmalloc                     : 0xffff88800fa57200 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800fa57200 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e53c00 bash            ] kfree                         : 0xffff888011d0b800 (size: 0x800  name: kmalloc-1k)
[task:0xffff888014e53c00 bash            ] kfree                         : 0xffff88800adb9800 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e53c00 bash            ] kmalloc_trace                 : 0xffff888011edb000 (size: 0x400  name: kmalloc-512)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800fa57000 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e53c00 bash            ] __kmalloc                     : 0xffff888014e97e40 (size: 0x40   name: kmalloc-32)
[task:0xffff888011823c00 kworker/0:1     ] kfree                         : 0xffff88800fa57000 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e53c00 bash            ] __kmalloc                     : 0xffff888012f81800 (size: 0x800  name: kmalloc-1k)
[task:0xffff888014e53c00 bash            ] __kmalloc                     : 0xffff888014e97c40 (size: 0x40   name: kmalloc-32)
[task:0xffff888014e53c00 bash            ] kfree                         : 0xffff888014e97c40 (size: 0x40   name: kmalloc-32)
[task:0xffff888014e53c00 bash            ] kmalloc_trace                 : 0xffff88800ac17180 (size: 0x80   name: kmalloc-64)
[task:0xffff888014e53c00 bash            ] __kmalloc                     : 0xffff888011eda400 (size: 0x400  name: kmalloc-512)
[task:0xffff888011823c00 kworker/0:1     ] __kmalloc                     : 0xffff88800fa57100 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800fa57100 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff88800fa57900 (size: 0x100  name: kmalloc-192)
[task:0xffff888011823c00 kworker/0:1     ] kfree                         : 0xffff88800fa57900 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff888007336c00 (size: 0x100  name: kmalloc-192)
[task:0xffff888011823c00 kworker/0:1     ] kfree                         : 0xffff888007336c00 (size: 0x100  name: kmalloc-192)
[task:0xffff888007398000 systemd         ] kmalloc_trace                 : 0xffff888014e97d80 (size: 0x40   name: kmalloc-32)
[task:0xffff888007398000 systemd         ] __kmalloc_node                : 0xffff888013ba4000 (size: 0x2000 name: kmalloc-4k)
[task:0xffff888007398000 systemd         ] kfree                         : 0xffff888013ba0000 (size: 0x2000 name: kmalloc-4k)
[task:0xffff888007398000 systemd         ] kfree                         : 0xffff888013ba4000 (size: 0x2000 name: kmalloc-4k)
[task:0xffff888007398000 systemd         ] kfree                         : 0xffff888014e97d80 (size: 0x40   name: kmalloc-32)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff888007336700 (size: 0x100  name: kmalloc-192)
[task:0xffff888011823c00 kworker/0:1     ] kfree                         : 0xffff888007336700 (size: 0x100  name: kmalloc-192)
[task:0xffff888011823c00 kworker/0:1     ] __kmalloc                     : 0xffff888007336a00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff888007336a00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e53c00 ls              ] kfree                         : 0xffff88800ac17180 (size: 0x80   name: kmalloc-64)
[task:0xffff888014e53c00 ls              ] kfree                         : 0xffff888011eda400 (size: 0x400  name: kmalloc-512)
[task:0xffff888014e53c00 ls              ] kfree                         : 0xffff888012f81800 (size: 0x800  name: kmalloc-1k)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff888007336d00 (size: 0x100  name: kmalloc-192)
[task:0xffff888011823c00 kworker/0:1     ] kfree                         : 0xffff888007336d00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e53c00 ls              ] kfree                         : 0xffff888011edb000 (size: 0x400  name: kmalloc-512)
[task:0xffff888011823c00 kworker/0:1     ] __kmalloc                     : 0xffff888007336b00 (size: 0x100  name: kmalloc-192)
[task:0xffff888007823c00 ksoftirqd/0     ] kfree                         : 0xffff888007336b00 (size: 0x100  name: kmalloc-192)
[task:0xffff888011823c00 kworker/0:1     ] __kmalloc                     : 0xffff88800b014600 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800b014600 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff88800b014400 (size: 0x100  name: kmalloc-192)
[task:0xffff888011823c00 kworker/0:1     ] kfree                         : 0xffff88800b014400 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff88800b014500 (size: 0x100  name: kmalloc-192)
[task:0xffff888011823c00 kworker/0:1     ] kfree                         : 0xffff88800b014500 (size: 0x100  name: kmalloc-192)
[task:0xffff888011823c00 kworker/0:1     ] __kmalloc                     : 0xffff88800b014700 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800b014700 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff88800b014300 (size: 0x100  name: kmalloc-192)
[task:0xffff888011823c00 kworker/0:1     ] kfree                         : 0xffff88800b014300 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff88800b014d00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800b014d00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800b014f00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800b014f00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800b014e00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800b014e00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff888007f59400 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff888007f59400 (size: 0x100  name: kmalloc-192)
[task:0xffff888011823c00 kworker/0:1     ] __kmalloc                     : 0xffff888007f59c00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff888007f59c00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff888007f59f00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800ba70f00 kworker/0:0     ] kfree                         : 0xffff888007f59f00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff888007f59e00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800ba70f00 kworker/0:0     ] kfree                         : 0xffff888007f59e00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff888007f59d00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800ba70f00 kworker/0:0     ] kfree                         : 0xffff888007f59d00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800ba70f00 kworker/0:0     ] __kmalloc                     : 0xffff88800a2a0700 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800a2a0700 (size: 0x100  name: kmalloc-192)
[task:0xffff88800ba70f00 kworker/0:0     ] __kmalloc                     : 0xffff88800a2a0f00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800a2a0f00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800a2a0a00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800ba70f00 kworker/0:0     ] kfree                         : 0xffff88800a2a0a00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800a2a0200 (size: 0x100  name: kmalloc-192)
[task:0xffff888011823c00 kworker/0:1     ] kfree                         : 0xffff88800a2a0200 (size: 0x100  name: kmalloc-192)
[task:0xffff888012fccb00 systemd-udevd   ] kmalloc_trace                 : 0xffff888009e22b80 (size: 0x40   name: kmalloc-32)
[task:0xffff888011823c00 kworker/0:1     ] __kmalloc                     : 0xffff88801189aa00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88801189aa00 (size: 0x100  name: kmalloc-192)
[task:0xffff888007398000 systemd         ] kfree                         : 0xffff888009e22b80 (size: 0x40   name: kmalloc-32)
[task:0xffff888011823c00 kworker/0:1     ] __kmalloc                     : 0xffff88800a51ee00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800a51ee00 (size: 0x100  name: kmalloc-192)
[task:0xffff888007398000 systemd         ] kmalloc_trace                 : 0xffff888009e22d00 (size: 0x40   name: kmalloc-32)
[task:0xffff888007398000 systemd         ] __kmalloc_node                : 0xffff88800b534000 (size: 0x2000 name: kmalloc-4k)
[task:0xffff888007398000 systemd         ] kmalloc_trace                 : 0xffff88800b532000 (size: 0x2000 name: kmalloc-4k)
[task:0xffff888007398000 systemd         ] kfree                         : 0xffff88800b532000 (size: 0x2000 name: kmalloc-4k)
[task:0xffff888011823c00 kworker/0:1     ] __kmalloc                     : 0xffff88800a51eb00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800a51eb00 (size: 0x100  name: kmalloc-192)
[task:0xffff888007398000 systemd         ] kfree                         : 0xffff88800b534000 (size: 0x2000 name: kmalloc-4k)
[task:0xffff888007398000 systemd         ] kfree                         : 0xffff888009e22d00 (size: 0x40   name: kmalloc-32)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800a51ea00 (size: 0x100  name: kmalloc-192)
[task:0xffff888011823c00 kworker/0:1     ] kfree                         : 0xffff88800a51ea00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff88800a51e900 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800a51e900 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800a51e400 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800a51e400 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff88800a51e500 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800a51e500 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800a51e600 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800a51e600 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff88800a51e700 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800a51e700 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff88800a51e300 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800a51e300 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e53c00 ls              ] kmalloc_trace                 : 0xffff888007327000 (size: 0x80   name: kmalloc-64)
[task:0xffff888014e53c00 ls              ] __kmalloc                     : 0xffff888007327280 (size: 0x80   name: kmalloc-64)
[task:0xffff888014e53c00 ls              ] __kmalloc                     : 0xffff888007327480 (size: 0x80   name: kmalloc-64)
[task:0xffff888014e53c00 ls              ] __kmalloc                     : 0xffff888007327700 (size: 0x80   name: kmalloc-64)
[task:0xffff888014e53c00 ls              ] __kmalloc                     : 0xffff88800ac17380 (size: 0x80   name: kmalloc-64)
[task:0xffff888014e53c00 ls              ] kfree                         : 0xffff88800ac17400 (size: 0x80   name: kmalloc-64)
[task:0xffff888014e53c00 ls              ] kfree                         : 0xffff888007327480 (size: 0x80   name: kmalloc-64)
[task:0xffff888014e53c00 ls              ] kfree                         : 0xffff88800ac17380 (size: 0x80   name: kmalloc-64)
[task:0xffff888014e53c00 ls              ] kfree                         : 0xffff888007327700 (size: 0x80   name: kmalloc-64)
[task:0xffff888014e53c00 ls              ] kfree                         : 0xffff888007327280 (size: 0x80   name: kmalloc-64)
[task:0xffff888014e53c00 ls              ] kfree                         : 0xffff888007327000 (size: 0x80   name: kmalloc-64)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800a51e800 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e53c00 ls              ] kfree                         : 0xffff88800802f000 (size: 0x1000 name: kmalloc-2k)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800a51e800 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e53c00 ls              ] kfree                         : 0xffff888009e00800 (size: 0x80   name: kmalloc-96)
[task:0xffff888014e53c00 ls              ] kfree                         : 0xffff888007327580 (size: 0x80   name: kmalloc-64)
[task:0xffffffff85014280 swapper/0       ] kfree                         : 0xffff888014e97ec0 (size: 0x40   name: kmalloc-32)
[task:0xffff88801396bc00 bash            ] kfree                         : 0xffff888014e97e40 (size: 0x40   name: kmalloc-32)
[task:0xffff888013968000 sshd            ] __kmalloc_node_track_caller   : 0xffff888012f82000 (size: 0x800  name: kmalloc-1k)
[task:0xffff888013968000 sshd            ] kfree                         : 0xffff888012f82000 (size: 0x800  name: kmalloc-1k)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff88800a51e000 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800a51e000 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800a51ed00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800a51ed00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff88800a51ef00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800a51ef00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800adc9000 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800adc9000 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff88800adc9100 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800adc9100 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800adc9d00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800adc9d00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800adc9b00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800adc9b00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff88800adc9a00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800adc9a00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff88800adc9c00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800adc9c00 (size: 0x100  name: kmalloc-192)
[task:0xffff888007398000 systemd         ] __kmalloc_node                : 0xffff88800b530000 (size: 0x2000 name: kmalloc-4k)
[task:0xffff888007398000 systemd         ] kmalloc_trace                 : 0xffff88800fbe6000 (size: 0x2000 name: kmalloc-4k)
[task:0xffff888007398000 systemd         ] kfree                         : 0xffff88800fbe6000 (size: 0x2000 name: kmalloc-4k)
[task:0xffff888007398000 systemd         ] kfree                         : 0xffff88800b530000 (size: 0x2000 name: kmalloc-4k)
[task:0xffff888007398000 systemd         ] kfree                         : 0xffff888014e97a80 (size: 0x40   name: kmalloc-32)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800adc9f00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800adc9f00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c4f9e00 kworker/dying   ] kfree                         : 0xffff888007f59100 (size: 0x100  name: kmalloc-192)
[task:0xffff88801192cb00 kworker/dying   ] kfree                         : 0xffff88800b019800 (size: 0x100  name: kmalloc-192)
[task:0xffff888007826900 swapper/1       ] kfree                         : 0xffff88800b797860 (size: 0x20   name: kmalloc-16)
[task:0xffff888007826900 swapper/1       ] kfree                         : 0xffff88800be6e900 (size: 0x100  name: kmalloc-128)
[task:0xffff888007824b00 rcu_preempt     ] kfree                         : 0xffff88800b7975c0 (size: 0x20   name: kmalloc-16)
[task:0xffff888007824b00 rcu_preempt     ] kfree                         : 0xffff888007a6fb00 (size: 0x100  name: kmalloc-128)
[task:0xffff888007824b00 rcu_preempt     ] kfree                         : 0xffff888010776a80 (size: 0x40   name: kmalloc-32)
[task:0xffff888007826900 swapper/1       ] kfree                         : 0xffff88800af93f40 (size: 0x40   name: kmalloc-32)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff88800adc9e00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800adc9e00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff888009e36000 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff888009e36000 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff888009e36400 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff888009e36400 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff888009e36300 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff888009e36300 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff888009e36200 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff888009e36200 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800beb7000 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800beb7000 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff88800beb7100 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800beb7100 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800beb7200 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800beb7200 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff88800beb7300 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800beb7300 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800beb7400 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800beb7400 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff88800beb7500 (size: 0x100  name: kmalloc-192)
[task:0xffff888011824b00 kworker/1:0     ] kmalloc_node_trace            : 0xffff88800aec2900 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800beb7500 (size: 0x100  name: kmalloc-192)
[task:0xffff888011824b00 kworker/1:0     ] kmalloc_trace                 : 0xffff888007327380 (size: 0x80   name: kmalloc-64)
[task:0xffff888011824b00 kworker/1:0     ] __kmalloc_node_track_caller   : 0xffff88800b7973a0 (size: 0x20   name: kmalloc-16)
[task:0xffff888007398f00 kthreadd        ] __kmalloc                     : 0xffff888009e22c00 (size: 0x40   name: kmalloc-32)
[task:0xffff888007398f00 kthreadd        ] kmalloc_trace                 : 0xffff88800c280000 (size: 0x100  name: kmalloc-128)
[task:0xffff888011824b00 kworker/1:0     ] kfree                         : 0xffff888007327380 (size: 0x80   name: kmalloc-64)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800beb7600 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800beb7600 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff88800beb7700 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800beb7700 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800beb7800 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800beb7800 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800beb7900 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800beb7900 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff88800beb7a00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800beb7a00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800beb7b00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800beb7b00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff88800beb7c00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800beb7c00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800beb7d00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800beb7d00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff88800beb7e00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff88800beb7e00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff88800beb7f00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff88800beb7f00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff888011b42000 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff888011b42000 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff888011b42100 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff888011b42100 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff888011b42200 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff888011b42200 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff888011b42300 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff888011b42300 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff888011b42400 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff888011b42400 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff888011b42500 (size: 0x100  name: kmalloc-192)
[task:0xffff888013968000 sshd            ] __kmalloc_node_track_caller   : 0xffff888011d0b000 (size: 0x800  name: kmalloc-1k)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff888011b42500 (size: 0x100  name: kmalloc-192)
[task:0xffff888013968000 sshd            ] kfree                         : 0xffff888011d0b000 (size: 0x800  name: kmalloc-1k)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff888011b42600 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff888011b42600 (size: 0x100  name: kmalloc-192)
[task:0xffff888012fccb00 systemd-udevd   ] kmalloc_trace                 : 0xffff888009e22cc0 (size: 0x40   name: kmalloc-32)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff888011b42700 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff888011b42700 (size: 0x100  name: kmalloc-192)
[task:0xffff888007398000 systemd         ] __kmalloc_node                : 0xffff888010736000 (size: 0x2000 name: kmalloc-4k)
[task:0xffff888007398000 systemd         ] kmalloc_trace                 : 0xffff88800bedc000 (size: 0x2000 name: kmalloc-4k)
[task:0xffff888007398000 systemd         ] kmalloc_trace                 : 0xffff88800bedc000 (size: 0x2000 name: kmalloc-4k)
[task:0xffff888007398000 systemd         ] kfree                         : 0xffff88800bedc000 (size: 0x2000 name: kmalloc-4k)
[task:0xffff888007398000 systemd         ] kfree                         : 0xffff888010736000 (size: 0x2000 name: kmalloc-4k)
[task:0xffff888007398000 systemd         ] kfree                         : 0xffff888014e97d40 (size: 0x40   name: kmalloc-32)
[task:0xffff888007398000 systemd         ] kfree                         : 0xffff888009e22cc0 (size: 0x40   name: kmalloc-32)
[task:0xffff888007398000 systemd         ] kmalloc_trace                 : 0xffff888014e97980 (size: 0x40   name: kmalloc-32)
[task:0xffff888007398000 systemd         ] __kmalloc_node                : 0xffff88800bede000 (size: 0x2000 name: kmalloc-4k)
[task:0xffff888007398000 systemd         ] kfree                         : 0xffff888011aa6000 (size: 0x2000 name: kmalloc-4k)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff888011b42800 (size: 0x100  name: kmalloc-192)
[task:0xffff888007398000 systemd         ] kfree                         : 0xffff88800bede000 (size: 0x2000 name: kmalloc-4k)
[task:0xffff888007398000 systemd         ] kfree                         : 0xffff888014e97980 (size: 0x40   name: kmalloc-32)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff888011b42800 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff888011b42900 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff888011b42900 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff888011b42a00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff888011b42a00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff888011b42b00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff888011b42b00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff888011b42c00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff888011b42c00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff888011b42d00 (size: 0x100  name: kmalloc-192)
[task:0xffff888007823c00 ksoftirqd/0     ] kfree                         : 0xffff888011b42d00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff888011b42e00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff888011b42e00 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff888011b42f00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff888011b42f00 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff888011448000 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff888011448000 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff888011448000 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff888011448100 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff888011448100 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff888011448200 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff888011448200 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff888011448300 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff888011448300 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff888011448400 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff888011448400 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff888011448500 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff888011448500 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff888011448600 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] kfree                         : 0xffff888011448600 (size: 0x100  name: kmalloc-192)
[task:0xffff88800c5d8000 kworker/0:4     ] __kmalloc                     : 0xffff888011448700 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff888011448700 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] __kmalloc                     : 0xffff888011448800 (size: 0x100  name: kmalloc-192)
[task:0xffff888011823c00 kworker/0:1     ] kfree                         : 0xffff888011448800 (size: 0x100  name: kmalloc-192)
[task:0xffff888011823c00 kworker/0:1     ] __kmalloc                     : 0xffff888011448900 (size: 0x100  name: kmalloc-192)
[task:0xffff888014e55a00 kworker/0:2     ] kfree                         : 0xffff888011448900 (size: 0x100  name: kmalloc-192)
^C

bata24 commented 2 weeks ago

It seems that when the thread hits breakpoint at thread 2 (not thread 1), the memory map may not be obtained correctly when using the python API. I think this is a bug in gdb or the gdb stub of qemu. Workaround: It seems to work correctly if you remove -smp N from the qemu startup options.

Could you please try this workaround and check if it solves your problem?

scratchadams commented 2 weeks ago

So removing the -smp option from the qemu cli args did not fix this for me. Here is the kernel/version info you requested in the previous reply: guest kernel:

root@localhost:~# uname -a
Linux localhost 6.5.0 #1 SMP PREEMPT_DYNAMIC Wed Aug 28 08:37:17 EDT 2024 x86_64 GNU/Linux

host kernel:

Linux hop 6.5.0-41-generic #41-Ubuntu SMP PREEMPT_DYNAMIC Mon May 20 15:55:15 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

gdb version:

gef> version
--------------------------------------------------- versions ---------------------------------------------------
OS:                     Ubuntu 23.10
kernel (uname -a):      Linux hop 6.5.0-41-generic #41-Ubuntu SMP PREEMPT_DYNAMIC Mon May 20 15:55:15 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
kernel (/proc/version): Linux version 6.5.0-41-generic (buildd@lcy02-amd64-037) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.2.0-4ubuntu3) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.41) #41-Ubuntu SMP PREEMPT_DYNAMIC Mon May 20 15:55:15 UTC 2024
System libc:            GNU C Library (Ubuntu GLIBC 2.38-1ubuntu6.3) stable release version 2.38.
GEF:                    Last modified: 2024-08-28 16:13:42 SHA1: d725cbecdd5a247ac0a79e838cfafad43488c3f9
gdb:                    14.0.50.20230907-git
python:                 3.11.6 (main, Apr 10 2024, 17:26:07) [GCC 13.2.0]
capstone:               5.0.1280
keystone:               Not found
unicorn:                Not found
ropper:                 Not found
gcc:                    gcc (Ubuntu 13.2.0-4ubuntu3) 13.2.0
readelf:                GNU readelf (GNU Binutils for Ubuntu) 2.41
objdump:                GNU objdump (GNU Binutils for Ubuntu) 2.41
seccomp-tools:          SeccompTools Version 1.6.0
one_gadget:             OneGadget Version 1.9.0
rp:                     the version 2.1 x64 built the Mar  9 2023 05:09:47 for Linux (Release) of rp++.
----------------------------------------------- gdb build config -----------------------------------------------
This GDB was configured as follows:
   configure --host=x86_64-linux-gnu --target=x86_64-linux-gnu
         --with-auto-load-dir=$debugdir:$datadir/auto-load
         --with-auto-load-safe-path=$debugdir:$datadir/auto-load
         --with-expat
         --with-gdb-datadir=/usr/share/gdb (relocatable)
         --with-jit-reader-dir=/usr/lib/gdb (relocatable)
         --without-libunwind-ia64
         --with-lzma
         --with-babeltrace
         --with-intel-pt
         --with-xxhash
         --with-python=/usr (relocatable)
         --with-python-libdir=/usr/lib (relocatable)
         --with-debuginfod
         --with-curses
         --without-guile
         --without-amd-dbgapi
         --enable-source-highlight
         --enable-threading
         --enable-tui
         --with-system-readline
         --with-separate-debug-dir=/usr/lib/debug (relocatable)
         --with-system-gdbinit=/etc/gdb/gdbinit
         --with-system-gdbinit-dir=/etc/gdb/gdbinit.d

("Relocatable" means the directory can be moved with the GDB installation
tree, and GDB will still find it.)

qemu startup cli args:

qemu-system-x86_64 -m 2G -kernel /root/scratch/temp/easylkb/kernel/linux-6.5//arch/x86/boot/bzImage -append "console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0 nokaslr" -drive file=/root/scratch/temp/easylkb/kernel/linux-6.5/img/bullseye.img,format=raw -net user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:10021-:22 -net nic,model=e1000 -nographic -s -pidfile /root/scratch/temp/easylkb/kernel/linux-6.5/img/vm.pid 2>&1 | tee /root/scratch/temp/easylkb/kernel/linux-6.5/img/vm.log

bata24 commented 2 weeks ago

I'm sorry, but I still don't know the cause.

kmalloc-tracer sets breakpoints for __kmalloc etc. Then, when it stops, it uses gdb.FinishBreakpoint to stop again when it returns from __kmalloc.

Could you please check if it will stop here (the moment it returns from __kmalloc)?

insert print("hit") in the top of stop method of KmallocRetBreakpoint class

If it hasn't reached this point, it may be the same issue as https://github.com/bata24/gef/issues/78, but the cause of this is unknown.

bata24 commented 2 weeks ago

Closed as cannot be reproduced.

bata24 / gef

kmalloc-tracer only triggering on kfree #88