carloslack / KoviD

Linux kernel rootkit
Other
275 stars 53 forks source link

Port kv to Ubuntu 20.04.6 LTS, kernel 5.4.0-164-generic #181-Ubuntu SM #72

Closed carloslack closed 1 year ago

carloslack commented 1 year ago

Motivated by https://github.com/carloslack/KoviD/issues/71

iusearch commented 1 year ago

One thing I noticed from strace log

rt_sigaction(SIGPIPE, {sa_handler=SIG_IGN, sa_mask=[PIPE], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f7d0cebd090}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
rt_sigaction(SIGWINCH, {sa_handler=0x7f7d0d48bf60, sa_mask=[WINCH], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f7d0cebd090}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
ioctl(1, TIOCGWINSZ, {ws_row=34, ws_col=135, ws_xpixel=1221, ws_ypixel=682}) = 0
ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
stat("/etc/apt/sources.list", {st_mode=S_IFREG|0644, st_size=2785, ...}) = 0
openat(AT_FDCWD, "/etc/apt/sources.list", O_RDONLY|O_NOCTTY|O_CLOEXEC) = 4
read(4, "# See http://help.ubuntu.com/com"..., 4096) = 2785
read(4, "", 1311)                       = 0
openat(AT_FDCWD, "/var/lib/apt/lists/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 5
fstat(5, {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0
getdents64(5, /* 66 entries */, 32768)  = 5904
getdents64(5, /* 0 entries */, 32768)   = 0
close(5)                                = 0
pipe([5, 6])                            = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD) = ? <unavailable>

This is with kv on. You can see strace is stucking on clone, while without kv

rt_sigaction(SIGPIPE, {sa_handler=SIG_IGN, sa_mask=[PIPE], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7ff9afb7d090}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
rt_sigaction(SIGWINCH, {sa_handler=0x7ff9b014bf60, sa_mask=[WINCH], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7ff9afb7d090}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
ioctl(1, TIOCGWINSZ, {ws_row=34, ws_col=135, ws_xpixel=1221, ws_ypixel=682}) = 0
ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
stat("/etc/apt/sources.list", {st_mode=S_IFREG|0644, st_size=2785, ...}) = 0
openat(AT_FDCWD, "/etc/apt/sources.list", O_RDONLY|O_NOCTTY|O_CLOEXEC) = 4
read(4, "# See http://help.ubuntu.com/com"..., 4096) = 2785
read(4, "", 1311)                       = 0
openat(AT_FDCWD, "/var/lib/apt/lists/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 5
fstat(5, {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0
getdents64(5, /* 66 entries */, 32768)  = 5904
getdents64(5, /* 0 entries */, 32768)   = 0
close(5)                                = 0
pipe([5, 6])                            = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7ff9af658ad0) = 1888
close(6)                                = 0
fcntl(5, F_GETFL)                       = 0 (flags O_RDONLY)
fstat(5, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
read(5, "", 4096)                       = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1888, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
close(5)                                = 0
wait4(1888, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 1888
stat("/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_focal_InRelease", {st_mode=S_IFREG|0644, st_size=264892, ...}) = 0
stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...}) = 0
geteuid()                               = 0
getpid()                                = 1887

It pass through that clone and continues its job. Without strace the apt install will crash the kernel.

carloslack commented 1 year ago

Thanks @iusearch !

iusearch commented 1 year ago

Ok, I found it's not really apt's fault. Basically unhide anything would cause a crash, even if it is a bash

[  261.995650] kovid: module verification failed: signature and/or required key missing - tainting kernel
[  262.027295] kv: using kprobe for kallsyms_lookup_name
[  262.042613] add sysaddr: ffffffff9f0da750
[  262.042625] addname '.QVVPFJ' ro=1
[  262.042628] new var, filename: '/var/.QVVPFJ'
[  262.042636] Installing: 'sys_exit_group' syscall=1
[  262.058151] add sysaddr: ffffffff9f0c38c0
[  262.059160] Installing: 'sys_clone' syscall=1
[  262.069031] add sysaddr: ffffffff9f0bc670
[  262.069880] Installing: 'sys_kill' syscall=1
[  262.082817] add sysaddr: ffffffff9f0d29c0
[  262.083992] Installing: 'sys_bpf' syscall=1
[  262.095898] add sysaddr: ffffffff9f246530
[  262.096816] Installing: 'tcp4_seq_show' syscall=0
[  262.101257] Installing: 'udp4_seq_show' syscall=0
[  262.105640] Installing: 'tcp6_seq_show' syscall=0
[  262.109990] Installing: 'udp6_seq_show' syscall=0
[  262.114404] Installing: 'packet_rcv' syscall=0
[  262.118753] Installing: 'tpacket_rcv' syscall=0
[  262.123295] Installing: 'account_process_tick' syscall=0
[  262.124405] Installing: 'account_system_time' syscall=0
[  262.125496] Installing: 'audit_log_start' syscall=0
[  262.126850] Installing: 'filldir' syscall=0
[  262.128825] Installing: 'filldir64' syscall=0
[  262.130661] Installing: 'tty_read' syscall=0
[  262.133946] ftrace hook 0 on sys_exit_group
[  262.133947] ftrace hook 1 on sys_clone
[  262.133947] ftrace hook 2 on sys_kill
[  262.133947] ftrace hook 3 on sys_bpf
[  262.133948] ftrace hook 4 on tcp4_seq_show
[  262.133949] ftrace hook 5 on udp4_seq_show
[  262.133950] ftrace hook 6 on tcp6_seq_show
[  262.133950] ftrace hook 7 on udp6_seq_show
[  262.133950] ftrace hook 8 on packet_rcv
[  262.133950] ftrace hook 9 on tpacket_rcv
[  262.133951] ftrace hook 10 on account_process_tick
[  262.133951] ftrace hook 11 on account_system_time
[  262.133952] ftrace hook 12 on audit_log_start
[  262.133952] ftrace hook 13 on filldir
[  262.133952] ftrace hook 14 on filldir64
[  262.133953] ftrace hook 15 on tty_read
[  262.134205] Waiting for event
[  262.134422] hide [0000000044965747] irq/102_pciehp : 2435
[  262.134484] hide [00000000322a7ff7] irq/101_pciehp : 2434
[  262.134497] hide [00000000eb823489] irq/100_pciehp : 2433
[  262.134499] addname '.kovid' ro=1
[  262.134500] addname 'kovid' ro=1
[  262.134500] addname '.kv.ko' ro=1
[  262.134501] addname '.lm.sh' ro=1
[  262.134501] addname '.sshd_orig' ro=1
[  262.134501] addname 'whitenose' ro=1
[  262.134502] addname 'pinknose' ro=1
[  262.134502] addname 'rednose' ro=1
[  262.134503] addname 'greynose' ro=1
[  262.134503] addname 'purplenose' ro=1
[  262.134504] addname 'blacknose' ro=1
[  262.134504] addname 'bluenose' ro=1
[  262.134536] kovid loaded.
[  289.172386] Got event
[  289.278917] hide [00000000f91a640a] sh : 2439
[  289.278988] hide [0000000013029127] bash : 2438
[  289.279005] Waiting for event
[  289.279018] Got event
[  289.279020] Waiting for event
[  304.903163] hide [00000000a0ddf223] bash : 2440
[  312.424545] unhide [00000000a0ddf223] bash : 2440
[  333.596283] general protection fault, probably for non-canonical address 0xdead000000000122: 0000 [#1] SMP NOPTI
[  333.596353] CPU: 1 PID: 2438 Comm: bash Kdump: loaded Tainted: G            E     5.15.0-86-generic #96~20.04.1-Ubuntu
[  333.596422] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown 2/2/2022
[  333.596464] RIP: 0010:__change_pid+0x61/0x110
[  333.596496] Code: 8d b4 e5 60 01 00 00 4d 8b 2e 49 83 fc 04 0f 87 9c 00 00 00 49 81 c4 a3 00 00 00 49 c1 e4 04 4a 8d 3c 23 48 8b 07 48 8b 4f 08 <48> 89 01 48 85 c0 74 04 48 89 48 08 48 b8 22 01 00 00 00 00 ad de
[  333.596587] RSP: 0018:ffffbd5440f2bb90 EFLAGS: 00010006
[  333.596617] RAX: dead000000000100 RBX: ffff9b6685fb3380 RCX: dead000000000122
[  333.596653] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9b6685fb3db0
[  333.596690] RBP: ffffbd5440f2bbb8 R08: 00000000000000a9 R09: ffff9b6683442124
[  333.596726] R10: 0000000000000663 R11: 000000000000cd21 R12: 0000000000000a30
[  333.596763] R13: ffff9b66809f8e00 R14: ffff9b6685fb3da8 R15: ffff9b6685fb3380
[  333.596799] FS:  00007fc829a50740(0000) GS:ffff9b67f7c80000(0000) knlGS:0000000000000000
[  333.596844] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  333.596876] CR2: 00007f63bc720110 CR3: 0000000100bae000 CR4: 0000000000750ee0
[  333.596926] PKRU: 55555554
[  333.596949] Call Trace:
[  333.596970]  <TASK>
[  333.596987]  ? show_regs.cold+0x1a/0x1f
[  333.597013]  ? __die_body+0x20/0x70
[  333.597037]  ? die_addr+0x3e/0x60
[  333.597060]  ? exc_general_protection+0x1c5/0x3f0
[  333.597092]  ? asm_exc_general_protection+0x27/0x30
[  333.597121]  ? __change_pid+0x61/0x110
[  333.597146]  detach_pid+0x10/0x20
[  333.597166]  release_task+0x27c/0x4a0
[  333.597188]  wait_task_zombie+0x4f9/0x5b0
[  333.597212]  wait_consider_task+0x3d7/0x560
[  333.597237]  do_wait+0x1e9/0x320
[  333.597257]  kernel_wait4+0xaf/0x150
[  333.597279]  ? task_stopped_code+0x50/0x50
[  333.597303]  __do_sys_wait4+0x89/0xa0
[  333.597325]  ? do_sigaction+0x12f/0x290
[  333.597350]  ? _copy_to_user+0x20/0x30
[  333.597373]  ? __x64_sys_rt_sigaction+0x7f/0x100
[  333.597401]  ? exit_to_user_mode_prepare+0x3d/0x1c0
[  333.597429]  __x64_sys_wait4+0x1e/0x30
[  333.597454]  do_syscall_64+0x59/0xc0
[  333.597477]  ? syscall_exit_to_user_mode+0x35/0x50
[  333.597505]  ? do_syscall_64+0x69/0xc0
[  333.597528]  ? irqentry_exit_to_user_mode+0x17/0x20
[  333.597556]  ? irqentry_exit+0x1d/0x30
[  333.597580]  ? exc_page_fault+0x89/0x170
[  333.597604]  entry_SYSCALL_64_after_hwframe+0x62/0xcc
[  333.597632] RIP: 0033:0x7fc829b35c3a
[  333.597655] Code: ff e9 0a 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 49 89 ca 64 8b 04 25 18 00 00 00 85 c0 75 15 b8 3d 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5e c3 0f 1f 44 00 00 48 83 ec 28 89 54 24 14
[  333.598844] RSP: 002b:00007fff5fd9f9f8 EFLAGS: 00000246 ORIG_RAX: 000000000000003d
[  333.600031] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fc829b35c3a
[  333.601237] RDX: 0000000000000000 RSI: 00007fff5fd9fa10 RDI: 00000000ffffffff
[  333.602433] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000007
[  333.603612] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000001
[  333.604719] R13: 00007fff5fd9faf0 R14: 0000000000000000 R15: 0000000000000000
[  333.605811]  </TASK>
[  333.606873] Modules linked in: kovid(E) nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua kvm_intel kvm joydev binfmt_misc input_leds serio_raw sch_fq_codel msr efi_pstore virtio_rng ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor hid_generic usbhid hid raid6_pq libcrc32c raid1 raid0 multipath linear virtio_gpu virtio_dma_buf drm_kms_helper crct10dif_pclmul syscopyarea crc32_pclmul ghash_clmulni_intel sysfillrect sysimgblt fb_sys_fops cec aesni_intel rc_core crypto_simd xhci_pci ahci virtio_net net_failover cryptd psmouse drm libahci virtio_blk xhci_pci_renesas failover
[  333.613232] kvm-guest: disable async PF for cpu 1
carloslack commented 1 year ago

Can you reproduce this in one of the supported kernels indicated in the README?

iusearch commented 1 year ago

It is using the Linux ubuntu-20.04 5.15.0-86-generic #96~20.04.1-Ubuntu SMP Thu Sep 21 13:23:37 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux kernel which is supported I believe.

carloslack commented 1 year ago

Yes, but I didn't port it to Ubuntu 20.04 , so it is basically not supported yet, hence this issue, but hide/unhide will be the first thing I will verify

carloslack commented 1 year ago

This is on Linux ubuntu 5.4.0-050400-generic

no issues found so far

I could run "apt update" without problems (in kv because of course apt eventually hungs) [43137.644676] hidden: '.DKFWTM' [43143.534772] Kthread : task 0000000043fba6fd : irq/102_pciehp : pid 65778 : group 65778 [43143.534776] Kthread : task 000000002b5728b3 : irq/101_pciehp : pid 65777 : group 65777 [43143.534777] Kthread : task 0000000003c4dbfa : irq/100_pciehp : pid 65776 : group 65776 [43143.534780] Task : tst : ino 530336 : task 000000008ab2687c : tst : pid 65781 : group 65781 [43143.534782] BD : openssl : ino 796831 : task 00000000b8836db6 : openssl : pid 65789 : group 65786 [43143.534784] BD : dash : ino 786638 : task 00000000076dd9bd : sh : pid 65788 : group 65786 [43143.534786] BD : bash : ino 786526 : task 00000000b4bd58aa : bash : pid 65786 : group 65786 [43157.166017] Kthread : task 0000000043fba6fd : irq/102_pciehp : pid 65778 : group 65778 [43157.166020] Kthread : task 000000002b5728b3 : irq/101_pciehp : pid 65777 : group 65777 [43157.166021] Kthread : task 0000000003c4dbfa : irq/100_pciehp : pid 65776 : group 65776 [43157.166024] Task : tst : ino 530336 : task 000000008ab2687c : tst : pid 65781 : group 65781 [43157.166026] BD : openssl : ino 796831 : task 00000000b8836db6 : openssl : pid 65789 : group 65786 [43157.166028] BD : dash : ino 786638 : task 00000000076dd9bd : sh : pid 65788 : group 65786 [43157.166029] BD : bash : ino 786526 : task 00000000b4bd58aa : bash : pid 65786 : group 65786 [43163.900844] unhide [000000008ab2687c] tst : 65781 [43166.506564] Kthread : task 0000000043fba6fd : irq/102_pciehp : pid 65778 : group 65778 [43166.506570] Kthread : task 000000002b5728b3 : irq/101_pciehp : pid 65777 : group 65777 [43166.506722] Kthread : task 0000000003c4dbfa : irq/100_pciehp : pid 65776 : group 65776 [43166.506730] BD : openssl : ino 796831 : task 00000000b8836db6 : openssl : pid 65789 : group 65786 [43166.506733] BD : dash : ino 786638 : task 00000000076dd9bd : sh : pid 65788 : group 65786 [43166.506736] BD : bash : ino 786526 : task 00000000b4bd58aa : bash : pid 65786 : group 65786 [43168.804999] hide [000000008ab2687c] tst : 65781 [43169.533450] Kthread : task 0000000043fba6fd : irq/102_pciehp : pid 65778 : group 65778 [43169.533452] Kthread : task 000000002b5728b3 : irq/101_pciehp : pid 65777 : group 65777 [43169.533454] Kthread : task 0000000003c4dbfa : irq/100_pciehp : pid 65776 : group 65776 [43169.533456] BD : openssl : ino 796831 : task 00000000b8836db6 : openssl : pid 65789 : group 65786 [43169.533458] BD : dash : ino 786638 : task 00000000076dd9bd : sh : pid 65788 : group 65786 [43169.533459] BD : bash : ino 786526 : task 00000000b4bd58aa : bash : pid 65786 : group 65786 [43169.533461] Task : tst : ino 530336 : task 000000008ab2687c : tst : pid 65781 : group 65781 [43380.996653] unhide [000000008ab2687c] tst : 65781 [44398.574616] /proc/kovid timeout [44398.574706] /proc/kv unloaded. [48475.484587] e1000: ens33 NIC Link is Down [48481.532944] e1000: ens33 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None [48540.511508] /proc/kv loaded, timeout: 1200s [48541.511299] Kthread : task 0000000043fba6fd : irq/102_pciehp : pid 65778 : group 65778 [48541.511302] Kthread : task 000000002b5728b3 : irq/101_pciehp : pid 65777 : group 65777 [48541.511304] Kthread : task 0000000003c4dbfa : irq/100_pciehp : pid 65776 : group 65776 [48541.511306] BD : openssl : ino 796831 : task 00000000b8836db6 : openssl : pid 65789 : group 65786 [48541.511308] BD : dash : ino 786638 : task 00000000076dd9bd : sh : pid 65788 : group 65786 [48541.511310] BD : bash : ino 786526 : task 00000000b4bd58aa : bash : pid 65786 : group 65786 [48544.747964] Kthread : task 0000000043fba6fd : irq/102_pciehp : pid 65778 : group 65778 [48544.747967] Kthread : task 000000002b5728b3 : irq/101_pciehp : pid 65777 : group 65777 [48544.747968] Kthread : task 0000000003c4dbfa : irq/100_pciehp : pid 65776 : group 65776 [48544.747970] BD : openssl : ino 796831 : task 00000000b8836db6 : openssl : pid 65789 : group 65786 [48544.747972] BD : dash : ino 786638 : task 00000000076dd9bd : sh : pid 65788 : group 65786 [48544.747973] BD : bash : ino 786526 : task 00000000b4bd58aa : bash : pid 65786 : group 65786 [48640.929094] hide [00000000bd8d9666] apt : 66171 [48641.119147] hide [00000000aef7d517] apt : 66174 [48641.121184] hide [00000000fa5e9351] sh : 66175 [48641.189923] unhide [00000000fa5e9351] sh : 66175 [48641.190422] unhide [00000000aef7d517] apt : 66174 [48641.671390] hide [000000003d896c83] gpgv : 66195 [48641.688934] hide [00000000aa409e8d] gpgv : 66196 [48641.707461] hide [00000000d833b31d] apt-key : 66197 [48641.758935] hide [00000000aea51e1e] apt-config : 66199 [48641.780442] unhide [00000000aea51e1e] apt-config : 66199 [48641.824735] hide [000000003a8607e3] apt-config : 66201 [48641.853513] unhide [000000003a8607e3] apt-config : 66201 [48641.877894] hide [0000000014c6aa6e] apt-config : 66204 [48641.887946] unhide [0000000014c6aa6e] apt-config : 66204 [48641.910188] hide [00000000b2f3f6d8] apt-config : 66207 [48641.919903] unhide [00000000b2f3f6d8] apt-config : 66207 [48641.945800] hide [00000000fdf7e0a3] apt-config : 66210 [48641.957697] unhide [00000000fdf7e0a3] apt-config : 66210 [48641.977884] hide [00000000e2589ec7] apt-config : 66212 [48641.988826] unhide [00000000e2589ec7] apt-config : 66212 [48642.017231] hide [00000000a6ca21ab] apt-config : 66214 [48642.032514] unhide [00000000a6ca21ab] apt-config : 66214 [48642.160396] hide [000000008dcfc62b] apt-config : 66223 [48642.176829] unhide [000000008dcfc62b] apt-config : 66223 [48642.194557] hide [00000000e5884aa8] apt-key : 66229 [48642.209085] unhide [00000000e5884aa8] apt-key : 66229 [48642.323791] hide [00000000a02de892] apt-key : 66254 [48642.337578] unhide [00000000a02de892] apt-key : 66254 [48642.339623] hide [000000000935004d] apt-key : 66257 [48642.356512] unhide [000000000935004d] apt-key : 66257 [48642.408849] hide [00000000981c9be4] gpgconf : 66263 [48642.452455] unhide [00000000981c9be4] gpgconf : 66263 [48642.456893] unhide [00000000d833b31d] apt-key : 66197 [48642.457806] unhide [00000000aa409e8d] gpgv : 66196 [48642.495056] hide [00000000a3306607] gpgv : 66268 [48642.498029] hide [0000000048aee5ea] apt-key : 66269 [48642.510157] hide [000000001f490051] apt-config : 66271 [48642.518275] unhide [000000001f490051] apt-config : 66271 [48642.537311] hide [000000006c3ac25f] apt-config : 66273 [48642.544415] unhide [000000006c3ac25f] apt-config : 66273 [48642.555663] hide [00000000b0e692f0] apt-config : 66275 [48642.561977] unhide [00000000b0e692f0] apt-config : 66275 [48642.585144] hide [00000000ba26fe3d] apt-config : 66277 [48642.590972] unhide [00000000ba26fe3d] apt-config : 66277 [48642.611835] hide [000000004a9766d1] apt-config : 66280 [48642.617971] unhide [000000004a9766d1] apt-config : 66280 [48642.636458] hide [000000009b5b3b8b] apt-config : 66282 [48642.645887] unhide [000000009b5b3b8b] apt-config : 66282 [48642.661886] hide [00000000b3651c7f] apt-config : 66284 [48642.671401] unhide [00000000b3651c7f] apt-config : 66284 [48642.716183] hide [00000000ccfe5ba6] apt-config : 66291 [48642.737750] unhide [00000000ccfe5ba6] apt-config : 66291 [48642.742863] hide [0000000076732b99] apt-key : 66295 [48642.752876] unhide [0000000076732b99] apt-key : 66295 [48642.781966] hide [00000000be6e08d1] apt-key : 66315 [48642.789060] unhide [00000000be6e08d1] apt-key : 66315 [48642.790194] hide [00000000a38c0833] apt-key : 66318 [48642.795633] unhide [00000000a38c0833] apt-key : 66318 [48642.824621] hide [00000000b1afd287] gpgconf : 66322 [48642.837727] unhide [00000000b1afd287] gpgconf : 66322 [48642.840893] unhide [0000000048aee5ea] apt-key : 66269 [48642.841904] unhide [00000000a3306607] gpgv : 66268 [48642.893622] hide [00000000481880f9] gpgv : 66327 [48642.897021] hide [000000009b000fec] apt-key : 66328 [48642.907413] hide [00000000ccc96ec1] apt-config : 66330 [48642.915000] unhide [00000000ccc96ec1] apt-config : 66330 [48642.926719] hide [00000000daae6a95] apt-config : 66332 [48642.935838] unhide [00000000daae6a95] apt-config : 66332 [48642.949563] hide [000000004fcc614f] apt-config : 66334 [48642.954960] unhide [000000004fcc614f] apt-config : 66334 [48642.970102] hide [0000000010b54c5a] apt-config : 66336 [48642.978178] unhide [0000000010b54c5a] apt-config : 66336 [48642.993005] hide [00000000a3fde702] apt-config : 66339 [48643.016706] unhide [00000000a3fde702] apt-config : 66339 [48643.033556] hide [00000000ec359c4c] apt-config : 66341 [48643.040196] unhide [00000000ec359c4c] apt-config : 66341 [48643.056757] hide [0000000015bc081d] apt-config : 66343 [48643.065214] unhide [0000000015bc081d] apt-config : 66343 [48643.095128] hide [00000000f7ac75b9] apt-config : 66350 [48643.103802] unhide [00000000f7ac75b9] apt-config : 66350 [48643.113721] hide [00000000919c0a82] apt-key : 66354 [48643.129351] unhide [00000000919c0a82] apt-key : 66354 [48643.179888] hide [000000001075784b] apt-key : 66382 [48643.185655] unhide [000000001075784b] apt-key : 66382 [48643.186133] hide [00000000e02a3052] apt-key : 66387 [48643.192034] unhide [00000000e02a3052] apt-key : 66387 [48643.207308] hide [00000000f8384847] gpgconf : 66393 [48643.232906] unhide [00000000f8384847] gpgconf : 66393 [48643.236186] unhide [000000009b000fec] apt-key : 66328 [48643.237266] unhide [00000000481880f9] gpgv : 66327 [48643.305519] hide [000000008c487193] gpgv : 66405 [48643.316743] hide [000000006f1fa135] apt-key : 66412 [48643.339368] hide [00000000c874cdfb] apt-config : 66415 [48643.347092] unhide [00000000c874cdfb] apt-config : 66415 [48643.368787] hide [000000005bbd0c37] apt-config : 66427 [48643.378925] unhide [000000005bbd0c37] apt-config : 66427 [48643.407096] hide [000000006443a7de] apt-config : 66437 [48643.426810] unhide [000000006443a7de] apt-config : 66437 [48643.456459] hide [000000000622bed4] apt-config : 66445 [48643.476744] unhide [000000000622bed4] apt-config : 66445 [48643.519352] hide [0000000048a82fca] apt-config : 66455 [48643.534396] unhide [0000000048a82fca] apt-config : 66455 [48643.574465] hide [000000000163d3d6] apt-config : 66460 [48643.587573] unhide [000000000163d3d6] apt-config : 66460 [48643.637081] hide [00000000b89c27a4] apt-config : 66467 [48643.661098] unhide [00000000b89c27a4] apt-config : 66467 [48643.748556] hide [000000004fb3e162] apt-config : 66485 [48643.765964] unhide [000000004fb3e162] apt-config : 66485 [48643.800700] hide [00000000768f931b] apt-key : 66508 [48643.825001] unhide [00000000768f931b] apt-key : 66508 [48643.868105] hide [00000000cde86745] apt-key : 66531 [48643.876736] unhide [00000000cde86745] apt-key : 66531 [48643.877269] hide [00000000a2c23a61] apt-key : 66534 [48643.890016] unhide [00000000a2c23a61] apt-key : 66534 [48643.918662] hide [00000000583eeb50] gpgconf : 66538 [48643.932910] unhide [00000000583eeb50] gpgconf : 66538 [48643.937947] unhide [000000006f1fa135] apt-key : 66412 [48643.940363] unhide [000000008c487193] gpgv : 66405 [48644.017344] hide [00000000642b481f] gpgv : 66543 [48644.020797] hide [0000000023c6604e] apt-key : 66544 [48644.059304] hide [00000000da9df529] apt-config : 66546 [48644.088865] unhide [00000000da9df529] apt-config : 66546 [48644.120916] hide [00000000ed79b4b9] apt-config : 66549 [48644.135226] unhide [00000000ed79b4b9] apt-config : 66549 [48644.155411] hide [000000002024ba79] apt-config : 66552 [48644.174181] unhide [000000002024ba79] apt-config : 66552 [48644.198262] hide [000000005d367529] apt-config : 66554 [48644.227643] unhide [000000005d367529] apt-config : 66554 [48644.252597] hide [00000000e5188357] apt-config : 66557 [48644.266894] unhide [00000000e5188357] apt-config : 66557 [48644.282406] hide [00000000010c06b0] apt-config : 66559 [48644.292455] unhide [00000000010c06b0] apt-config : 66559 [48644.308129] hide [0000000068f9059d] apt-config : 66561 [48644.316313] unhide [0000000068f9059d] apt-config : 66561 [48644.394946] hide [000000006306b5a1] apt-config : 66568 [48644.403114] unhide [000000006306b5a1] apt-config : 66568 [48644.413744] hide [0000000069325cf8] apt-key : 66572 [48644.421678] unhide [0000000069325cf8] apt-key : 66572 [48644.469387] hide [000000000f11e534] apt-key : 66592 [48644.480332] unhide [000000000f11e534] apt-key : 66592 [48644.480979] hide [0000000063128f14] apt-key : 66595 [48644.493356] unhide [0000000063128f14] apt-key : 66595 [48644.514754] hide [000000006c6b3a81] gpgconf : 66599 [48644.535782] unhide [000000006c6b3a81] gpgconf : 66599 [48644.538912] unhide [0000000023c6604e] apt-key : 66544 [48644.541427] unhide [00000000642b481f] gpgv : 66543 [48644.833497] unhide [000000003d896c83] gpgv : 66195 [48644.856517] hide [00000000e87ab74d] apt : 66604 [48644.858530] hide [00000000b2a3d2f4] sh : 66605 [48644.862826] unhide [00000000b2a3d2f4] sh : 66605 [48644.868997] hide [00000000da6f9e52] sh : 66608 [48644.920286] hide [0000000042bc15f0] gdbus : 66611 [48645.199046] unhide [0000000042bc15f0] gdbus : 66611 [48645.203865] unhide [00000000da6f9e52] sh : 66608 [48645.205820] hide [00000000202f245a] sh : 66620 [48645.596250] hide [00000000c85d63e0] appstreamcli : 66622 [48645.601106] hide [00000000033cf70d] tar : 66623 [48645.618747] unhide [00000000033cf70d] tar : 66623 [48645.624827] hide [000000004fbc557b] tar : 66625 [48645.640113] unhide [000000004fbc557b] tar : 66625 [48645.646401] hide [0000000083bb57e8] tar : 66627 [48645.655503] unhide [0000000083bb57e8] tar : 66627 [48645.667911] hide [000000007a9c311d] tar : 66629 [48645.676712] unhide [000000007a9c311d] tar : 66629 [48645.684624] hide [0000000081d17a63] tar : 66631 [48645.698427] unhide [0000000081d17a63] tar : 66631 [48645.701515] hide [0000000000ba946e] tar : 66633 [48645.709036] unhide [0000000000ba946e] tar : 66633 [48645.722283] hide [0000000077faaae1] tar : 66635 [48645.736286] unhide [0000000077faaae1] tar : 66635 [48645.744503] hide [000000006d4e2348] tar : 66637 [48645.759356] unhide [000000006d4e2348] tar : 66637 [48645.769937] hide [000000001a7eaf8d] tar : 66639 [48645.784670] unhide [000000001a7eaf8d] tar : 66639 [48645.788941] hide [000000009802a6bf] tar : 66641 [48645.792569] unhide [000000009802a6bf] tar : 66641 [48645.800074] hide [0000000097d5ab19] tar : 66643 [48645.805631] unhide [0000000097d5ab19] tar : 66643 [48645.810736] hide [00000000f0b29028] tar : 66645 [48645.815402] unhide [00000000f0b29028] tar : 66645 [48645.841662] hide [0000000070a640c7] tar : 66647 [48645.863876] unhide [0000000070a640c7] tar : 66647 [48645.870534] hide [00000000dda26c31] tar : 66649 [48645.882760] unhide [00000000dda26c31] tar : 66649 [48645.890901] hide [00000000e0fd289a] tar : 66651 [48645.907641] unhide [00000000e0fd289a] tar : 66651 [48645.927723] hide [0000000015e3d609] tar : 66653 [48645.962136] unhide [0000000015e3d609] tar : 66653 [48645.973356] hide [00000000daeb1417] tar : 66655 [48646.023972] unhide [00000000daeb1417] tar : 66655 [48646.029350] hide [00000000c06ddd11] tar : 66657 [48646.038825] unhide [00000000c06ddd11] tar : 66657 [48646.043100] hide [00000000dd28b026] tar : 66659 [48646.051426] unhide [00000000dd28b026] tar : 66659 [48646.058731] hide [00000000682488c4] tar : 66661 [48646.066481] unhide [00000000682488c4] tar : 66661 [48646.073168] hide [000000005ec3a8a7] tar : 66663 [48646.076257] unhide [000000005ec3a8a7] tar : 66663 [48646.086067] hide [00000000045258b2] tar : 66665 [48646.095307] unhide [00000000045258b2] tar : 66665 [48646.129127] hide [000000007597066a] tar : 66667 [48646.134961] unhide [000000007597066a] tar : 66667 [48646.147748] hide [00000000695a4be8] tar : 66669 [48646.156043] unhide [00000000695a4be8] tar : 66669 [48646.160465] hide [00000000d0c62463] tar : 66671 [48646.163933] unhide [00000000d0c62463] tar : 66671 [48646.168516] hide [00000000c100eac0] tar : 66673 [48646.172676] unhide [00000000c100eac0] tar : 66673 [48646.174965] hide [000000003411d3e6] tar : 66675 [48646.178355] unhide [000000003411d3e6] tar : 66675 [48646.459699] hide [0000000052f15e78] tar : 66677 [48646.554032] unhide [0000000052f15e78] tar : 66677 [48646.557808] hide [00000000fa0c72d5] tar : 66679 [48646.701415] unhide [00000000fa0c72d5] tar : 66679 [48646.705218] hide [000000005265dbba] tar : 66681 [48646.709939] unhide [000000005265dbba] tar : 66681 [48646.716245] hide [00000000276a70c9] tar : 66683 [48646.721206] unhide [00000000276a70c9] tar : 66683 [48646.727560] hide [00000000e533e811] tar : 66685 [48646.737842] unhide [00000000e533e811] tar : 66685 [48646.740519] hide [00000000a061df66] tar : 66687 [48646.744049] unhide [00000000a061df66] tar : 66687 [48650.011886] unhide [00000000c85d63e0] appstreamcli : 66622 [48650.015225] unhide [00000000202f245a] sh : 66620 [48650.020180] hide [000000004427d165] sh : 66689 [48650.130065] hide [0000000035c9800c] cnf-update-db : 66691 [48650.147647] unhide [0000000035c9800c] cnf-update-db : 66691 [48650.148661] unhide [000000004427d165] sh : 66689 [48650.149553] hide [00000000f11b4223] sh : 66693 [48650.152495] hide [000000009d7797ed] update-motd-upd : 66694 [48650.157607] hide [00000000371dc283] apt-config : 66695 [48650.162067] unhide [00000000371dc283] apt-config : 66695 [48650.170281] hide [000000004f0475ff] apt-config : 66697 [48650.174022] unhide [000000004f0475ff] apt-config : 66697 [48650.181465] hide [00000000a686f6cd] apt-config : 66699 [48650.185655] unhide [00000000a686f6cd] apt-config : 66699 [48650.193179] hide [00000000217c9f47] apt-config : 66701 [48650.198583] unhide [00000000217c9f47] apt-config : 66701 [48650.204602] hide [00000000ad695606] apt-config : 66703 [48650.210384] unhide [00000000ad695606] apt-config : 66703 [48650.229054] hide [0000000081abc30b] update-motd-upd : 66706 [48650.239898] unhide [0000000081abc30b] mktemp : 66706 [48650.370807] hide [0000000072a2a9a5] apt-check : 66708 [48655.627026] unhide [00000000b8836db6] openssl : 65789 [48655.627179] unhide [00000000076dd9bd] sh : 65788 [48655.627229] unhide [00000000b4bd58aa] bash : 65786 [48658.196415] Kthread : task 0000000043fba6fd : irq/102_pciehp : pid 65778 : group 65778 [48658.196417] Kthread : task 000000002b5728b3 : irq/101_pciehp : pid 65777 : group 65777 [48658.196419] Kthread : task 0000000003c4dbfa : irq/100_pciehp : pid 65776 : group 65776 [48658.196421] Task : apt : ino 786482 : task 00000000bd8d9666 : apt : pid 66171 : group 66171 [48658.196422] Task : apt : ino 786482 : task 00000000e87ab74d : apt : pid 66604 : group 66604 [48658.196424] Task : dash : ino 786638 : task 00000000f11b4223 : sh : pid 66693 : group 66693 [48658.196425] Task : dash : ino 786638 : task 000000009d7797ed : update-motd-upd : pid 66694 : group 66694 [48658.196426] Task : python3.8 : ino 797711 : task 0000000072a2a9a5 : apt-check : pid 66708 : group 66708 [48666.052553] Kthread : task 0000000043fba6fd : irq/102_pciehp : pid 65778 : group 65778 [48666.052556] Kthread : task 000000002b5728b3 : irq/101_pciehp : pid 65777 : group 65777 [48666.052558] Kthread : task 0000000003c4dbfa : irq/100_pciehp : pid 65776 : group 65776 [48666.052561] Task : apt : ino 786482 : task 00000000bd8d9666 : apt : pid 66171 : group 66171 [48666.052563] Task : apt : ino 786482 : task 00000000e87ab74d : apt : pid 66604 : group 66604 [48666.052564] Task : dash : ino 786638 : task 00000000f11b4223 : sh : pid 66693 : group 66693 [48666.052566] Task : dash : ino 786638 : task 000000009d7797ed : update-motd-upd : pid 66694 : group 66694 [48666.052568] Task : python3.8 : ino 797711 : task 0000000072a2a9a5 : apt-check : pid 66708 : group 66708 [48667.093761] unhide [0000000072a2a9a5] apt-check : 66708 [48667.122384] unhide [000000009d7797ed] update-motd-upd : 66694 [48667.122622] unhide [00000000f11b4223] sh : 66693 [48667.122759] unhide [00000000e87ab74d] apt : 66604 [48669.114289] Kthread : task 0000000043fba6fd : irq/102_pciehp : pid 65778 : group 65778 [48669.114290] Kthread : task 000000002b5728b3 : irq/101_pciehp : pid 65777 : group 65777 [48669.114291] Kthread : task 0000000003c4dbfa : irq/100_pciehp : pid 65776 : group 65776 [48669.114293] Task : apt : ino 786482 : task 00000000bd8d9666 : apt : pid 66171 : group 66171 [48676.950426] hide [0000000037067c40] apt : 66783 [48676.952501] hide [00000000b22cf14a] sh : 66784 [48676.966597] hide [00000000f5c42642] apt-esm-hook : 66785 [48677.104531] unhide [00000000f5c42642] apt-esm-hook : 66785 [48677.105113] unhide [00000000b22cf14a] sh : 66784 [48677.105323] unhide [0000000037067c40] apt : 66783 [48677.115974] unhide [00000000bd8d9666] apt : 66171 [48679.842847] Kthread : task 0000000043fba6fd : irq/102_pciehp : pid 65778 : group 65778 [48679.842852] Kthread : task 000000002b5728b3 : irq/101_pciehp : pid 65777 : group 65777 [48679.842855] Kthread : task 0000000003c4dbfa : irq/100_pciehp : pid 65776 : group 65776 [48683.875037] Kthread : task 0000000043fba6fd : irq/102_pciehp : pid 65778 : group 65778 [48683.875041] Kthread : task 000000002b5728b3 : irq/101_pciehp : pid 65777 : group 65777 [48683.875043] Kthread : task 0000000003c4dbfa : irq/100_pciehp : pid 65776 : group 65776 [48696.965274] Got event [48697.075176] hide [0000000097fa9365] openssl : 66791 [48697.075219] hide [00000000f6b13f8c] sh : 66790 [48697.075226] hide [00000000f84b06b5] bash : 66788 [48697.075231] Waiting for event [48697.075233] Got event [48697.075233] Waiting for event [48699.833176] Kthread : task 0000000043fba6fd : irq/102_pciehp : pid 65778 : group 65778 [48699.833182] Kthread : task 000000002b5728b3 : irq/101_pciehp : pid 65777 : group 65777 [48699.833185] Kthread : task 0000000003c4dbfa : irq/100_pciehp : pid 65776 : group 65776 [48699.833192] BD : openssl : ino 796831 : task 0000000097fa9365 : openssl : pid 66791 : group 66788 [48699.833215] BD : dash : ino 786638 : task 00000000f6b13f8c : sh : pid 66790 : group 66788 [48699.833219] BD : bash : ino 786526 : task 00000000f84b06b5 : bash : pid 66788 : group 66788

iusearch commented 1 year ago

Is it on ubuntu 20.04?

iusearch commented 1 year ago

I can also reproduce it on a vultr machine, but it seems it has a fault but not crash

[  +2.848384] hide [00000000d3d91a70] apt : 3872
[  +0.344743] hide [000000006f9424fd] sh : 3875
[  +0.008307] hide [0000000037279305] apt-esm-json-ho : 3876
[  +0.000558] hide [00000000d7006337] sh : 3877
[  +0.011800] unhide [00000000d7006337] sh : 3877
[  +0.004043] unhide [00000000d3d91a70] apt : 3872
[  +0.063892] unhide [0000000037279305] apt-esm-json-ho : 3876
[  +0.000254] unhide [000000006f9424fd] sh : 3875
[  +3.742155] hide [000000002cc79c2f] apt : 3885
[  +0.367222] hide [00000000451df44f] sh : 3888
[  +0.011155] hide [000000002676052b] apt-esm-json-ho : 3889
[  +0.000721] hide [0000000057d73cbf] sh : 3890
[  +0.000008] general protection fault: 0000 [#1] SMP NOPTI
[  +0.000017] CPU: 0 PID: 11 Comm: migration/0 Tainted: G            E     5.4.0-163-generic #180-Ubuntu
[  +0.000015] Hardware name: Vultr VHP, BIOS  
[  +0.000011] RIP: 0010:_hide_task+0x67/0x100 [kovid]
[  +0.000009] Code: c0 75 19 b8 f4 ff ff ff 5b 41 5c 41 5d 5d c3 b8 f2 ff ff ff eb f2 b8 ea ff ff ff c3 49 8b 85 38 09 00 00 49 8b 95 40 09 00 00 <48> 89 02 48 85 c0 74 04 48 89 50 08 48 b8 00 01 00 00 00 00 ad de
[  +0.000028] RSP: 0018:ffffb8c500073e30 EFLAGS: 00010086
[  +0.000009] RAX: 0000000000000000 RBX: ffffb8c500a33ea0 RCX: 0000000000000000
[  +0.000011] RDX: 000fffffc0010200 RSI: ffffffffc0973c08 RDI: ffff8d9ffc801700
[  +0.000011] RBP: ffffb8c500073e48 R08: ffff8d9ffce350a0 R09: ffff8d9ffc801700
[  +0.000011] R10: ffff8d9ff1245a18 R11: ffff8d9ffce2ffb8 R12: ffff8d9ff23a2ac0
[  +0.000011] R13: ffffe14580c5eac0 R14: 0000000000000003 R15: ffffb8c500a33e18
[  +0.000013] FS:  0000000000000000(0000) GS:ffff8d9ffce00000(0000) knlGS:0000000000000000
[  +0.000013] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  +0.000509] CR2: 00005607aa079c20 CR3: 000000003b14e000 CR4: 0000000000140ef0
[  +0.000471] Call Trace:
[  +0.000482]  ? show_regs.cold+0x1a/0x1f
[  +0.000496]  ? __die+0x90/0xd9
[  +0.000713]  ? die+0x30/0x50
[  +0.000457]  ? do_general_protection+0xcc/0x160
[  +0.000450]  ? general_protection+0x28/0x30
[  +0.000445]  ? _hide_task+0x38/0x100 [kovid]
[  +0.000455]  ? _hide_task+0x67/0x100 [kovid]
[  +0.000439]  multi_cpu_stop+0x71/0x110
[  +0.000448]  ? stop_machine_yield+0x10/0x10
[  +0.000546]  cpu_stopper_thread+0x50/0x100
[  +0.000556]  smpboot_thread_fn+0xd0/0x170
[  +0.000433]  kthread+0x104/0x140
[  +0.000433]  ? sort_range+0x30/0x30
[  +0.000434]  ? kthread_park+0x90/0x90
[  +0.000451]  ret_from_fork+0x35/0x40
[  +0.000434] Modules linked in: kovid(E) tcp_bbr sch_fq nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua binfmt_misc snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec snd_hda_core snd_hwdep input_leds serio_raw snd_pcm snd_timer snd soundcore joydev qemu_fw_cfg mac_hid nf_log_ipv6 ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt nf_log_ipv4 nf_log_common ipt_REJECT nf_reject_ipv4 sch_fq_codel xt_LOG mlx5_core xt_limit xt_addrtype tls mlxfw pci_hyperv_intf xt_tcpudp xt_conntrack nf_conntrack ixgbe ramoops nf_defrag_ipv6 nf_defrag_ipv4 reed_solomon xfrm_algo mdio efi_pstore dca ip6table_filter ip6_tables iptable_filter bnxt_en bpfilter msr virtio_rng ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mpt3sas raid_class scsi_transport_sas megaraid_sas crct10dif_pclmul crc32_pclmul hid_generic usbhid ghash_clmulni_intel hid aesni_intel
[  +0.000023]  bochs_drm crypto_simd drm_vram_helper cryptd glue_helper ttm psmouse ahci libahci i2c_i801 lpc_ich drm_kms_helper virtio_net syscopyarea net_failover sysfillrect virtio_blk sysimgblt failover fb_sys_fops drm
[  +0.004525] ---[ end trace 14df4e7c2c6bbd72 ]---
[  +0.357646] RIP: 0010:_hide_task+0x67/0x100 [kovid]
[  +0.000564] Code: c0 75 19 b8 f4 ff ff ff 5b 41 5c 41 5d 5d c3 b8 f2 ff ff ff eb f2 b8 ea ff ff ff c3 49 8b 85 38 09 00 00 49 8b 95 40 09 00 00 <48> 89 02 48 85 c0 74 04 48 89 50 08 48 b8 00 01 00 00 00 00 ad de
[  +0.001012] RSP: 0018:ffffb8c500073e30 EFLAGS: 00010086
[  +0.000468] RAX: 0000000000000000 RBX: ffffb8c500a33ea0 RCX: 0000000000000000
[  +0.000468] RDX: 000fffffc0010200 RSI: ffffffffc0973c08 RDI: ffff8d9ffc801700
[  +0.000453] RBP: ffffb8c500073e48 R08: ffff8d9ffce350a0 R09: ffff8d9ffc801700
[  +0.000456] R10: ffff8d9ff1245a18 R11: ffff8d9ffce2ffb8 R12: ffff8d9ff23a2ac0
[  +0.000433] R13: ffffe14580c5eac0 R14: 0000000000000003 R15: ffffb8c500a33e18
[  +0.000438] FS:  0000000000000000(0000) GS:ffff8d9ffce00000(0000) knlGS:0000000000000000
[  +0.000434] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  +0.000429] CR2: 00005607aa079c20 CR3: 000000003b14e000 CR4: 0000000000140ef0
[  +0.000444] note: migration/0[11] exited with preempt_count 1

This is by apt upgrade. Not quite the same stack trace so I guess not really the same issue.

carloslack commented 1 year ago

I see, I still need to get my hands on the same exact version you are using and check 5.4.0-163 Thanks

carloslack commented 1 year ago

@iusearch could run make and share the full output? Maybe that will give us a hint

iusearch commented 1 year ago

For vultr

root@vultr:~/KoviD# make
/usr/bin/as --64 src/persist.S -statistics -fatal-warnings \
        -size-check=error -o src/persist.o
/usr/bin/as: total time in assembly: 0.001709
frag chains:

        0x563261041ea0 .text                     5 frags

        0x563261041f38 .data                     2 frags

        0x563261041fd0 .bss                      2 frags
fixups: 5
1 mini local symbols created, 0 converted
/usr/bin/ld -Ttext 200000 --oformat binary -o src/persist src/persist.o
make  -C  /lib/modules/5.4.0-163-generic/build M=/root/KoviD modules
make[1]: Entering directory '/usr/src/linux-headers-5.4.0-163-generic'
  CC [M]  /root/KoviD/src/pid.o
In file included from /root/KoviD/src/pid.c:17:
/root/KoviD/src/lkm.h:12:9: note: #pragma message: !!! Be careful: Build kovid in DEBUG mode !!!
   12 | #pragma message "!!! Be careful: Build kovid in DEBUG mode !!!"
      |         ^~~~~~~
  CC [M]  /root/KoviD/src/fs.o
In file included from /root/KoviD/src/fs.c:18:
/root/KoviD/src/lkm.h:12:9: note: #pragma message: !!! Be careful: Build kovid in DEBUG mode !!!
   12 | #pragma message "!!! Be careful: Build kovid in DEBUG mode !!!"
      |         ^~~~~~~
  CC [M]  /root/KoviD/src/sys.o
In file included from /root/KoviD/src/sys.c:19:
/root/KoviD/src/lkm.h:12:9: note: #pragma message: !!! Be careful: Build kovid in DEBUG mode !!!
   12 | #pragma message "!!! Be careful: Build kovid in DEBUG mode !!!"
      |         ^~~~~~~
/root/KoviD/src/sys.c: In function ‘m_execve’:
/root/KoviD/src/sys.c:34:26: warning: initialization of ‘const char * const*’ from ‘long unsigned int’ makes pointer from integer without a cast [-Wint-conversion]
   34 | #define PT_REGS_PARM2(x) ((x)->si)
      |                          ^
/root/KoviD/src/sys.c:360:49: note: in expansion of macro ‘PT_REGS_PARM2’
  360 |     struct user_arg_ptr argvx = { .ptr.native = PT_REGS_PARM2(regs) };
      |                                                 ^~~~~~~~~~~~~
/root/KoviD/src/sys.c:34:26: note: (near initialization for ‘argvx.ptr.native’)
   34 | #define PT_REGS_PARM2(x) ((x)->si)
      |                          ^
/root/KoviD/src/sys.c:360:49: note: in expansion of macro ‘PT_REGS_PARM2’
  360 |     struct user_arg_ptr argvx = { .ptr.native = PT_REGS_PARM2(regs) };
      |                                                 ^~~~~~~~~~~~~
/root/KoviD/src/sys.c: At top level:
/root/KoviD/src/sys.c:452:9: note: #pragma message: tcp6_seq_show untested
  452 | #pragma message "tcp6_seq_show untested"
      |         ^~~~~~~
/root/KoviD/src/sys.c:467:9: note: #pragma message: udp6_seq_show untested
  467 | #pragma message "udp6_seq_show untested"
      |         ^~~~~~~
/root/KoviD/src/sys.c: In function ‘_tty_write_log’:
/root/KoviD/src/sys.c:619:5: warning: ISO C90 forbids variable length array ‘ttybuf’ [-Wvla]
  619 |     char ttybuf[len+16];
      |     ^~~~
  CC [M]  /root/KoviD/src/sock.o
In file included from /root/KoviD/src/sock.c:21:
/root/KoviD/src/lkm.h:12:9: note: #pragma message: !!! Be careful: Build kovid in DEBUG mode !!!
   12 | #pragma message "!!! Be careful: Build kovid in DEBUG mode !!!"
      |         ^~~~~~~
  CC [M]  /root/KoviD/src/util.o
In file included from /root/KoviD/src/util.c:16:
/root/KoviD/src/lkm.h:12:9: note: #pragma message: !!! Be careful: Build kovid in DEBUG mode !!!
   12 | #pragma message "!!! Be careful: Build kovid in DEBUG mode !!!"
      |         ^~~~~~~
  CC [M]  /root/KoviD/src/vm.o
In file included from /root/KoviD/src/vm.c:4:
/root/KoviD/src/lkm.h:12:9: note: #pragma message: !!! Be careful: Build kovid in DEBUG mode !!!
   12 | #pragma message "!!! Be careful: Build kovid in DEBUG mode !!!"
      |         ^~~~~~~
  LD [M]  /root/KoviD/kovid.o
  Building modules, stage 2.
  MODPOST 1 modules
  LD [M]  /root/KoviD/kovid.ko
make[1]: Leaving directory '/usr/src/linux-headers-5.4.0-163-generic'

For virtual machine

grider@ubuntu-qemu:~/KoviD$ make
/usr/bin/as --64 src/persist.S -statistics -fatal-warnings \
        -size-check=error -o src/persist.o
/usr/bin/as: total time in assembly: 0.006155
frag chains:

        0x55894655bea0 .text                     5 frags

        0x55894655bf38 .data                     2 frags

        0x55894655bfd0 .bss                      2 frags
fixups: 5
1 mini local symbols created, 0 converted
/usr/bin/ld -Ttext 200000 --oformat binary -o src/persist src/persist.o
make  -C  /lib/modules/5.15.0-86-generic/build M=/home/grider/KoviD modules
make[1]: Entering directory '/usr/src/linux-headers-5.15.0-86-generic'
  CC [M]  /home/grider/KoviD/src/kovid.o
In file included from /home/grider/KoviD/src/kovid.c:26:
/home/grider/KoviD/src/lkm.h:12:9: note: #pragma message: !!! Be careful: Build kovid in DEBUG mode !!!
   12 | #pragma message "!!! Be careful: Build kovid in DEBUG mode !!!"
      |         ^~~~~~~
/home/grider/KoviD/src/kovid.c:78:9: note: #pragma message: !! Warning: Unsupported kernel version GOOD LUCK WITH THAT! !!
   78 | #pragma message "!! Warning: Unsupported kernel version GOOD LUCK WITH THAT! !!"
      |         ^~~~~~~
  CC [M]  /home/grider/KoviD/src/pid.o
In file included from /home/grider/KoviD/src/pid.c:17:
/home/grider/KoviD/src/lkm.h:12:9: note: #pragma message: !!! Be careful: Build kovid in DEBUG mode !!!
   12 | #pragma message "!!! Be careful: Build kovid in DEBUG mode !!!"
      |         ^~~~~~~
  CC [M]  /home/grider/KoviD/src/fs.o
In file included from /home/grider/KoviD/src/fs.c:18:
/home/grider/KoviD/src/lkm.h:12:9: note: #pragma message: !!! Be careful: Build kovid in DEBUG mode !!!
   12 | #pragma message "!!! Be careful: Build kovid in DEBUG mode !!!"
      |         ^~~~~~~
  CC [M]  /home/grider/KoviD/src/sys.o
In file included from /home/grider/KoviD/src/sys.c:19:
/home/grider/KoviD/src/lkm.h:12:9: note: #pragma message: !!! Be careful: Build kovid in DEBUG mode !!!
   12 | #pragma message "!!! Be careful: Build kovid in DEBUG mode !!!"
      |         ^~~~~~~
/home/grider/KoviD/src/sys.c: In function ‘m_execve’:
/home/grider/KoviD/src/sys.c:34:26: warning: initialization of ‘const char * const*’ from ‘long unsigned int’ makes pointer from integer without a cast [-Wint-conversion]
   34 | #define PT_REGS_PARM2(x) ((x)->si)
      |                          ^
/home/grider/KoviD/src/sys.c:360:49: note: in expansion of macro ‘PT_REGS_PARM2’
  360 |     struct user_arg_ptr argvx = { .ptr.native = PT_REGS_PARM2(regs) };
      |                                                 ^~~~~~~~~~~~~
/home/grider/KoviD/src/sys.c:34:26: note: (near initialization for ‘argvx.ptr.native’)
   34 | #define PT_REGS_PARM2(x) ((x)->si)
      |                          ^
/home/grider/KoviD/src/sys.c:360:49: note: in expansion of macro ‘PT_REGS_PARM2’
  360 |     struct user_arg_ptr argvx = { .ptr.native = PT_REGS_PARM2(regs) };
      |                                                 ^~~~~~~~~~~~~
/home/grider/KoviD/src/sys.c: At top level:
/home/grider/KoviD/src/sys.c:452:9: note: #pragma message: tcp6_seq_show untested
  452 | #pragma message "tcp6_seq_show untested"
      |         ^~~~~~~
/home/grider/KoviD/src/sys.c:467:9: note: #pragma message: udp6_seq_show untested
  467 | #pragma message "udp6_seq_show untested"
      |         ^~~~~~~
/home/grider/KoviD/src/sys.c: In function ‘_tty_write_log’:
/home/grider/KoviD/src/sys.c:619:5: warning: ISO C90 forbids variable length array ‘ttybuf’ [-Wvla]
  619 |     char ttybuf[len+16];
      |     ^~~~
/home/grider/KoviD/src/sys.o: warning: objtool: m_audit_log_start()+0x3f: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: _get_sys_addr.cold()+0xd: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: _keylog_cleanup_list()+0x5a: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: m_account_system_time()+0x13: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: m_account_process_tick()+0xd: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: m_tpacket_rcv()+0x61: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: m_packet_rcv()+0x61: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: m_udp6_seq_show()+0x55: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: m_tcp6_seq_show()+0x55: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: m_udp4_seq_show()+0x55: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: m_tcp4_seq_show()+0x55: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: m_filldir64()+0x24: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: m_filldir()+0x24: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: m_exit_group()+0x3c: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: _sys_file.part.0.constprop.0()+0x36: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: sys_ttyfile.part.0()+0x13: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: bpf_map_value_size.isra.0()+0x2f: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: _load_syscall_variant.constprop.0.cold()+0x56: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: kv_kall_load_addr.part.0()+0x17: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: m_tty_read()+0x3c: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: _find_tcp4udp4_match_cb()+0xa1: BP used as a scratch register
/home/grider/KoviD/src/sys.o: warning: objtool: m_clone()+0x42: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: _keylog_cleanup()+0x6: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: kv_kall_load_addr()+0x1b: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: m_bpf()+0x4c: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: m_kill()+0xe8: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: fh_install_hook()+0xab: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: fh_remove_hook()+0x20: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: fh_install_hooks.cold()+0x67: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: fh_remove_hooks()+0x19: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: sys_ttyfile()+0x1a: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: sys_sslfile()+0x28: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: sys_init()+0x9a: call without frame pointer save/setup
/home/grider/KoviD/src/sys.o: warning: objtool: sys_deinit()+0x9: call without frame pointer save/setup
  CC [M]  /home/grider/KoviD/src/sock.o
In file included from /home/grider/KoviD/src/sock.c:21:
/home/grider/KoviD/src/lkm.h:12:9: note: #pragma message: !!! Be careful: Build kovid in DEBUG mode !!!
   12 | #pragma message "!!! Be careful: Build kovid in DEBUG mode !!!"
      |         ^~~~~~~
  CC [M]  /home/grider/KoviD/src/util.o
In file included from /home/grider/KoviD/src/util.c:16:
/home/grider/KoviD/src/lkm.h:12:9: note: #pragma message: !!! Be careful: Build kovid in DEBUG mode !!!
   12 | #pragma message "!!! Be careful: Build kovid in DEBUG mode !!!"
      |         ^~~~~~~
  CC [M]  /home/grider/KoviD/src/vm.o
In file included from /home/grider/KoviD/src/vm.c:4:
/home/grider/KoviD/src/lkm.h:12:9: note: #pragma message: !!! Be careful: Build kovid in DEBUG mode !!!
   12 | #pragma message "!!! Be careful: Build kovid in DEBUG mode !!!"
      |         ^~~~~~~
  LD [M]  /home/grider/KoviD/kovid.o
  MODPOST /home/grider/KoviD/Module.symvers
  CC [M]  /home/grider/KoviD/kovid.mod.o
  LD [M]  /home/grider/KoviD/kovid.ko
  BTF [M] /home/grider/KoviD/kovid.ko
Skipping BTF generation for /home/grider/KoviD/kovid.ko due to unavailability of vmlinux
make[1]: Leaving directory '/usr/src/linux-headers-5.15.0-86-generic'
carloslack commented 1 year ago

@iusearch I've managed to install the exact same kernel:

hash@ubuntu:~/Codes/KoviD$ uname -a Linux ubuntu 5.4.0-164-generic #181-Ubuntu SMP Fri Sep 1 13:41:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

However, I am unable to reproduce the crash you've seen. I don't have the whole picture because I don't know your setup, but as far as 5.4.0-164 is concerned, it's been working for me without issues.

In other news, I found a bpf incompatibility for that old kernel that I attempt to address in kvdev branch (could not test just yet) and you could give a try, although I don't think it will solve your issue.

https://github.com/carloslack/KoviD/commit/0a2eb64750de3201609a7e60116c75c5ee61f61a

iusearch commented 1 year ago

May sounds a bit count intuitive but the issue seems not only relating to kernel. I booted a working kernel from 22.04 on 20.04, crash still happens. Might be QEMU setup but unsure about that. I will try your patch first

iusearch commented 1 year ago

If you'd like to, I'm willing to share my virtual machine disk image so you can see if the same setup happens on your machine

carloslack commented 1 year ago

yeah, at this moment, that's what the tests are indicating.,

carloslack commented 1 year ago

If you'd like to, I'm willing to share my virtual machine disk image so you can see if the same setup happens on your machine

If you want to share that's good, maybe a dropbox or something, I just don't promisse to test it too soon

iusearch commented 1 year ago

If you'd like to, I'm willing to share my virtual machine disk image so you can see if the same setup happens on your machine

If you want to share that's good, maybe a dropbox or something, I just don't promisse to test it too soon

No problem at atll. Thanks a lot for help. I uploaded image to Mega.

carloslack commented 1 year ago

https://github.com/carloslack/KoviD/commit/0a2eb64750de3201609a7e60116c75c5ee61f61a