Kernel NULL pointer dereference shortly after booting

[mamarley]

This morning shortly after I rebooted one of my servers with ksmbd, I got the following NULL pointer dereference. I'm unsure of exactly what might have triggered this regarding client activity, but the server had just rebooted within a few minutes ago. The clients are all Linux boxes using cifsd.ko.

Apr 14 06:31:09 <redacted> kernel: BUG: kernel NULL pointer dereference, address: 0000000000000010
Apr 14 06:31:09 <redacted> kernel: #PF: supervisor read access in kernel mode
Apr 14 06:31:09 <redacted> kernel: #PF: error_code(0x0000) - not-present page
Apr 14 06:31:09 <redacted> kernel: PGD 0 P4D 0 
Apr 14 06:31:09 <redacted> kernel: Oops: 0000 [#1] PREEMPT SMP PTI
Apr 14 06:31:09 <redacted> kernel: CPU: 5 PID: 1354 Comm: kworker/5:19 Not tainted 5.17.3-051703-generic #202204131854
Apr 14 06:31:09 <redacted> kernel: Hardware name: LENOVO 10ANS00G00/SHARKBAY, BIOS FBKTE0AUS 12/23/2021
Apr 14 06:31:09 <redacted> kernel: Workqueue: ksmbd-io handle_ksmbd_work [ksmbd]
Apr 14 06:31:09 <redacted> kernel: RIP: 0010:idr_remove+0x1/0x20
Apr 14 06:31:09 <redacted> kernel: Code: 6d cc 76 dc 45 89 f8 4c 89 e1 48 8d 55 cc 4c 89 f6 48 89 df 44 89 6d cc e8 2c fe ff ff eb b4 e8 f5 c2 66 00 0f 1f 44 00 00 55 <8b> 47 10 31 d2 48 29 c6 48 89 e5 e8 af 54 00 00 5d c3 66 66 2e 0f
Apr 14 06:31:09 <redacted> kernel: RSP: 0018:ffffa48d00e43d20 EFLAGS: 00010246
Apr 14 06:31:09 <redacted> kernel: RAX: 0000000000000000 RBX: ffff91aead05fa00 RCX: ffff91ae939295b8
Apr 14 06:31:09 <redacted> kernel: RDX: 00000000000000ff RSI: 0000000000000000 RDI: 0000000000000000
Apr 14 06:31:09 <redacted> kernel: RBP: ffffa48d00e43d60 R08: ffff91ae92d7dcc0 R09: ffff91ae884b27e0
Apr 14 06:31:09 <redacted> kernel: R10: 0000000000000000 R11: ffff91ae8e1db6e8 R12: ffff91ae9499d108
Apr 14 06:31:09 <redacted> kernel: R13: 0000000000000000 R14: ffff91aed078be04 R15: ffff91ae92d7dcc0
Apr 14 06:31:09 <redacted> kernel: FS:  0000000000000000(0000) GS:ffff91b18eb40000(0000) knlGS:0000000000000000
Apr 14 06:31:09 <redacted> kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 14 06:31:09 <redacted> kernel: CR2: 0000000000000010 CR3: 00000001138be003 CR4: 00000000001706e0
Apr 14 06:31:09 <redacted> kernel: Call Trace:
Apr 14 06:31:09 <redacted> kernel:  <TASK>
Apr 14 06:31:09 <redacted> kernel:  ? __ksmbd_close_fd+0xb2/0x2d0 [ksmbd]
Apr 14 06:31:09 <redacted> kernel:  ? ksmbd_vfs_read+0x91/0x190 [ksmbd]
Apr 14 06:31:09 <redacted> kernel:  ksmbd_fd_put+0x29/0x40 [ksmbd]
Apr 14 06:31:09 <redacted> kernel:  smb2_read+0x210/0x390 [ksmbd]
Apr 14 06:31:09 <redacted> kernel:  __process_request+0xa4/0x180 [ksmbd]
Apr 14 06:31:09 <redacted> kernel:  __handle_ksmbd_work+0xf0/0x290 [ksmbd]
Apr 14 06:31:09 <redacted> kernel:  handle_ksmbd_work+0x2d/0x50 [ksmbd]
Apr 14 06:31:09 <redacted> kernel:  process_one_work+0x21d/0x3f0
Apr 14 06:31:09 <redacted> kernel:  worker_thread+0x50/0x3d0
Apr 14 06:31:09 <redacted> kernel:  ? rescuer_thread+0x390/0x390
Apr 14 06:31:09 <redacted> kernel:  kthread+0xee/0x120
Apr 14 06:31:09 <redacted> kernel:  ? kthread_complete_and_exit+0x20/0x20
Apr 14 06:31:09 <redacted> kernel:  ret_from_fork+0x22/0x30
Apr 14 06:31:09 <redacted> kernel:  </TASK>
Apr 14 06:31:09 <redacted> kernel: Modules linked in: cmac nls_utf8 ksmbd crc32_generic rdma_cm iw_cm ib_cm ib_core cifs_arc4 nf_conntrack_ftp nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_masq nft_redir nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink 8021q garp mrp stp llc intel_rapl_msr mei_pxp mei_hdcp nls_iso8859_1 intel_rapl_common x86_pkg_temp_thermal snd_hda_codec_realtek intel_powerclamp snd_hda_codec_generic ledtrig_audio kvm_intel kvm crct10dif_pclmul ghash_clmulni_intel aesni_intel crypto_simd snd_hda_codec_hdmi cryptd rapl ftdi_sio snd_hda_intel think_lmi usbserial efi_pstore intel_cstate wmi_bmof snd_intel_dspcfg firmware_attributes_class joydev input_leds at24 snd_intel_sdw_acpi snd_hda_codec sit tunnel4 ip_tunnel snd_hda_core snd_hwdep snd_pcm snd_timer snd mei_me mac_hid mei soundcore sch_fq_codel it87 hwmon_vid drivetemp coretemp tls iTCO_wdt intel_pmc_bxt iTCO_vendor_support autofs4 raid10 raid1 raid0 multipath linear raid456
Apr 14 06:31:09 <redacted> kernel:  async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c i915 hid_generic i2c_algo_bit drm_kms_helper usbhid syscopyarea sysfillrect hid sysimgblt fb_sys_fops cec rc_core gpio_ich ixgbe ttm xfrm_algo i2c_i801 ahci xhci_pci dca crc32_pclmul drm e1000e libahci lpc_ich i2c_smbus xhci_pci_renesas mdio wmi video
Apr 14 06:31:09 <redacted> kernel: CR2: 0000000000000010
Apr 14 06:31:09 <redacted> kernel: ---[ end trace 0000000000000000 ]---
Apr 14 06:31:09 <redacted> kernel: RIP: 0010:idr_remove+0x1/0x20
Apr 14 06:31:09 <redacted> kernel: Code: 6d cc 76 dc 45 89 f8 4c 89 e1 48 8d 55 cc 4c 89 f6 48 89 df 44 89 6d cc e8 2c fe ff ff eb b4 e8 f5 c2 66 00 0f 1f 44 00 00 55 <8b> 47 10 31 d2 48 29 c6 48 89 e5 e8 af 54 00 00 5d c3 66 66 2e 0f
Apr 14 06:31:09 <redacted> kernel: RSP: 0018:ffffa48d00e43d20 EFLAGS: 00010246
Apr 14 06:31:09 <redacted> kernel: RAX: 0000000000000000 RBX: ffff91aead05fa00 RCX: ffff91ae939295b8
Apr 14 06:31:09 <redacted> kernel: RDX: 00000000000000ff RSI: 0000000000000000 RDI: 0000000000000000
Apr 14 06:31:09 <redacted> kernel: RBP: ffffa48d00e43d60 R08: ffff91ae92d7dcc0 R09: ffff91ae884b27e0
Apr 14 06:31:09 <redacted> kernel: R10: 0000000000000000 R11: ffff91ae8e1db6e8 R12: ffff91ae9499d108
Apr 14 06:31:09 <redacted> kernel: R13: 0000000000000000 R14: ffff91aed078be04 R15: ffff91ae92d7dcc0
Apr 14 06:31:09 <redacted> kernel: FS:  0000000000000000(0000) GS:ffff91b18eb40000(0000) knlGS:0000000000000000
Apr 14 06:31:09 <redacted> kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 14 06:31:09 <redacted> kernel: CR2: 0000000000000010 CR3: 00000001138be003 CR4: 00000000001706e0
Apr 14 06:31:09 <redacted> kernel: note: kworker/5:19[1354] exited with preempt_count 1

After this, the server stopped working correctly (including no longer accepting SSH connections) and had to be manually rebooted from the console. If there is any other data I can supply to help track this down, please let me know.

[namjaejeon]

I am trying to reproduce this issue. but can't do that yet. Can you tell me how to shutdown ksmbd server when rebooting your server ?

[mamarley]

It shuts down using the included systemd unit. The crash occurred after it restarted though, not on shutdown. This is the first time I have seen it in the entire time I have been using ksmbd though, so I'm not sure how much luck you will have reproducing it.

[hclee]

A session appears to have been logged off before processing read requests. I don't know how this can happen.

[namjaejeon]

@mamarley I have fixed this issue. Please use the latest ksmbd. (patch is https://github.com/cifsd-team/ksmbd/commit/f4218ef4641001c7fb33cc4271bdc9591f5ed4a5)