search

cifsd-team / ksmbd

ksmbd kernel server(SMB/CIFS server)
159 stars 23 forks source link

cifsd: smb1 crash on openwrt #426

Closed ptpt52 closed 4 years ago

ptpt52 commented 4 years ago

logs:

<3>[250047.755839] EXT4-fs (sdb1): unable to read superblock
<3>[250047.762206] EXT4-fs (sdb1): unable to read superblock
<3>[250047.769003] EXT4-fs (sdb1): unable to read superblock
<3>[250047.843178] exFAT-fs (sdb1): invalid fs_name
<3>[250047.847569] exFAT-fs (sdb1): failed to read boot sector
<3>[250047.852893] exFAT-fs (sdb1): failed to recognize exfat type
<3>[250047.859938] EXT4-fs (sdb1): unable to read superblock
<3>[250047.866320] EXT4-fs (sdb1): unable to read superblock
<3>[250047.873122] EXT4-fs (sdb1): unable to read superblock
<3>[251927.841628] ksmbd: query_fs_info:4575: cannot create vfs path
<4>[251927.862669] Kernel bug detected[#1]:
<4>[251927.866346] CPU: 1 PID: 14551 Comm: kworker/1:0 Not tainted 5.4.51 #0
<4>[251927.872954] Workqueue: ksmbd-io ksmbd_ipc_init [ksmbd]
<4>[251927.878175] $ 0   : 00000000 00000001 00000000 00000100
<4>[251927.883475] $ 4   : 8582dd64 00000000 ffffffff 00c8ef5d
<4>[251927.888773] $ 8   : 8e783200 8e783200 00000000 fefefeff
<4>[251927.894068] $12   : 00000000 00000000 5f116547 00000000
<4>[251927.899363] $16   : 00000000 808dd5a0 8dc45500 00000000
<4>[251927.904658] $20   : 8e783381 8e783480 8e783380 00000000
<4>[251927.909953] $24   : 00000000 80218d30                  
<4>[251927.915249] $28   : 8582c000 8582dc10 8e737e00 8012d888
<4>[251927.920544] Hi    : 00000000
<4>[251927.923495] Lo    : 00cfac00
<4>[251927.926463] epc   : 8012c568 BUG+0x0/0x4
<4>[251927.930460] ra    : 8012d888 deactivate_slab.isra.92+0x0/0x3d4
<4>[251927.936350] Status: 11007c03 KERNEL EXL IE 
<4>[251927.940607] Cause : 50800024 (ExcCode 09)
<4>[251927.944684] PrId  : 0001992f (MIPS 1004Kc)
<4>[251927.948850] Modules linked in: ksmbd natcap qcserial pppoe ppp_async option l2tp_ppp cdc_mbim usb_wwan sierra_net sierra rndis_host qmi_wwan pptp pppox ppp_mppe ppp_generic mt76x2e mt76x2_common mt76x02_lib mt7603e mt76 mac80211 ipt_REJECT huawei_cdc_ncm cfg80211 cdc_ncm cdc_ether xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_socket xt_recent xt_quota xt_pkttype xt_physdev xt_owner xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange xt_ipp2p xt_iface xt_hl xt_helper xt_hashlimit xt_esp xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_addrtype xt_TPROXY xt_TCPMSS xt_REDIRECT xt_NETMAP xt_MASQUERADE xt_LOG xt_IPMARK xt_HL xt_DSCP xt_CT xt_CLASSIFY wireguard usbserial usbnet usblp ts_fsm ts_bm slhc sch_cake r8152 nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject_bridge nft_reject nft_redir nft_quota nft_objref nft_numgen nft_meta_bridge nft_log nft_limit nft_hash nft_fwd_netdev nft_dup_netdev nft_ct nft_counter nf_tproxy_ipv6
<4>[251927.949073]  nf_tproxy_ipv4 nf_tables_set nf_tables nf_socket_ipv6 nf_socket_ipv4 nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_rtsp nf_nat_pptp nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_log_ipv4 nf_dup_netdev nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtsp nf_conntrack_rtcache nf_conntrack_pptp nf_conntrack_netlink nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast ts_kmp nf_conntrack_amanda nf_conncount macvlan iptable_raw iptable_nat iptable_mangle iptable_filter ipt_ah ipt_ECN ipheth ip_tables crc_ccitt compat_xtables compat cdc_wdm br_netfilter asn1_decoder natflow fuse tcp_bbr sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred ledtrig_usbport ledtrig_heartbeat xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet
<4>[251928.035858]  ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6t_NPT ip6t_rt ip6t_mh ip6t_ipv6header ip6t_hbh ip6t_frag ip6t_eui64 ip6t_ah nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 msdos ip6_gre ip_gre gre ifb sit l2tp_netlink l2tp_core udp_tunnel ip6_udp_tunnel ip6_tunnel tunnel6 tunnel4 ip_tunnel tun vfat fat autofs4 nls_utf8 nls_iso8859_1 nls_cp437 sha512_generic sha256_generic libsha256 sha1_generic seqiv jitterentropy_rng drbg md5 md4 hmac ghash_generic gf128mul gcm ecb des_generic libdes ctr cmac ccm arc4 uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd ohci_platform ohci_hcd softdog fsl_mph_dr_of ehci_platform ehci_fsl sd_mod scsi_mod ehci_hcd gpio_button_hotplug ext4 mbcache jbd2 exfat usbcore nls_base usb_common mii crc32c_generic
<4>[251928.122337]  [last unloaded: natcap]
<4>[251928.212369] Process kworker/1:0 (pid: 14551, threadinfo=4eb6b890, task=f47d56e3, tls=00000000)
<4>[251928.221024] Stack : 8e783201 8de09188 8e783580 8f540000 00000001 8e783201 8dc45500 00000001
<4>[251928.229435]         fffffffe 8dc45500 8f540000 8de2abdc 00000201 8011ce58 41b47b86 8fd0f100
<4>[251928.237846]         00000cc0 000005a7 00000000 8de40000 8f540000 8e7835c8 8e530000 80a17800
<4>[251928.246257]         806b5680 8011f82c bf745c79 8dd580fc 8582ddb0 00004000 00000201 0000c3c2
<4>[251928.254667]         00000201 00000000 806b5794 806b57ac 00000001 806b5e08 8582dd64 80690380
<4>[251928.263081]         ...
<4>[251928.265610] Call Trace:
<4>[251928.268155] [<8012c568>] BUG+0x0/0x4
<4>[251928.271811] [<8012d888>] deactivate_slab.isra.92+0x0/0x3d4
<4>[251928.277355] Code: 00000000  1000ffda  24021000 <000c000d> 27bdffe8  afb00010  afbf0014  9082fffc  30420002 
<4>[251928.287159] 
<4>[251928.290800] ---[ end trace 4bb82e6dacf66f4f ]---
<0>[251928.314610] Kernel panic - not syncing: Fatal exception
namjaejeon commented 4 years ago

I don't think this issue is related with ksmbd. kernel oops happen at BUG ON trap of deactivate_slab(). Currently, mount failures occur when your application mount with your strorage. your HW is not stable ?

<3>[250047.755839] EXT4-fs (sdb1): unable to read superblock <3>[250047.762206] EXT4-fs (sdb1): unable to read superblock <3>[250047.769003] EXT4-fs (sdb1): unable to read superblock <3>[250047.843178] exFAT-fs (sdb1): invalid fs_name <3>[250047.847569] exFAT-fs (sdb1): failed to read boot sector <3>[250047.852893] exFAT-fs (sdb1): failed to recognize exfat type <3>[250047.859938] EXT4-fs (sdb1): unable to read superblock <3>[250047.866320] EXT4-fs (sdb1): unable to read superblock <3>[250047.873122] EXT4-fs (sdb1): unable to read superblock
neheb commented 4 years ago

ramips USB is quite a disaster honestly.

ptpt52 commented 4 years ago

another crash log post here: (with lastest code)

<3>[17323.463186] ksmbd: smb2_check_sign_req:7751: bad smb2 signature
<3>[17323.473268] ksmbd: smb2_check_sign_req:7751: bad smb2 signature
<1>[17330.218916] CPU 3 Unable to handle kernel paging request at virtual address fffffff8, epc == 86bc8990, ra == 86bc8978
<4>[17330.229580] Oops[#1]:
<4>[17330.231861] CPU: 3 PID: 18818 Comm: kworker/u8:1 Not tainted 5.4.65 #0
<4>[17330.238374] Workqueue: writeback wb_workfn (flush-8:17-fuseblk)
<4>[17330.244270] $ 0   : 00000000 00000001 ffffffd8 00000000
<4>[17330.249476] $ 4   : 8199463c 00000000 00000000 00000001
<4>[17330.254685] $ 8   : ffffffff 0000000c 00000010 fefefeff
<4>[17330.259908] $12   : 00000000 00000000 00000000 00000000
<4>[17330.265113] $16   : 81994440 81994440 00000000 00000003
<4>[17330.270319] $20   : 81994538 8627e65c 8692bc00 00000000
<4>[17330.275525] $24   : 00000010 8055379c                  
<4>[17330.280732] $28   : 81440000 81441c98 80690000 86bc8978
<4>[17330.285941] Hi    : 00000000
<4>[17330.288805] Lo    : 0001d800
<4>[17330.291762] epc   : 86bc8990 fuse_file_poll+0x658/0x7ac [fuse]
<4>[17330.297607] ra    : 86bc8978 fuse_file_poll+0x640/0x7ac [fuse]
<4>[17330.303412] Status: 11007c03  KERNEL EXL IE 
<4>[17330.307589] Cause : 40800008 (ExcCode 02)
<4>[17330.311578] BadVA : fffffff8
<4>[17330.314441] PrId  : 0001992f (MIPS 1004Kc)
<4>[17330.318518] Modules linked in: ksmbd qcserial pppoe ppp_async option cdc_mbim usb_wwan sierra_net sierra rndis_host qmi_wwan pptp pppox ppp_mppe ppp_generic mt76x2e mt76x2_common mt76x02_lib mt7603e mt76 mac80211 ipt_REJECT huawei_cdc_ncm cfg80211 cdc_ncm cdc_ether xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_socket xt_recent xt_quota xt_pkttype xt_physdev xt_owner xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange xt_ipp2p xt_iface xt_hl xt_helper xt_hashlimit xt_esp xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_addrtype xt_TPROXY xt_TCPMSS xt_REDIRECT xt_NETMAP xt_MASQUERADE xt_LOG xt_IPMARK xt_HL xt_DSCP xt_CT xt_CLASSIFY wireguard usbserial usbnet usblp ts_fsm ts_bm slhc sch_cake r8152 nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject_bridge nft_reject nft_redir nft_quota nft_objref nft_numgen nft_meta_bridge nft_log nft_limit nft_hash nft_fwd_netdev nft_dup_netdev nft_ct nft_counter nf_tproxy_ipv6 nf_tproxy_ipv4
<4>[17330.318726]  nf_tables_set nf_tables nf_socket_ipv6 nf_socket_ipv4 nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_rtsp nf_nat_pptp nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_log_ipv4 nf_dup_netdev nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtsp nf_conntrack_rtcache nf_conntrack_pptp nf_conntrack_netlink nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast ts_kmp nf_conntrack_amanda nf_conncount macvlan iptable_raw iptable_nat iptable_mangle iptable_filter ipt_ah ipt_ECN ipheth ip_tables crc_ccitt compat_xtables compat cdc_wdm br_netfilter asn1_decoder natflow natcap tcp_bbr sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred ledtrig_usbport ledtrig_heartbeat xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip
<4>[17330.405317]  ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6t_NPT ip6t_rt ip6t_mh ip6t_ipv6header ip6t_hbh ip6t_frag ip6t_eui64 ip6t_ah nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 msdos ip6_gre ip_gre gre fuse ifb ip6_udp_tunnel udp_tunnel sit ip6_tunnel tunnel6 tunnel4 ip_tunnel tun vfat fat autofs4 nls_utf8 nls_iso8859_1 nls_cp437 sha512_generic sha256_generic libsha256 sha1_generic seqiv jitterentropy_rng drbg md5 md4 hmac ghash_generic gf128mul gcm ecb des_generic libdes ctr cmac ccm arc4 uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd ohci_platform ohci_hcd softdog fsl_mph_dr_of ehci_platform ehci_fsl sd_mod scsi_mod ehci_hcd gpio_button_hotplug ext4 mbcache jbd2 exfat usbcore nls_base usb_common mii crc32c_generic
<4>[17330.575310] Process kworker/u8:1 (pid: 18818, threadinfo=6a09ff70, task=80a8e904, tls=00000000)
<4>[17330.583970] Stack : 00000883 46aa9cc6 00000001 00000000 81994440 86bcf050 87c95100 81994440
<4>[17330.592297]         81441d20 81441e30 81994440 81994440 81441d20 80170be4 00001000 00000000
<4>[17330.600623]         00000000 00000000 00000000 00000000 819944e4 8627e648 81441e30 81994440
<4>[17330.608950]         80828e60 80170ff8 00000000 00000000 00000000 00000000 0040f700 fffdffff
<4>[17330.617282]         00000400 819944a8 00000400 00000000 00000000 00000000 ffffffff 7fffffff
<4>[17330.625614]         ...
<4>[17330.628144] Call Trace:
<4>[17330.630630] [<86bc8990>] fuse_file_poll+0x658/0x7ac [fuse]
<4>[17330.636127] Code: 00001025  8e0301a8  2462ffd8 <c064fff8> 24840001  e064fff8  1080fffc  00000000  0000000f 
<4>[17330.645844] 
<4>[17330.648343] ---
namjaejeon commented 4 years ago

Sorry, kernel ops happen in fuse_file_poll(). It does not seem to be the ksmbd problem.

neheb commented 4 years ago

@ptpt52 is this on NTFS-3G?

xdarklight commented 4 years ago

I can reproduce the original crash in deactivate_slab.isra on a BT Home Hub 5A (Big Endian MIPS, lantiq xrx200 target in OpenWrt). I'm using an USB hard disk with ext4 as filesystem.

how to not reproduce it - variant 1:

how to not reproduce it - variant 2:

how to reproduce it:

any suggestions are welcome

namjaejeon commented 4 years ago

@xdarklight Oh.. If you reproduce it easily, can you take a look ? Thanks!

xdarklight commented 4 years ago

because my crash is not exactly the same I opened #440 it's also crashing for me inside the SLUB allocator, so there may be some memory corruption going on (and then it may turn out that both issues are the same). but since I'm not sure I created another issue

namjaejeon commented 4 years ago

@ptpt52 I pushed Martin's patches into #ksmbd-next branch. Can you try to reproduce your issues with #ksmbd-next branch ?

ptpt52 commented 4 years ago

@namjaejeon seems fixed? build with #ksmbd-next after days test, no crash till now.

namjaejeon commented 4 years ago

@ptpt52 Cool~ I will release the next version this weekends 👍 Thanks for your test!

Neustradamus commented 4 years ago

Currently there is a problem, people are lost, the code is mixed in 2 places:

It is possible to do a little change, to be perfect and have the good main place: https://github.com/cifsd-team/cifsd?

After all steps, you can create a personal fork to work on the code before integration in the main place which will be https://github.com/cifsd-team/cifsd.

@namjaejeon

Thanks in advance.

Neustradamus commented 4 years ago

@namjaejeon: You can see some examples here:

@neheb, @xdarklight, @ptpt52, @Andy2244, @luizluca: Guys, can you explain to the main dev?

Thanks in advance.

It is linked to:

Same for ksmbd-tools (formely named cifsd-tools):

Same for cifsd-test-result:

Same for cifsd-perf:

namjaejeon commented 4 years ago

@ptpt52 please check 3.2.5 release(https://github.com/cifsd-team/cifsd/releases/tag/3.2.5). If you have any issue, Let me know it. Thanks!

Neustradamus commented 4 years ago

@ptpt52: Have you tested 3.2.5?

ptpt52 commented 4 years ago

yes, tested 3.2.5 for days, and not crash till now.

2020年10月22日 10:31，Neustradamus notifications@github.com 写道：

@ptpt52 https://github.com/ptpt52: Have you tested 3.2.5?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cifsd-team/cifsd/issues/426#issuecomment-714182001, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABIVTYVVTFLD7FER5TDVHUTSL6KO5ANCNFSM4O7HH2XA.

Neustradamus commented 4 years ago

@ptpt52: Thanks for your reply :)

ptpt52 commented 4 years ago

@namjaejeon hi, I found another crash, should I open an new issue?

<6>[   44.327632] br-lan: port 2(wlan1) entered blocking state
<6>[   44.327671] br-lan: port 2(wlan1) entered disabled state
<6>[   44.332440] device wlan1 entered promiscuous mode
<6>[   44.498275] ath10k_ahb a000000.wifi: NOTE:  Firmware DBGLOG output disabled in debug_mask: 0x10000000
<4>[   50.293548] ath10k_pci 0000:01:00.0: unsupported HTC service id: 1536
<4>[   50.295252] ath10k_pci 0000:01:00.0: 10.4 wmi init: vdevs: 16  peers: 48  tid: 96
<4>[   50.299040] ath10k_pci 0000:01:00.0: msdu-desc: 2500  skid: 32
<6>[   50.382655] ath10k_pci 0000:01:00.0: wmi print 'P 48/48 V 16 K 144 PH 176 T 186  msdu-desc: 2500  sw-crypt: 0 ct-sta: 0'
<6>[   50.384113] ath10k_pci 0000:01:00.0: wmi print 'free: 84920 iram: 13316 sram: 11224'
<4>[   50.775597] ath10k_pci 0000:01:00.0: Firmware lacks feature flag indicating a retry limit of > 2 is OK, requested limit: 4
<4>[   50.778033] warning: (zone_netdev_event:346)dev=wlan0 set zone=127 type=0
<4>[   50.785668] {natflow}:natflow_netdev_event(): catch NETDEV_UP event for dev=wlan0, add ingress hook
<6>[   50.801224] br-lan: port 3(wlan0) entered blocking state
<6>[   50.801369] br-lan: port 3(wlan0) entered disabled state
<6>[   50.807645] device wlan0 entered promiscuous mode
<3>[   51.751870] ess_edma c080000.edma: IPv6 not supported
<6>[   59.264008] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1: link becomes ready
<6>[   59.264635] br-lan: port 2(wlan1) entered blocking state
<6>[   59.269596] br-lan: port 2(wlan1) entered forwarding state
<6>[   61.684438] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
<6>[   61.685058] br-lan: port 3(wlan0) entered blocking state
<6>[   61.690058] br-lan: port 3(wlan0) entered forwarding state
<3>[   62.595805] ess_edma c080000.edma: IPv6 not supported
<4>[   73.831619] ath10k_pci 0000:01:00.0: Invalid VHT mcs 15 peer stats
<6>[  102.932938] ksmbd: kill command received
<3>[  103.440069] ksmbd: ksmbd_conn_handler_loop:339: sock_read failed: -108
<3>[  103.440185] Unable to close RPC pipe 1
<3>[  103.445969] ksmbd: ksmbd_conn_handler_loop:339: sock_read failed: -108
<3>[  103.449764] Unable to close RPC pipe 0
<1>[  104.084441] 8<--- cut here ---
<1>[  104.084721] Unable to handle kernel NULL pointer dereference at virtual address 0000000c
<1>[  104.086369] 8<--- cut here ---
<1>[  104.086780] pgd = cc524862
<1>[  104.094857] Unable to handle kernel NULL pointer dereference at virtual address 0000000c
<1>[  104.097674] [0000000c] *pgd=00000000
<1>[  104.100338] pgd = cc524862
<0>[  104.108605] Internal error: Oops: 17 [#1] SMP ARM
<4>[  104.108614] Modules linked in: ksmbd qcserial pppoe ppp_async option cdc_mbim ath10k_pci ath10k_core ath usb_wwan sierra_net sierra rndis_host qmi_wwan pptp pppox ppp_mppe ppp_generic mac80211 ipt_REJECT huawei_cdc_ncm cfg80211 cdc_ncm cdc_ether xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_socket xt_recent xt_quota xt_pkttype xt_physdev xt_owner xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange xt_ipp2p xt_iface xt_hl xt_helper xt_hashlimit xt_esp xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_addrtype xt_TPROXY xt_TCPMSS xt_REDIRECT xt_NETMAP xt_MASQUERADE xt_LOG xt_IPMARK xt_HL xt_DSCP xt_CT xt_CLASSIFY wireguard usbserial usbnet usblp ts_fsm ts_bm slhc sch_cake r8152 nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject_bridge nft_reject nft_redir nft_quota nft_objref nft_numgen nft_meta_bridge nft_log nft_limit nft_hash nft_fwd_netdev nft_dup_netdev nft_ct nft_counter nf_tproxy_ipv6 nf_tproxy_ipv4 nf_tables_set nf_tables
<4>[  104.108884]  nf_socket_ipv6
<1>[  104.112134] [0000000c] *pgd=00000000
<4>[  104.114657]  nf_socket_ipv4
<4>[  104.206395]  nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_rtsp nf_nat_pptp nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_log_ipv4 nf_dup_netdev nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtsp nf_conntrack_rtcache nf_conntrack_pptp nf_conntrack_netlink nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast ts_kmp nf_conntrack_amanda nf_conncount macvlan iptable_raw iptable_nat iptable_mangle iptable_filter ipt_ah ipt_ECN ipheth ip_tables hwmon crc_ccitt compat_xtables compat cdc_wdm br_netfilter asn1_decoder natflow natcap fuse tcp_bbr sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred ledtrig_usbport ledtrig_heartbeat xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark
<4>[  104.206627]  ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6t_NPT ip6t_rt ip6t_mh ip6t_ipv6header ip6t_hbh ip6t_frag ip6t_eui64 ip6t_ah nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 msdos ip6_gre ip_gre gre ifb ip6_udp_tunnel udp_tunnel sit ip6_tunnel tunnel6 tunnel4 ip_tunnel tun vfat fat autofs4 nls_utf8 nls_iso8859_1 nls_cp437 sha512_generic sha512_arm sha1_generic sha1_arm_neon sha1_arm md5 md4 ghash_generic ghash_arm_ce gf128mul gcm cmac ccm arc4 uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd fsl_mph_dr_of ehci_platform ehci_fsl sd_mod scsi_mod ehci_hcd gpio_button_hotplug ext4 mbcache jbd2 exfat mii crc32c_generic
<4>[  104.353711] CPU: 3 PID: 102 Comm: kworker/3:2 Not tainted 5.4.72 #0
<4>[  104.375935] Hardware name: Generic DT based system
<4>[  104.382207] Workqueue: ksmbd-io handle_ksmbd_work [ksmbd]
<4>[  104.387027] PC is at smb2_sess_setup+0x504/0x990 [ksmbd]
<4>[  104.392490] LR is at ksmbd_session_lookup_slowpath+0x98/0xb0 [ksmbd]
<4>[  104.397810] pc : [<bf9aef18>]    lr : [<bf9a5570>]    psr: a0000013
<4>[  104.404145] sp : cee9beb8  ip : ca494580  fp : ca61aa00
<4>[  104.410132] r10: bf9d71e0  r9 : ca61a600  r8 : ca61a000
<4>[  104.415341] r7 : cd727900  r6 : ca61aa00  r5 : cd727900  r4 : ccce8240
<4>[  104.420554] r3 : cd4d3400  r2 : ffffff00  r1 : 00000000  r0 : ca494640
<4>[  104.427160] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
<4>[  104.433665] Control: 10c5387d  Table: 8d23c06a  DAC: 00000051
<0>[  104.440870] Process kworker/3:2 (pid: 102, stack limit = 0x2f40fe42)
<0>[  104.446595] Stack: (0xcee9beb8 to 0xcee9c000)
<0>[  104.453019] bea0:                                                       c0b04ea8 ca494580
<0>[  104.457284] bec0: 00000000 cd4d3400 00090003 ca494240 ccce8240 ca61aa00 ccce8240 ccce829c
<0>[  104.465446] bee0: 00000001 bf9d6800 bf9d71b4 bf9cdfc0 bf9caab4 bf9a8cc0 c08c66c8 cee9bf14
<0>[  104.473603] bf00: bf9a8aec ccce829c cf9d9780 cfde4580 ff7f4200 00000000 000000c0 00000000
<0>[  104.481764] bf20: ffffe000 c0337d08 00000008 c0b03d00 cf9d9780 cf9d9794 cfde4580 00000008
<0>[  104.489924] bf40: c0b03d00 cfde4598 cfde4580 c0337fa4 c0b0cde8 c08c5780 00000000 c0b0ce2c
<0>[  104.498084] bf60: cf9d9780 cfa10c00 cee9a000 cee8b300 cf897eac cfa10c1c cf9d9780 c0337f60
<0>[  104.506243] bf80: 00000000 c033d564 00000001 cee8b300 c033d434 00000000 00000000 00000000
<0>[  104.514401] bfa0: 00000000 00000000 00000000 c03010e8 00000000 00000000 00000000 00000000
<0>[  104.522562] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[  104.530720] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
<4>[  104.539155] [<bf9aef18>] (smb2_sess_setup [ksmbd]) from [<bf9a8cc0>] (handle_ksmbd_work+0x1d4/0x4d8 [ksmbd])
<4>[  104.547116] [<bf9a8cc0>] (handle_ksmbd_work [ksmbd]) from [<c0337d08>] (process_one_work+0x218/0x470)
<4>[  104.556941] [<c0337d08>] (process_one_work) from [<c0337fa4>] (worker_thread+0x44/0x5dc)
<4>[  104.566053] [<c0337fa4>] (worker_thread) from [<c033d564>] (kthread+0x130/0x134)
<4>[  104.574215] [<c033d564>] (kthread) from [<c03010e8>] (ret_from_fork+0x14/0x2c)
<4>[  104.581577] Exception stack(0xcee9bfb0 to 0xcee9bff8)
<4>[  104.588614] bfa0:                                     00000000 00000000 00000000 00000000
<4>[  104.593742] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<4>[  104.601898] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000
<0>[  104.610060] Code: e59dc004 e58d300c e58d1008 e59c000c (e591100c) 
<0>[  104.616487] Internal error: Oops: 17 [#2] SMP ARM
<4>[  104.617623] ---[ end trace 22973d4e10d9cb01 ]---
<4>[  104.622723] Modules linked in: ksmbd qcserial pppoe ppp_async option cdc_mbim ath10k_pci ath10k_core ath usb_wwan sierra_net sierra rndis_host qmi_wwan pptp pppox ppp_mppe ppp_generic mac80211 ipt_REJECT huawei_cdc_ncm cfg80211 cdc_ncm cdc_ether xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_socket xt_recent xt_quota xt_pkttype xt_physdev xt_owner xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange xt_ipp2p xt_iface xt_hl xt_helper xt_hashlimit xt_esp xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_addrtype xt_TPROXY xt_TCPMSS xt_REDIRECT xt_NETMAP xt_MASQUERADE xt_LOG xt_IPMARK xt_HL xt_DSCP xt_CT xt_CLASSIFY wireguard usbserial usbnet usblp ts_fsm ts_bm slhc sch_cake r8152 nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject_bridge nft_reject nft_redir nft_quota nft_objref nft_numgen nft_meta_bridge nft_log nft_limit nft_hash nft_fwd_netdev nft_dup_netdev nft_ct nft_counter nf_tproxy_ipv6 nf_tproxy_ipv4 nf_tables_set nf_tables
ptpt52 commented 4 years ago

This happened 3 days ago, ksmbd version 3.2.5

device: arm device, RT-ACRH17, ipq40xx

namjaejeon commented 4 years ago

@ptpt52 As current, we can know that kernel oops happen in smb2_sess_setup() with your kernel log and related to kill command. maybe, there is racy issue between session connection and session disconnection. If you have vmlinux + addr2line, you can give more information(e.g. source line where kernel oops happen) to us by using addr2line. possible ?

ptpt52 commented 3 years ago 
  14ea0:   0a0000c8    beq 151c8 <smb2_sess_setup+0x7b4>
   14ea4:   e3a03001    mov r3, #1
   14ea8:   e5c73018    strb    r3, [r7, #24]
   14eac:   e5963000    ldr r3, [r6]
   14eb0:   e5933008    ldr r3, [r3, #8]
   14eb4:   e3130040    tst r3, #64 ; 0x40
   14eb8:   0affff81    beq 14cc4 <smb2_sess_setup+0x2b0>
   14ebc:   e5963004    ldr r3, [r6, #4]
   14ec0:   e593302c    ldr r3, [r3, #44]   ; 0x2c
   14ec4:   e3530000    cmp r3, #0
   14ec8:   0affff7d    beq 14cc4 <smb2_sess_setup+0x2b0>
   14ecc:   e1a00007    mov r0, r7
   14ed0:   e12fff33    blx r3
   14ed4:   e2509000    subs    r9, r0, #0
   14ed8:   1a0000cc    bne 15210 <smb2_sess_setup+0x7fc>
   14edc:   e3a03001    mov r3, #1
   14ee0:   e5c73019    strb    r3, [r7, #25]
   14ee4:   e3a03004    mov r3, #4
   14ee8:   e5c89047    strb    r9, [r8, #71]   ; 0x47
   14eec:   e5c83046    strb    r3, [r8, #70]   ; 0x46
   14ef0:   e5c79018    strb    r9, [r7, #24]
   14ef4:   eaffff72    b   14cc4 <smb2_sess_setup+0x2b0>
   14ef8:   ebfffffe    bl  b4d8 <ksmbd_session_lookup_slowpath>
            14ef8: R_ARM_CALL   ksmbd_session_lookup_slowpath
   14efc:   e2503000    subs    r3, r0, #0
   14f00:   0affff61    beq 14c8c <smb2_sess_setup+0x278>
   14f04:   e5931008    ldr r1, [r3, #8]
   14f08:   e59dc004    ldr ip, [sp, #4]
   14f0c:   e58d300c    str r3, [sp, #12]
   14f10:   e58d1008    str r1, [sp, #8]
   14f14:   e59c000c    ldr r0, [ip, #12]
   14f18:   e591100c    ldr r1, [r1, #12]   <<<<<<<<<<<<<<<<<<<<<<<<<<<< crash at this line
   14f1c:   ebfffffe    bl  0 <strcmp>
            14f1c: R_ARM_CALL   strcmp
   14f20:   e3500000    cmp r0, #0
   14f24:   1affff58    bne 14c8c <smb2_sess_setup+0x278>
   14f28:   e59dc004    ldr ip, [sp, #4]
   14f2c:   e59d1008    ldr r1, [sp, #8]
   14f30:   e59c2010    ldr r2, [ip, #16]
   14f34:   e5913010    ldr r3, [r1, #16]
   14f38:   e1520003    cmp r2, r3
   14f3c:   1affff52    bne 14c8c <smb2_sess_setup+0x278>
   14f40:   e5911014    ldr r1, [r1, #20]
   14f44:   e59c0014    ldr r0, [ip, #20]
   14f48:   ebfffffe    bl  0 <memcmp>
            14f48: R_ARM_CALL   memcmp

it crash at this line:

14f18:   e591100c    ldr r1, [r1, #12]   <<<<<<<<<<<<<<<<<<<<<<<<<<<< crash at this line
ptpt52 commented 3 years ago

likely crash at this function

 611 static void destroy_previous_session(struct ksmbd_user *user, uint64_t id)
 612 {
 613     struct ksmbd_session *prev_sess = ksmbd_session_lookup_slowpath(id);
 614     struct ksmbd_user *prev_user;
 615 
 616     if (!prev_sess)
 617         return;
 618 
 619     prev_user = prev_sess->user;
 620 
 621     if (strcmp(user->name, prev_user->name) ||
 622         user->passkey_sz != prev_user->passkey_sz ||
 623         memcmp(user->passkey, prev_user->passkey, user->passkey_sz))
 624         return;
 625 
 626     ksmbd_session_destroy(prev_sess);
 627 }
ptpt52 commented 3 years ago

@namjaejeon 

strcmp(user->name, prev_user->name) 
here, prev_user may be NULL
namjaejeon commented 3 years ago

@ptpt52 Thanks! I will take a look!

namjaejeon commented 3 years ago

@ptpt52 I sent the patch for this issue to the mailing list. Can you check it ?

ptpt52 commented 3 years ago

Ok I will try it out.