search

cifsd-team / ksmbd

ksmbd kernel server(SMB/CIFS server)
152 stars 23 forks source link

cifsd crash on openwrt: kernel panic #424

Closed ptpt52 closed 4 years ago

ptpt52 commented 4 years ago

here is the log:

<6>[   35.465730] device wlan0 entered promiscuous mode
<6>[   35.475590] br-lan: port 3(wlan0) entered blocking state
<6>[   35.486266] br-lan: port 3(wlan0) entered forwarding state
<6>[   36.168564] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
<6>[   46.894164] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1: link becomes ready
<6>[   46.907647] br-lan: port 2(wlan1) entered blocking state
<6>[   46.918326] br-lan: port 2(wlan1) entered forwarding state
<3>[   60.056777] ksmbd: build_sess_rsp_extsec:1195: Invalid phase
<1>[   62.238419] CPU 1 Unable to handle kernel paging request at virtual address 00000010, epc == 8deb51a8, ra == 8de92388
<4>[   62.259579] Oops[#1]:
<4>[   62.264130] CPU: 1 PID: 31 Comm: kworker/1:1 Tainted: G         C        5.4.48 #0
<4>[   62.279295] Workqueue: ksmbd-io ksmbd_ipc_init [ksmbd]
<4>[   62.289515] $ 0   : 00000000 00000001 00006045 00000000
<4>[   62.299914] $ 4   : 8dc306e0 8f4dc004 00000000 8fd37dcc
<4>[   62.310324] $ 8   : 00000065 37313443 6c704038 65736165
<4>[   62.320730] $12   : 6db39ee1 f51ba518 eb8e147d f0fbb466
<4>[   62.331128] $16   : 8f4dc000 80690000 8dc3073c 8dec2318
<4>[   62.341525] $20   : 00000072 8f642854 8dec0000 8deb7400
<4>[   62.351925] $24   : 00000003 a90e5c76                  
<4>[   62.362325] $28   : 8fd36000 8fd37de8 8debb77c 8de92388
<4>[   62.372725] Hi    : 0038a0af
<4>[   62.378443] Lo    : d5a6dc00
<4>[   62.384207] epc   : 8deb51a8 smb1_set_sign_rsp+0x4c/0x134 [ksmbd]
<4>[   62.396349] ra    : 8de92388 ksmbd_ipc_init+0x8a0/0xcb0 [ksmbd]
<4>[   62.408115] Status: 11007c03  KERNEL EXL IE 
<4>[   62.416438] Cause : 40800008 (ExcCode 02)
<4>[   62.424404] BadVA : 00000010
<4>[   62.430125] PrId  : 0001992f (MIPS 1004Kc)
<4>[   62.438261] Modules linked in: ksmbd natcap qcserial pppoe ppp_async option l2tp_ppp cdc_mbim usb_wwan sierra_net sierra rndis_host qmi_wwan pptp pppox ppp_mppe ppp_generic mt76x2e mt76x2_common mt76x02_lib mt7603e mt76 mac80211 ipt_REJECT huawei_cdc_ncm cfg80211 cdc_ncm cdc_ether xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_socket xt_recent xt_quota xt_pkttype xt_physdev xt_owner xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange xt_ipp2p xt_iface xt_hl xt_helper xt_hashlimit xt_esp xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_addrtype xt_TPROXY xt_TCPMSS xt_REDIRECT xt_NETMAP xt_MASQUERADE xt_LOG xt_IPMARK xt_HL xt_DSCP xt_CT xt_CLASSIFY wireguard usbserial usbnet usblp ts_fsm ts_bm slhc sch_cake r8152 nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject_bridge nft_reject nft_redir nft_quota nft_objref nft_numgen nft_meta_bridge nft_log nft_limit nft_hash nft_fwd_netdev nft_dup_netdev nft_ct nft_counter nf_tproxy_ipv6
<4>[   62.438568]  nf_tproxy_ipv4 nf_tables_set nf_tables nf_socket_ipv6 nf_socket_ipv4 nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_rtsp nf_nat_pptp nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_log_ipv4 nf_dup_netdev nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtsp nf_conntrack_rtcache nf_conntrack_pptp nf_conntrack_netlink nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast ts_kmp nf_conntrack_amanda nf_conncount macvlan iptable_raw iptable_nat iptable_mangle iptable_filter ipt_ah ipt_ECN ipheth ip_tables crc_ccitt compat_xtables compat cdc_wdm br_netfilter asn1_decoder natflow fuse tcp_bbr sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred ledtrig_usbport ledtrig_heartbeat xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet
<4>[   62.611817]  ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6t_NPT ip6t_rt ip6t_mh ip6t_ipv6header ip6t_hbh ip6t_frag ip6t_eui64 ip6t_ah nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 msdos ip6_gre ip_gre gre ifb sit l2tp_netlink l2tp_core udp_tunnel ip6_udp_tunnel ip6_tunnel tunnel6 tunnel4 ip_tunnel tun vfat fat autofs4 nls_utf8 nls_iso8859_1 nls_cp437 sha512_generic sha256_generic libsha256 sha1_generic seqiv jitterentropy_rng drbg md5 md4 hmac ghash_generic gf128mul gcm ecb des_generic libdes ctr cmac ccm arc4 uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd ohci_platform ohci_hcd softdog fsl_mph_dr_of ehci_platform ehci_fsl sd_mod scsi_mod ehci_hcd gpio_button_hotplug ext4 mbcache jbd2 exfat(C) usbcore nls_base usb_common mii crc32c_generic
<4>[   62.784445]  [last unloaded: natcap]
<4>[   62.964599] Process kworker/1:1 (pid: 31, threadinfo=fa69a75b, task=1df4ca74, tls=00000000)
<4>[   62.981212] Stack : 8dc306e0 00000001 00000072 8deac5ac 8f4dc004 8deb7400 8debb77c 8de8ea00
<4>[   62.997857]         5f016798 00000000 0a9a2c07 8f642800 8dc306e0 c898a4ba 8dc306e0 8f642800
<4>[   63.014495]         8dc306e0 8de92388 80690000 80052190 8fca8ac0 805d0000 8fca8df4 81013dc0
<4>[   63.031125]         8fc3bb20 8debb7b0 8dc3073c 8fcb0180 81013a40 ff7db700 00000000 00000040
<4>[   63.047755]         00000000 80690000 81013a40 80040f5c 81013a40 81013a40 00000008 81013a58
<4>[   63.064384]         ...
<4>[   63.069247] Call Trace:
<4>[   63.074184] [<8deb51a8>] smb1_set_sign_rsp+0x4c/0x134 [ksmbd]
<4>[   63.085668] [<8de92388>] ksmbd_ipc_init+0x8a0/0xcb0 [ksmbd]
<4>[   63.096771] Code: a203000f  8c830004  afa50010 <8c620010> 24420001  ac620010  aa020015  ba020012  aa000019 
<4>[   63.116181] 
<4>[   63.123215] ---[ end trace 4bf8aa729ba48ad1 ]---
<0>[   63.159626] Kernel panic - not syncing: Fatal exception
neheb commented 4 years ago

Platform is little endian.

ptpt52 commented 4 years ago

yep it on mt7621.

ptpt52 commented 4 years ago

it is version 3.2.2 I am trying to update to master HEAD

neheb commented 4 years ago

https://github.com/cifsd-team/cifsd/pull/419

namjaejeon commented 4 years ago

Hm.. work->sess seems to be NULL in smb1_set_sign_rsp. can you check the below change fix your issue ?

diff --git a/server.c b/server.c
index 819a4cf..aed03d9 100644
--- a/server.c
+++ b/server.c
@@ -226,9 +226,9 @@ static void __handle_ksmbd_work(struct ksmbd_work *work,
                        }
                }

-               if ((work->sess && work->sess->sign) ||
+               if (work->sess && ((work->sess->sign) ||
                     smb3_11_final_sess_setup_resp(work) ||
-                    conn->ops->is_sign_req(work, command))
+                    conn->ops->is_sign_req(work, command)))
                        conn->ops->set_sign_rsp(work);
        } while (is_chained_smb2_message(work));
ptpt52 commented 4 years ago

@namjaejeon I will try it, and it may take time to do test. I will feed back if any update

namjaejeon commented 4 years ago

@ptpt52 Okay, I will wait. Good night!

Bernie commented 4 years ago

I think I have the same issue on ath79 architecture with my Archer A7 router:

Time: 1593919964.182145
Modules:        ksmbd@84180000+36409    ath9k@86ee0000+18ffb    ath9k_common@86f70000+2f6d      pppoe@86dfc000+23f0     ppp_async@86dfa000+1af0 ath9k_hw@86e80000+5556c ath10k_pci@86d60000+85cf   ath10k_core@86e00000+5de74      ath@86c68000+49f3       pppox@876d7000+56a      ppp_generic@86cb0000+58e2       nf_conntrack_ipv6@86c5c000+1200 mac80211@86d80000+71ba8    iptable_nat@86c86000+2f0        ipt_REJECT@86c54000+3f0 ipt_MASQUERADE@86c5f000+2d0     cfg80211@86cc0000+39134 xt_time@86c57000+710    xt_tcpudp@86c4b000+710  xt_tcpmss@86c43000+450     xt_statistic@86c32000+370       xt_state@86c15000+2f0   xt_nat@86c4d000+630     xt_multiport@86c45000+510       xt_mark@86c50000+2d0    xt_mac@86c37000+290     xt_limit@86c5a000+4d0      xt_length@86c52000+2f0  xt_hl@86c36000+350      xt_ecn@86c51000+570     xt_dscp@86c2e000+410    xt_conntrack@86c4f000+930       xt_comment@86c31000+210 xt_TCPMSS@8769c000+af0     xt_REDIRECT@86c27000+2f0        xt_LOG@86c2f000+330     xt_HL@876cd000+550      xt_FLOWOFFLOAD@86c33000+c00     xt_DSCP@86c30000+5f0    xt_CLASSIFY@877db000+270   ums_usbat@86c08000+2050 ums_sddr55@86c1e000+1750        ums_sddr09@86c10000+23d0        ums_karma@877d7000+6d0  ums_jumpshot@86c20000+f90       ums_isd200@86c18000+1700  ums_freecom@86c1b000+8d0 ums_datafab@87710000+12f0       ums_cypress@87669000+a50        ums_alauda@86c0c000+22b0        slhc@86c06000+145b      nf_reject_ipv4@877d6000+883     nf_nat_redirect@877d9000+5fb       nf_nat_masquerade_ipv4@877eb000+6fc     nf_conntrack_ipv4@877d0000+1110 nf_nat_ipv4@8779f000+d51        nf_nat@877e8000+24cb    nf_log_ipv4@8766d000+d30   nf_flow_table_hw@87719000+870   nf_flow_table@87714000+363f     nf_defrag_ipv6@8771c000+12be    nf_defrag_ipv4@8769d000+416     nf_conntrack_rtcache@877c9000+a70       nf_conntrack@877f0000+e1c5 iptable_mangle@877c8000+3d0     iptable_filter@877b6000+2b0     ipt_ECN@87761000+590    ip_tables@877c4000+276d crc_ccitt@87757000+40b  compat@877b0000+1977       ledtrig_usbport@8778f000+ae0    xt_set@877b2000+1b50    ip_set_list_set@8775c000+1c50   ip_set_hash_netportnet@877b8000+6c10    ip_set_hash_netport@877a8000+6170       ip_set_hash_netnet@87798000+6770   ip_set_hash_netiface@877a0000+6130      ip_set_hash_net@87790000+5b70   ip_set_hash_mac@87764000+28d0   ip_set_hash_ipportnet@87788000+66d0     ip_set_hash_ipportip@87780000+54b0 ip_set_hash_ipport@87778000+5090        ip_set_hash_ipmark@87770000+4dd0        ip_set_hash_ip@87768000+4cf0    ip_set_bitmap_port@87746000+17b0  ip_set_bitmap_ipmac@87758000+1b70        ip_set_bitmap_ip@8774e000+1ab0  ip_set@87750000+586a    nfnetlink@8773c000+1187 nf_log_ipv6@87742000+ff0        nf_log_common@8773f000+a4fip6table_mangle@87735000+4f0     ip6table_filter@8771e000+2b0    ip6_tables@87738000+26e1        ip6t_REJECT@87728000+430        x_tables@87730000+36af  nf_reject_ipv6@87725000+9a8ip6_gre@87708000+38b0   ip_gre@876d8000+3435    gre@875b8000+8d3        ip6_tunnel@876d0000+588f        tunnel6@876fe000+73e    ip_tunnel@876b8000+2e10 tun@876c0000+5891       multipath@876bc000+1370    raid456@876e0000+1cdce  libcrc32c@8767d000+297  async_raid6_recov@8767e000+421  async_pq@8764a000+706   async_xor@8765c000+449  xor@876b6000+189b       raid6_pq@876a0000+15430    async_memcpy@87656000+2ae       async_tx@875a4000+439   raid10@87670000+b130    raid1@87640000+7750     raid0@87648000+1ee0     md_mod@87680000+1b67d   nls_utf8@875ad000+370      sha512_generic@875b4000+24d9    sha256_generic@875b0000+28d9    seqiv@87599000+730      jitterentropy_rng@875a6000+1c78 drbg@875a8000+44b0      md5@8759a000+1290  md4@87597000+d10        hmac@87589000+9f0       ghash_generic@87586000+790      gf128mul@8758a000+17fe  gcm@8758c000+2720       ecb@87584000+590        des_generic@87590000+494b  ctr@8763e000+cb0        cmac@87622000+870       ccm@874ec000+1ff0       uas@87408000+2d80       usb_storage@87470000+a44b       ehci_platform@8741e000+13d0     sd_mod@87438000+7240       scsi_mod@87420000+16b12 ehci_hcd@87480000+8ad7  gpio_button_hotplug@87fd0000+1af0       ext4@87500000+5f595     mbcache@87fba000+c6e    jbd2@87fe0000+cb81      usbcore@87440000+20823     nls_base@87fd4000+1420  usb_common@87fb1000+9f7 crc16@87fc1000+407      aead@87fb2000+e61       crypto_null@87fb6000+c22        cryptomgr@87fc0000+799  crc32c_generic@87fb0000+590        crypto_hash@87fbc000+2752

<< unrelated logs excluded >>

<3>[  256.758812] ksmbd: kill_server_store:480: kill command received
<3>[  314.919136] ksmbd: __rpc_method:85: Unsupported RPC:
<3>[  314.924479] ksmbd: create_smb2_pipe:1762: Unable to open RPC pipe: -22
<1>[  314.959247] CPU 0 Unable to handle kernel paging request at virtual address 00000000, epc == 804104d0, ra == 8014fcb8
<4>[  314.970425] Oops[#1]:
<4>[  314.972780] CPU: 0 PID: 3207 Comm: kworker/0:4 Not tainted 4.14.180 #0
<4>[  314.979600] Workqueue: ksmbd-io ksmbd_ipc_init [ksmbd]
<4>[  314.984906] task: 854f4000 task.stack: 85516000
<4>[  314.989580] $ 0   : 00000000 80660000 00000000 00000003
<4>[  314.994983] $ 4   : 00000000 00000001 85517d24 00000000
<4>[  315.000386] $ 8   : 00000001 00000000 00001000 00000029
<4>[  315.005780] $12   : 0022652d 0dc52760 ffffffff 00000200
<4>[  315.011183] $16   : 00000001 85517d24 01400000 80510000
<4>[  315.016586] $20   : 00000000 84957b00 856bd000 841adb00
<4>[  315.021989] $24   : 80509040 1d0e2ce0
<4>[  315.027392] $28   : 85516000 85517c60 80510000 8014fcb8
<4>[  315.032796] Hi    : 0022652d
<4>[  315.035762] Lo    : 0dc52760
<4>[  315.038737] epc   : 804104d0 strlen+0xc/0x20
<4>[  315.043160] ra    : 8014fcb8 getname_kernel+0x2c/0xf4
<4>[  315.048368] Status: 1100dc03      KERNEL EXL IE
<4>[  315.052687] Cause : 00800008 (ExcCode 02)
<4>[  315.056826] BadVA : 00000000
<4>[  315.059793] PrId  : 00019750 (MIPS 74Kc)
<4>[  315.063841] Modules linked in: ksmbd ath9k ath9k_common pppoe ppp_async ath9k_hw ath10k_pci ath10k_core ath pppox ppp_generic nf_conntrack_ipv6 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CLASSIFY ums_usbat ums_sddr55 ums_sddr09 ums_karma ums_jumpshot ums_isd200 ums_freecom ums_datafab ums_cypress ums_alauda slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt compat ledtrig_usbport
<4>[  315.137502]  xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ip6_gre ip_gre gre ip6_tunnel tunnel6 ip_tunnel tun multipath raid456 libcrc32c async_raid6_recov async_pq async_xor xor raid6_pq async_memcpy async_tx raid10 raid1 raid0 md_mod nls_utf8 sha512_generic sha256_generic seqiv jitterentropy_rng drbg md5 md4 hmac ghash_generic gf128mul gcm ecb des_generic ctr cmac ccm uas usb_storage ehci_platform sd_mod scsi_mod ehci_hcd gpio_button_hotplug
<4>[  315.211072]  ext4 mbcache jbd2 usbcore nls_base usb_common crc16 aead crypto_null cryptomgr crc32c_generic crypto_hash
<4>[  315.222142] Process kworker/0:4 (pid: 3207, threadinfo=85516000, task=854f4000, tls=00000000)
<4>[  315.230936] Stack : 014000c0 87c6a000 8369ba80 802e8f5c 00000001 85517d24 00000000 00000001
<4>[  315.239573]         85517d24 80150210 85459a00 8369ba80 852dfda5 000005a8 85459ae4 85517db8
<4>[  315.248210]         00004000 86893c00 00000000 84184ad8 8369ba80 0000009c 000005a8 85459ae4
<4>[  315.256848]         01080020 00004000 85459a00 8034cfe8 841b05a8 85517d04 86893c00 800b4054
<4>[  315.265485]         000005a8 841ad880 86893c00 87722700 87722780 86893c00 856bf700 84957b00
<4>[  315.274113]         ...
<4>[  315.276642] Call Trace:
<4>[  315.279164] [<804104d0>] strlen+0xc/0x20
<4>[  315.283223] [<8014fcb8>] getname_kernel+0x2c/0xf4
<4>[  315.288081] [<80150210>] kern_path+0x1c/0x48
<4>[  315.292538] [<84184ad8>] ksmbd_vfs_kern_path+0x38/0x1cc [ksmbd]
<4>[  315.298680] [<8419265c>] ksmbd_queue_work+0x1c88/0x2590 [ksmbd]
<4>[  315.304803] Code: 10000002  00801025  24420001 <80430000> 1460fffd  00000000  03e00008  00441023  00852821
<4>[  315.314882]
<4>[  315.317031] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 1, skipped old beacon
<4>[  315.324635] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 2, skipped old beacon
<4>[  315.332290] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
<4>[  315.340637] ---[ end trace 80c769b33d9e33bb ]---

This is on openwrt v19.07.3 with ksmbd-server 3.2.1-1.

Any update on this issue?

Thanks!

Bernie commented 4 years ago

Actually, just noticed the stack traces are different. Let me know if I should file a different bug for this.

Thanks!

namjaejeon commented 4 years ago

@Bernie Yes, Your kernel oops is not related with this ISSUE ptpt52 raise. anyway, I will check it.

Bernie commented 4 years ago

Created new issue, #425.

Thank you.

BrainSlayer commented 4 years ago

@Bernie the patch provided by @namjaejeon will fix this issue. had the same discovery on dd-wrt and users reporting that this patch fixes the issue

ptpt52 commented 4 years ago

yep, for 5 days running no crash.

namjaejeon commented 4 years ago

Thanks for your confirmation! Pushed the patch into #ksmbd-next.