search

Microsemi / switchtec-kernel

A kernel module for the Microsemi PCIe switch
GNU General Public License v2.0
45 stars 31 forks source link

backport_centos7_3.10_with_NTB crash the kernel #44

Closed danglus closed 5 years ago

danglus commented 6 years ago

Hi, I have compiled thebackport_centos7_3.10_with_NTB branch. Unfortunately, when I try to insmod switchtec-kernel/switchtec.ko the system crashes with the following messages in '/var/crash':

` [ 376.295681] BUG: unable to handle kernel paging request at ffffc90006b76018 [ 376.303509] IP: [] ioread8+0x40/0x50 [ 376.309277] PGD 17fc60067 PUD 8ffc03067 PMD 105c9ce067 PTE 0 [ 376.315660] Oops: 0000 [#1] SMP [ 376.319289] Modules linked in: switchtec(OE+) ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack libcrc32c br_netfilter bridge stp llc overlay(T) nvmet_rdma(OE) nvmet(OE) nvme_rdma(OE) nvme_fabrics(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) mlx5_fp ga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc sb_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support pcspkr i2c_i801 joydev sg mxm_wmi lpc_ich mei_me mei ioatdma shpchp dca ipmi_si ipmi_devintf ipmi_msghandler [ 376.399214] wmi acpi_power_meter acpi_pad mst_pciconf(OE) ip_tables ext4 mbcache jbd2 mlx5_ib(OE) sr_mod cdrom ib_core(OE) sd_mod crc_t10dif crct10dif_generic uas usb_storage mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci libata crct10dif_pclmul crct10dif_common e1000e crc32c_intel mlx5_core(OE) nvme(OE) mlxfw(OE) devlink nvme_core(OE) mlx_compat(OE) i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: mst_pci] [ 376.446946] CPU: 10 PID: 238 Comm: kworker/10:1 Tainted: G OE ------------ T 3.10.0-693.el7.vastos.5.x86_64 #1 [ 376.459213] Hardware name: Newisys NSS2247G/NSS-HW2EC, BIOS V11.03 03/09/2018 [ 376.467206] Workqueue: events work_for_cpu_fn [ 376.472095] task: ffff88105caa8000 ti: ffff88105ca84000 task.ti: ffff88105ca84000 [ 376.480463] RIP: 0010:[] [] ioread8+0x40/0x50 [ 376.488952] RSP: 0018:ffff88105ca87d88 EFLAGS: 00010292 [ 376.494899] RAX: ffffc90006b84000 RBX: ffff88104ee72000 RCX: 0000000000000000 [ 376.502888] RDX: 0000000000000000 RSI: 0000000000000292 RDI: ffffc90006b76018 [ 376.510878] RBP: ffff88105ca87dc8 R08: 0000000000000002 R09: ffff88105ca87cfc [ 376.518865] R10: 0000000000000001 R11: ffff88105c3c1440 R12: ffff88107e773000 [ 376.526853] R13: ffff88107e773098 R14: 0000000000000000 R15: 0000000000000280 [ 376.534843] FS: 0000000000000000(0000) GS:ffff88105e000000(0000) knlGS:0000000000000000 [ 376.543901] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 376.550333] CR2: ffffc90006b76018 CR3: 00000000019f2000 CR4: 00000000003407e0 [ 376.558323] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 376.566311] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 376.574299] Stack: [ 376.576548] ffffffffc0888e20 0000000000000202 ffff88105ca87dc8 ffff88107e773000 [ 376.584875] 0000000000000000 ffffffffc088c000 ffff88107e773098 0000000000000280 [ 376.593205] ffff88105ca87e00 ffffffff8136a815 ffff88105e016cc0 ffff880858e43b98 [ 376.601535] Call Trace: [ 376.604278] [] ? switchtec_pci_probe+0x300/0x6cf [switchtec] [ 376.612465] [] local_pci_probe+0x45/0xa0 [ 376.618705] [] work_for_cpu_fn+0x14/0x20 [ 376.624946] [] process_one_work+0x17a/0x440 [ 376.631478] [] worker_thread+0x278/0x3c0 [ 376.637718] [] ? manage_workers.isra.24+0x2a0/0x2a0 [ 376.645027] [] kthread+0xcf/0xe0 [ 376.650488] [] ? insert_kthread_work+0x40/0x40 [ 376.658354] [] ret_from_fork+0x58/0x90 [ 376.665439] [] ? insert_kthread_work+0x40/0x40 [ 376.673299] Code: ec 0f b6 c0 c3 66 0f 1f 44 00 00 55 48 c7 c6 a2 20 93 81 48 89 e5 e8 90 ff ff ff b8 ff 00 00 00 5d c3 66 0f 1f 84 00 00 00 00 00 <8a> 07 0f b6 c0 c3 66 2e 0f 1f 84 00 00 00 00 00 48 81 ff ff ff [ 376.697326] RIP [] ioread8+0x40/0x50 [ 376.704244] RSP [ 376.709202] CR2: ffffc90006b76018 '

Any idea how to debug it?

Thanks, Dan

wesleywesley commented 6 years ago

@danglus Pls check the BAR0 size of the configure file. It should be 4MB.

We have been reported the same case before, the root cause is "BAR0 size is only 4KB". Which lead to accessing register in GAS out of 4KB range, crash the system.

danglus commented 6 years ago

@wesleywesley Are you talking about the 'BAR size' field the 'Management Endpoint Settings' of ChipLink configuration file editor? Actually, I don't have any Management Endpoint defined at the moment.

wesleywesley commented 6 years ago

@danglus No, there must be VEP(management ep/NT ep) defined, otherwise switchtec.ko module could not run into switchtec_pci_probe routine, and touch 0x2018 partition id register which belong to System Information Region, but out of 4K range.

You can double check by following steps:

  1. determine the VEP's BDF test@server1:~/switchtec-kernel$ lspci|grep PMC 81:00.0 PCI bridge: PMC-Sierra Inc. Device 8546 81:00.1 Memory controller: PMC-Sierra Inc. Device 8546 82:00.0 PCI bridge: PMC-Sierra Inc. Device 8546 82:01.0 PCI bridge: PMC-Sierra Inc. Device 8546 82:02.0 PCI bridge: PMC-Sierra Inc. Device 8546 82:03.0 PCI bridge: PMC-Sierra Inc. Device 8546 82:04.0 PCI bridge: PMC-Sierra Inc. Device 8546 82:05.0 PCI bridge: PMC-Sierra Inc. Device 8546 82:06.0 PCI bridge: PMC-Sierra Inc. Device 8546 82:07.0 PCI bridge: PMC-Sierra Inc. Device 8546 82:08.0 PCI bridge: PMC-Sierra Inc. Device 8546

  2. check VEP's BAR size test@server1:~/switchtec-kernel$ sudo lspci -vvv -s 81:00.1 [sudo] password for test: 81:00.1 Memory controller: PMC-Sierra Inc. Device 8546 Subsystem: PMC-Sierra Inc. Device 8546 Physical Slot: 4 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx- Latency: 0, Cache Line Size: 32 bytes Region 0: Memory at 387fff800000 (64-bit, prefetchable) [size=4M] Capabilities: [40] MSI: Enable- Count=1/4 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [50] MSI-X: Enable+ Count=4 Masked-

In the example: BDF for VEP is: 81:00.1 Memory controller: PMC-Sierra Inc. Device 8546 BAR size is: Region 0: Memory at 387fff800000 (64-bit, prefetchable) [size=4M]

danglus commented 5 years ago

@wesleywesley my system has 2 switches, each with two partitions. Therefore, for each switch, I created a single management endpoint, and mapped it to both partitions. Is that a supported configuration?

lspci shows that all 4 functions has a 4K bar size. Still, upon 'sudo insmod switchtec.ko' the kernel crashes (this time, with nothing in /var/crash - but I have recorded the console tty and it looks more or less the same).

wesleywesley commented 5 years ago

@danglus For you first question: We should support two partitions set to same management EP template, this doesn’t mean two partitions share one management EP. Two partitions share one template to create its own management EP. But haven’t test this case.

For kernel crashes, change BAR0 size to 4M.

danglus commented 5 years ago

Thanks! it works. (I did a 4K instead of 4M)