MalleshKoti
commented
7 years ago XML used in the guest for hostdev device:
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x1'/>
</hostdev>
cdeadmin
Guest failed to start with CX5 passthrough, same time host hit with EEH :
[ 1315.195786] mlx5_0:wait_for_async_commands:732:(pid 3719): done with all pending requests [ 1317.267625] mlx5_1:wait_for_async_commands:732:(pid 3719): done with all pending requests [ 1317.762382] EEH: PHB#0 failure detected, location: N/A [ 1317.762435] CPU: 0 PID: 91 Comm: kworker/0:1 Not tainted 4.13.0-4.dev.git49564cb.el7.centos.ppc64le #1 [ 1317.762590] Workqueue: events work_for_cpu_fn [ 1317.762710] Call Trace: [ 1317.762764] [c000003fe7f87930] [c000000000ac906c] dump_stack+0xb0/0xf4 (unreliable) [ 1317.762889] [c000003fe7f87970] [c00000000003b3b0] eeh_dev_check_failure+0x1f0/0x590 [ 1317.763016] [c000003fe7f87a10] [c0000000000a1204] pnv_pci_read_config+0xc4/0x140 [ 1317.763136] [c000003fe7f87a50] [c0000000005ccff4] pci_bus_read_config_word+0xb4/0x100 [ 1317.763265] [c000003fe7f87ab0] [c0000000005d653c] pci_raw_set_power_state+0xfc/0x280 [ 1317.763388] [c000003fe7f87b40] [c0000000005da310] pci_set_power_state+0xd0/0x1d0 [ 1317.763518] [c000003fe7f87b80] [c000000000785e2c] vfio_pci_probe+0x13c/0x250 [ 1317.763634] [c000003fe7f87bd0] [c0000000005dfe3c] local_pci_probe+0x6c/0x130 [ 1317.763751] [c000003fe7f87c60] [c0000000001169e8] work_for_cpu_fn+0x38/0x60 [ 1317.763862] [c000003fe7f87c90] [c00000000011bc00] process_one_work+0x1a0/0x490 [ 1317.763985] [c000003fe7f87d30] [c00000000011c168] worker_thread+0x278/0x520 [ 1317.764089] [c000003fe7f87dc0] [c0000000001244a8] kthread+0x168/0x1b0 [ 1317.764201] [c000003fe7f87e30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c [ 1317.764320] EEH: Detected error on PHB#0 [ 1317.764363] EEH: This PCI device has failed 1 times in the last hour [ 1317.764516] EEH: Notify device drivers to shutdown [ 1317.764615] EEH: Collect temporary log [ 1317.764681] PHB4 PHB#0 Diag-data (Version: 1) [ 1317.764762] brdgCtl: 0000ffff [ 1317.764821] RootSts: ffffffff ffffffff ffffffff ffffffff 0000ffff [ 1317.764924] RootErrSts: ffffffff ffffffff ffffffff [ 1317.765009] RootErrLog: ffffffff ffffffff ffffffff ffffffff [ 1317.765112] sourceId: ffffffff [ 1317.765167] nFir: 0000800000000000 0030001c00000000 0000800000000000 [ 1317.765271] PhbSts: 0000001800000000 0000001800000000 [ 1317.765358] Lem: 0000000100300100 ffffffffffffffff 0000000000000000 [ 1317.765467] PhbErr: 00000c8000000000 0000008000000000 2148000098000240 a008400000000000 [ 1317.765598] RxeArbErr: 4000000020000000 4000000000000000 0050000400000000 0000000000000000 [ 1317.765727] RxeMrgErr: 0000000000000001 0000000000000001 0000000000000000 0000000000000000 [ 1317.765859] PblErr: 0000000001000000 0000000001000000 0000000000000000 0000000000000000 [ 1317.765991] PcieDlp: 0000000000000000 0000000000000000 0fd5000000000000 [ 1317.766099] RegbErr: 0040004e54200800 0040000000000000 62000a1018000000 1800000000000000 [ 1317.766233] EEH: Reset without hotplug activity [ 1317.792802] vfio-pci 0000:01:00.0: Refused to change power state, currently in D3 [ 1317.792918] iommu: Removing device 0000:01:00.0 from group 0 [ 1317.822800] mlx5_core 0000:01:00.0: Refused to change power state, currently in D3 [ 1317.822903] mlx5_core 0000:01:00.0: Using 64-bit DMA iommu bypass [ 1317.823786] mlx5_core 0000:01:00.0: firmware version: 65535.65535.65535 [ 1327.853266] mlx5_core 0000:01:00.0: Firmware over 10000 MS in pre-initializing state, aborting [ 1327.853363] mlx5_core 0000:01:00.0: mlx5_load_one failed with error code -16 [ 1327.853588] mlx5_core: probe of 0000:01:00.0 failed with error -16 [ 1327.887193] Unable to handle kernel paging request for data at address 0x00000008 [ 1327.887349] Faulting instruction address: 0xc000000000ad6c98 [ 1327.887462] Oops: Kernel access of bad area, sig: 11 [#1] [ 1327.887553] SMP NR_CPUS=1024 [ 1327.887554] NUMA [ 1327.887610] PowerNV [ 1327.887688] Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx5_ib ib_core at24 opal_prd ofpart ipmi_powernv ipmi_devintf powernv_flash ipmi_msghandler i2c_opal mtd kvm_hv nfsd auth_rpcgss oid_registry nfs_acl lockd kvm grace [ 1327.888930] sunrpc ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect tg3 sysimgblt fb_sys_fops ttm drm mlx5_core i2c_core ptp pps_core [ 1327.889168] CPU: 14 PID: 492 Comm: systemd-journal Not tainted 4.13.0-4.dev.git49564cb.el7.centos.ppc64le #1 [ 1327.889334] task: c000003fe7a14a00 task.stack: c000003fdf660000 [ 1327.889432] NIP: c000000000ad6c98 LR: c0000000003c63c8 CTR: c0000000003c4430 [ 1327.889551] REGS: c000003fdf663ac0 TRAP: 0300 Not tainted (4.13.0-4.dev.git49564cb.el7.centos.ppc64le) [ 1327.889704] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> [ 1327.889718] CR: 28002828 XER: 00000000 [ 1327.889874] CFAR: c0000000003c63c4 DAR: 0000000000000008 DSISR: 40000000 SOFTE: 1 GPR00: c0000000003c6318 c000003fdf663d40 c000000001397a00 c00020397b4e9e00 GPR04: c000000002330368 c00020397552ded0 c00020397552de00 c00020397d455f00 GPR08: 0000000000000000 0000000000000000 c00020397d455f00 c000000000af0f48 GPR12: c0000000003c4430 c00000000fd88c00 000000007552de00 0000000000000000 GPR16: 0000000000000000 0000000000000019 0000000000000000 c00020397b4e9e58 GPR20: c00020397b4e9e18 0000000000000000 0000000000000000 c000000002330308 GPR24: c000000002330300 00000000de045700 c00020397552de30 c000003fde045700 GPR28: c00020397b4e9e00 c00020397552de00 c000000002330368 c00020397552ded0 [ 1327.891071] NIP [c000000000ad6c98] rb_insert_color+0x18/0x1a0 [ 1327.891183] LR [c0000000003c63c8] SyS_epoll_ctl+0x828/0xc10 [ 1327.891273] Call Trace: [ 1327.891315] [c000003fdf663d40] [c0000000003c6318] SyS_epoll_ctl+0x778/0xc10 (unreliable) [ 1327.891454] [c000003fdf663e30] [c00000000000b8e0] system_call+0x58/0x6c [ 1327.891565] Instruction dump: [ 1327.891623] e8410018 7fc9f378 7fbeeb78 4bffff9c 60000000 60420000 e9430000 2faa0000 [ 1327.891756] 419e0154 e92a0000 792807e1 4c820020 61270001 7d254b78 7faa4040
[ 1327.891897] ---[ end trace 6efc2b1ce3f4f85b ]---
Aafter this adapter is not recovered.. and we do not see the adapter in host 'lspci' until we reboot the host.
Steps to Reproduce:
Edit the guest in shutdown mode and add CX5 as hostdev device :
Start the guest, it fails to boot and Call Trace is seen on host where we hit EEH
Here, we are using in-box driver as MOFED is not present yet in HostOS.
Attaching lspci -vvv before starting guest: lspci_vvv.txt