search

aws / aws-fpga

Official repository of the AWS EC2 FPGA Hardware and Software Development Kit
Other
1.51k stars 516 forks source link

SDAccel HDK IPI Flow Interrupt Issue #430

Closed yangensh closed 5 years ago

yangensh commented 6 years ago

Hi, I am integrating our IP into SDAccel Environment. What I have done are inside IPI GUI. I have checked "Use Auxiliary (non-AXI) Signal Ports" in "Enable IP Interface" tab inside VIVADO. Also I connect four interrupt signals to cl_sh_apppf_irq_req[3:0]. And I have successfully generate the AGFI ID.

But when I load the image to FPGA slot 0, I could not find the total four interrupts from the output of command "sudo lspci -xxxvvvv". Only one interrupt is available. Where are the other 3 interrupts ?

Below are the command output.

00:1e.0 Memory controller: Device 1d0f:1041
    Subsystem: Xilinx Corporation Device 0007
    Physical Slot: 30
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0
    Region 0: Memory at 85618000 (64-bit, prefetchable) [size=16K]
    Region 2: Memory at 8561c000 (64-bit, prefetchable) [size=16K]
    Region 4: Memory at 85000000 (64-bit, prefetchable) [size=4M]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [70] Express (v2) Endpoint, MSI 00
        DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 25.000W
        DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 128 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
        LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
             EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
00: 0f 1d 41 10 06 00 10 00 00 00 80 05 00 00 80 00
10: 0c 80 61 85 00 00 00 00 0c c0 61 85 00 00 00 00
20: 0c 00 00 85 00 00 00 00 00 00 00 00 ee 10 07 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 00 00 00 00
40: 01 70 03 00 08 00 00 00 05 70 80 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 11 70 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 10 00 02 00 23 80 64 00 10 28 09 00 03 f1 43 00
80: 00 00 03 11 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 16 00 00 00 00 00 00 00 0e 00 00 00
a0: 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 0f 1d 10 f0 dd fe 51 1d 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Could anyone help to locate where the four interrupts are ? Or Anything I have missed ? Thanks !

kevin-xilinx commented 6 years ago

Looking at the output of your lspci command and examining the PCIe specification, The Message Control Register for MSI indicates '80' hex. The 0 bit is an indicator of whether or not MSI is enabled. In this case it is not. It is also worth nothing that according to how bits 3:1 are set, there is only 1 MSI vector requested. This means that all 4 interrupt lines from the FPGA will map into this vector, meaning the interrupts are 'muxed' into a single interrupt line. For more details on how these lines are mapped, you can look at PG195 page 55 Table 2-89 (http://www.xilinx.com/support/documentation/ip_documentation/xdma/v4_1/pg195-pcie-dma.pdf).

yangensh commented 5 years ago

Hi, We have enabled MSI. And in the waveform of Vivado ILA, the single clock pulse of irq_req[0] from CL to SH has been successfully monitored, it is synchronous to clk_main_a0_out. But we could not get irq_ack[0]. And the software interrupt routine could not be invoked. I have no idea where the interrupt is. Any guidance to address the issue will be greatly appreciated ! Below is the output of lspci. 

00:1d.0 Memory controller: Amazon.com, Inc. Device f010
    Subsystem: Device fedd:1d51
    Physical Slot: 29
    Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Region 0: Memory at 82000000 (32-bit, non-prefetchable) [size=32M]
    Region 1: Memory at 85400000 (32-bit, non-prefetchable) [size=2M]
    Region 2: Memory at 85600000 (64-bit, prefetchable) [size=64K]
    Region 4: Memory at 2000000000 (64-bit, prefetchable) [size=128G]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [60] MSI-X: Enable+ Count=33 Masked-
        Vector table: BAR=2 offset=00008000
        PBA: BAR=2 offset=00008fe0
    Capabilities: [70] Express (v2) Endpoint, MSI 00
        DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 25.000W
        DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 128 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
        LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
             EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
    Kernel driver in use: dpu
    Kernel modules: xocl
00: 0f 1d 10 f0 02 04 10 00 00 00 80 05 00 00 80 00
10: 00 00 00 82 00 00 40 85 0c 00 60 85 00 00 00 00
20: 0c 00 00 00 20 00 00 00 00 00 00 00 dd fe 51 1d
30: 00 00 00 00 40 00 00 00 00 00 00 00 00 00 00 00
40: 01 60 03 00 08 00 00 00 05 60 80 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 11 70 20 80 02 80 00 00 e2 8f 00 00 00 00 00 00
70: 10 00 02 00 23 80 64 00 10 28 00 00 03 f1 43 00
80: 00 00 03 11 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 16 00 00 00 00 00 00 00 0e 00 00 00
a0: 03 00 1e 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 0f 1d 10 f0 dd fe 51 1d 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

00:1e.0 Memory controller: Amazon.com, Inc. Device 1041
    Subsystem: Xilinx Corporation Device 0007
    Physical Slot: 30
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0
    Region 0: Memory at 85618000 (64-bit, prefetchable) [size=16K]
    Region 2: Memory at 8561c000 (64-bit, prefetchable) [size=16K]
    Region 4: Memory at 85000000 (64-bit, prefetchable) [size=4M]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [70] Express (v2) Endpoint, MSI 00
        DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 25.000W
        DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 128 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
        LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
             EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
00: 0f 1d 41 10 06 00 10 00 00 00 80 05 00 00 80 00
10: 0c 80 61 85 00 00 00 00 0c c0 61 85 00 00 00 00
20: 0c 00 00 85 00 00 00 00 00 00 00 00 ee 10 07 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 00 00 00 00
40: 01 70 03 00 08 00 00 00 05 70 80 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 11 70 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 10 00 02 00 23 80 64 00 10 28 00 00 03 f1 43 00
80: 00 00 03 11 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 16 00 00 00 00 00 00 00 0e 00 00 00
a0: 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 0f 1d 10 f0 dd fe 51 1d 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
kevin-xilinx commented 5 years ago

It appears that Bus Master Enable has not been set, shown in the Control section of the lspci report. In your original report, it showed it was enabled "BusMaster+", but your recent log shows it is not enabled "BusMaster-". Bus Master Enable must be enabled (+) so that the IP can send the MSI/MSI-X interrupt to the Host.

If you use a driver, you should be able to use the PCIe Kernel call pci_set_master(). But if you don't have a driver and just want to brute force it, you can run the setpci command with Root access.

Example:

setpci -s 00:1d.0 04.b=6
-s <bus>:<device>.<function> : Taken from the lspci output
04 : PCI Express Configuration Space offset for the “Control” register : You can see the Configuration Space layout in the PCIe PG (http://www.xilinx.com/support/documentation/ip_documentation/pcie4_uscale_plus/v1_3/pg213-pcie4-ultrascale-plus.pdf)
b : Byte length (just need to get 8 bits)
= : Issue a write with the following data. Without this, you will do a read instead
6 : write value 7 to retain the Memory Enable bit, and set the Bus Master enable bit. The bit definition of the Command register or any other PCIe register will be as defined in the PCI-SIG PCIe spec.

Just to note, it appears that you have both MSI and MSI-X enabled. When both are enabled, MSI-X will take precedence so you will see MSI-X and not the MSI interrupt going to the Host.