search

Xilinx / dma_ip_drivers

Xilinx QDMA IP Drivers
https://xilinx.github.io/dma_ip_drivers/
562 stars 415 forks source link

/dev/xdma0_events_0: Protocol error, read interrupt signal faild! #236

Open ARC-MX opened 1 year ago

ARC-MX commented 1 year ago

When I read the interrupt signal from /dev/xdma0_events_0, it stuck. A look at the interrupt event file reveals the following error:

$cat /dev/xdma0_events_0
cat: /dev/xdma0_events_0: Protocol error

I don't know why.

image

hinxx commented 1 year ago

I think you need to read 4 bytes at a time. Try with 

dd bs=4 count=1 if=/dev/.. | hexdump -Cv
ARC-MX commented 1 year ago

dd bs=4 count=1 if=/dev/

还是卡主了 image

或者您能给一个应用interrupt的上位机例程吗，我的上位机程序应该是没有问题的，cat /dev/interrupt 可以看到中断已经触发，但是上位机该如何检测到这个中断请求，有没有别的办法

hinxx commented 1 year ago

Sorry I don't speak/read/write Chinese.

In python you could do this to wait for an event:

    def wait_daq_done_irq(self):
        # Open the event0 device and try to read one event
        fd_handle = os.open(self.device_id + '_events_0', os.O_RDONLY)

        status = os.pread(fd_handle, 4, 0)
        os.close(fd_handle)
        return status

In C/C++ is the same approach; open the char device and read 4 bytes out. The read() will block until the XDMA kernel driver has an event to give you. See the driver source to understand when the read will succeed:

https://github.com/Xilinx/dma_ip_drivers/blob/9f02769a2eddde008158c96efa39d7edb6512578/XDMA/linux-kernel/xdma/cdev_events.c#L51-L59

You can of course use poll() / select(), too, in order to wait on an event and possibly timeout earlier to avoid blocking indefinitely.

ARC-MX commented 1 year ago

Thank you for your reply. I know this approach. I did the same thing, but when I read this handle, the program doesn't return anything and keeps waiting. I tried to check the contents of the file xdma0_event_0 using cat /dev/xdma0_events_0 and found that it gave me a protocol error. I can't find a solution to the problem. image

hinxx commented 1 year ago

AFAICT, if the read() does not return and keeps waiting, then there are no events/IRQs to report. Are you sure your FPGA firmware is generating interrupts? There are several way that the interrupts can be reported to the XDMA driver, depending on how the FPGA firmware is built; legacy or MSI. Make sure you are using the right same methid on the Linux side.

In my setup things look like this:

dmesg:

[    7.560228] xdma: loading out-of-tree module taints kernel.
[    7.560324] xdma: module verification failed: signature and/or required key missing - tainting kernel
[    7.574015] xdma:xdma_mod_init: Xilinx XDMA Reference Driver xdma v2020.2.2
[    7.574017] xdma:xdma_mod_init: desc_blen_max: 0xfffffff/268435455, timeout: h2c 10 c2h 10 sec.
[    7.575092] xdma:xdma_device_open: xdma device 0000:08:00.0, 0x000000009770ea4d.
[    7.575308] xdma:map_single_bar: BAR0 at 0xde000000 mapped at 0x00000000fa90b4ee, length=16777216(/16777216)
[    7.575318] xdma:map_single_bar: BAR1 at 0xdf000000 mapped at 0x00000000f3ad2268, length=65536(/65536)
[    7.575321] xdma:map_bars: config bar 1, pos 1.
[    7.575325] xdma:identify_bars: 2 BARs: config 1, user 0, bypass -1.
[    7.612718] xdma:pci_keep_intx_enabled: 0000:08:00.0: clear INTX_DISABLE, 0x406 -> 0x6.
[    7.612777] xdma:irq_msix_channel_setup: engine 0-H2C0-MM, irq#174.
[    7.612789] xdma:irq_msix_channel_setup: engine 0-C2H0-MM, irq#175.
[    7.612805] xdma:irq_msix_user_setup: 0-USR-0, IRQ#176 with 0x00000000d18cd994
[    7.612845] xdma:irq_msix_user_setup: 0-USR-1, IRQ#177 with 0x00000000a99ed93b
[    7.612859] xdma:irq_msix_user_setup: 0-USR-2, IRQ#178 with 0x0000000006b14317
[    7.612870] xdma:irq_msix_user_setup: 0-USR-3, IRQ#179 with 0x00000000c42e66ec
[    7.612879] xdma:irq_msix_user_setup: 0-USR-4, IRQ#180 with 0x000000001f687857
[    7.612887] xdma:irq_msix_user_setup: 0-USR-5, IRQ#181 with 0x0000000075e6dbae
[    7.612895] xdma:irq_msix_user_setup: 0-USR-6, IRQ#182 with 0x00000000b8bb1db4
[    7.612903] xdma:irq_msix_user_setup: 0-USR-7, IRQ#183 with 0x00000000013b72f9
[    7.612915] xdma:irq_msix_user_setup: 0-USR-8, IRQ#184 with 0x00000000a777c4d0
[    7.612942] xdma:irq_msix_user_setup: 0-USR-9, IRQ#185 with 0x00000000e5db671a
[    7.612959] xdma:irq_msix_user_setup: 0-USR-10, IRQ#186 with 0x00000000fe4c03a1
[    7.612968] xdma:irq_msix_user_setup: 0-USR-11, IRQ#187 with 0x000000003d0054b2
[    7.612976] xdma:irq_msix_user_setup: 0-USR-12, IRQ#188 with 0x00000000a69028d2
[    7.612984] xdma:irq_msix_user_setup: 0-USR-13, IRQ#189 with 0x000000009985dac8
[    7.612992] xdma:irq_msix_user_setup: 0-USR-14, IRQ#190 with 0x000000004366a118
[    7.613000] xdma:irq_msix_user_setup: 0-USR-15, IRQ#191 with 0x0000000088deaf8e
[    7.613008] xdma:probe_one: 0000:08:00.0 xdma0, pdev 0x000000009770ea4d, xdev 0x00000000e706f02f, 0x00000000fa257ed8, usr 16, ch 1,1.
[    7.651063] xdma:cdev_xvc_init: xcdev 0x00000000beb3d9ee, bar 0, offset 0x40000.

lspci:

$ sudo lspci -s 08:00.0 -vv
08:00.0 Serial controller: Xilinx Corporation Device 8034 (prog-if 01 [16450])
    Subsystem: Xilinx Corporation Device 0007
    Physical Slot: 3-1
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 16
    Region 0: Memory at de000000 (32-bit, non-prefetchable) [size=16M]
    Region 1: Memory at df000000 (32-bit, non-prefetchable) [size=64K]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [60] MSI-X: Enable+ Count=32 Masked-
        Vector table: BAR=1 offset=00008000
        PBA: BAR=1 offset=00008fe0
    Capabilities: [70] Express (v2) Endpoint, MSI 00
...

XDMA driver parameters

$ cat /sys/module/xdma/parameters/interrupt_mode 
0
$ cat /sys/module/xdma/parameters/poll_mode 
0
ARC-MX commented 1 year ago

My FPGA firmware is generated interrupts. I saw through debugging that the usr_irq_req was pulled up, and after a few clock cycles the usr_irq_ack responded by pulling down the usr_irq_req. I have recently changed three machines, respectively Ubuntu18.04.1, Ubuntu20.04.1 are the same situation. It was a disaster for me. image I use debug mode to build xdma driver: dmesg:

[  151.495935] xdma:xdma_mod_init: Xilinx XDMA Reference Driver xdma v2020.2.2
[  151.495942] xdma:xdma_mod_init: desc_blen_max: 0xfffffff/268435455, timeout: h2c 10 c2h 10 sec.
[  151.496020] xdma:xdma_device_open: xdma device 0000:04:00.0, 0x000000009fa99ad4.
[  151.496025] xdma:alloc_dev_instance: xdev = 0x00000000926ecdbd
[  151.496031] xdma:xdev_list_add: dev 0000:04:00.0, xdev 0x00000000926ecdbd, xdma idx 0.
[  151.496301] xdma:request_regions: pci_request_regions()
[  151.496307] xdma:map_single_bar: BAR0: 65536 bytes to be mapped.
[  151.496332] xdma:map_single_bar: BAR0 at 0x81e00000 mapped at 0x000000005d19c40a, length=65536(/65536)
[  151.496344] xdma:is_config_bar: BAR 0 is the XDMA config BAR
[  151.496348] xdma:map_bars: config bar 0, pos 0.
[  151.496350] xdma:identify_bars: xdev 0x00000000926ecdbd, bars 1, config at 0.
[  151.496353] xdma:identify_bars: 1 BARs: config 0, user -1, bypass -1.
[  151.496356] xdma:set_dma_mask: sizeof(dma_addr_t) == 8
[  151.496359] xdma:set_dma_mask: pci_set_dma_mask()
[  151.496360] xdma:set_dma_mask: Using a 64-bit DMA mask.
[  151.496378] xdma:__write_register: channel_interrupts_disable: w reg 0x2000(0x000000002e0e5b36), 0xffffffff.
[  151.496382] xdma:__write_register: user_interrupts_disable: w reg 0x2000(0x00000000dd09e041), 0xffffffff.
[  151.496385] xdma:read_interrupts: ioread32(0x000000002d048130) returned 0x00000000 (user_int_request).
[  151.496390] xdma:read_interrupts: ioread32(0x00000000f2dc418f) returned 0x00000000 (channel_int_request)
[  151.496397] xdma:probe_for_engine: found AXI H2C 0 engine, reg. off 0x0, id 0x1fc0,0x0.
[  151.496401] xdma:engine_init: channel 0, offset 0x0, dir 1.
[  151.496405] xdma:engine_init: engine 00000000e6ea7ae1 name 0-H2C0-MM irq_bitmask=0x00000001
[  151.496423] xdma:__write_register: engine_init_regs: w reg 0x12f273b51bac(0x000000003ff6f356), 0x2000000.
[  151.496427] xdma:engine_alignments: engine 00000000e6ea7ae1 name 0-H2C0-MM alignments=0x00010140
[  151.496438] xdma:engine_alignments: align_bytes = 1
[  151.496440] xdma:engine_alignments: granularity_bytes = 1
[  151.496442] xdma:engine_alignments: address_bits = 64
[  151.496444] xdma:__write_register: engine_init_regs: w reg 0x12f273b51c30(0x000000009162abda), 0xf83e1e.
[  151.496447] xdma:probe_for_engine: found AXI H2C 1 engine, reg. off 0x100, id 0x1fc0,0x1.
[  151.496462] xdma:engine_init: channel 1, offset 0x100, dir 1.
[  151.496466] xdma:engine_init: engine 000000003b4b415e name 0-H2C1-MM irq_bitmask=0x00000002
[  151.496482] xdma:__write_register: engine_init_regs: w reg 0x12f273b51b4c(0x00000000e465877a), 0x2000000.
[  151.496485] xdma:engine_alignments: engine 000000003b4b415e name 0-H2C1-MM alignments=0x00010140
[  151.496495] xdma:engine_alignments: align_bytes = 1
[  151.496497] xdma:engine_alignments: granularity_bytes = 1
[  151.496498] xdma:engine_alignments: address_bits = 64
[  151.496500] xdma:__write_register: engine_init_regs: w reg 0x12f273b51bd0(0x00000000ec3da176), 0xf83e1e.
[  151.496502] xdma:probe_for_engine: found AXI H2C 2 engine, reg. off 0x200, id 0x1fc0,0x2.
[  151.496520] xdma:engine_init: channel 2, offset 0x200, dir 1.
[  151.496523] xdma:engine_init: engine 00000000c4b83e2d name 0-H2C2-MM irq_bitmask=0x00000004
[  151.496540] xdma:__write_register: engine_init_regs: w reg 0x12f273b51aec(0x000000004e4a3d28), 0x2000000.
[  151.496542] xdma:engine_alignments: engine 00000000c4b83e2d name 0-H2C2-MM alignments=0x00010140
[  151.496552] xdma:engine_alignments: align_bytes = 1
[  151.496553] xdma:engine_alignments: granularity_bytes = 1
[  151.496555] xdma:engine_alignments: address_bits = 64
[  151.496556] xdma:__write_register: engine_init_regs: w reg 0x12f273b51b70(0x00000000d3fa570f), 0xf83e1e.
[  151.496558] xdma:probe_for_engine: found AXI H2C 3 engine, reg. off 0x300, id 0x1fc0,0x3.
[  151.496564] xdma:engine_init: channel 3, offset 0x300, dir 1.
[  151.496567] xdma:engine_init: engine 0000000019a14776 name 0-H2C3-MM irq_bitmask=0x00000008
[  151.496582] xdma:__write_register: engine_init_regs: w reg 0x12f273b51a8c(0x0000000079aeea5e), 0x2000000.
[  151.496585] xdma:engine_alignments: engine 0000000019a14776 name 0-H2C3-MM alignments=0x00010140
[  151.496595] xdma:engine_alignments: align_bytes = 1
[  151.496597] xdma:engine_alignments: granularity_bytes = 1
[  151.496598] xdma:engine_alignments: address_bits = 64
[  151.496599] xdma:__write_register: engine_init_regs: w reg 0x12f273b51b10(0x00000000eba151e6), 0xf83e1e.
[  151.496602] xdma:probe_for_engine: found AXI C2H 0 engine, reg. off 0x1000, id 0x1fc1,0x0.
[  151.496607] xdma:engine_init: channel 0, offset 0x1000, dir 2.
[  151.496611] xdma:engine_init: engine 0000000003624d6b name 0-C2H0-MM irq_bitmask=0x00000010
[  151.496627] xdma:__write_register: engine_init_regs: w reg 0x12f273b5262c(0x0000000040d69a58), 0x2000000.
[  151.496629] xdma:engine_alignments: engine 0000000003624d6b name 0-C2H0-MM alignments=0x00010140
[  151.496640] xdma:engine_alignments: align_bytes = 1
[  151.496641] xdma:engine_alignments: granularity_bytes = 1
[  151.496642] xdma:engine_alignments: address_bits = 64
[  151.496644] xdma:__write_register: engine_init_regs: w reg 0x12f273b526b0(0x00000000775d6255), 0xf83e1e.
[  151.496646] xdma:probe_for_engine: found AXI C2H 1 engine, reg. off 0x1100, id 0x1fc1,0x1.
[  151.496657] xdma:engine_init: channel 1, offset 0x1100, dir 2.
[  151.496660] xdma:engine_init: engine 0000000057cd95f7 name 0-C2H1-MM irq_bitmask=0x00000020
[  151.496674] xdma:__write_register: engine_init_regs: w reg 0x12f273b525cc(0x00000000d222435d), 0x2000000.
[  151.496677] xdma:engine_alignments: engine 0000000057cd95f7 name 0-C2H1-MM alignments=0x00010140
[  151.496686] xdma:engine_alignments: align_bytes = 1
[  151.496688] xdma:engine_alignments: granularity_bytes = 1
[  151.496689] xdma:engine_alignments: address_bits = 64
[  151.496691] xdma:__write_register: engine_init_regs: w reg 0x12f273b52650(0x00000000f076a4ca), 0xf83e1e.
[  151.496693] xdma:probe_for_engine: found AXI C2H 2 engine, reg. off 0x1200, id 0x1fc1,0x2.
[  151.496698] xdma:engine_init: channel 2, offset 0x1200, dir 2.
[  151.496701] xdma:engine_init: engine 000000007d654039 name 0-C2H2-MM irq_bitmask=0x00000040
[  151.496716] xdma:__write_register: engine_init_regs: w reg 0x12f273b5256c(0x00000000906b6f41), 0x2000000.
[  151.496719] xdma:engine_alignments: engine 000000007d654039 name 0-C2H2-MM alignments=0x00010140
[  151.496727] xdma:engine_alignments: align_bytes = 1
[  151.496729] xdma:engine_alignments: granularity_bytes = 1
[  151.496730] xdma:engine_alignments: address_bits = 64
[  151.496731] xdma:__write_register: engine_init_regs: w reg 0x12f273b525f0(0x000000006c7a9525), 0xf83e1e.
[  151.496734] xdma:probe_for_engine: found AXI C2H 3 engine, reg. off 0x1300, id 0x1fc1,0x3.
[  151.496739] xdma:engine_init: channel 3, offset 0x1300, dir 2.
[  151.496742] xdma:engine_init: engine 00000000c145653f name 0-C2H3-MM irq_bitmask=0x00000080
[  151.496756] xdma:__write_register: engine_init_regs: w reg 0x12f273b5250c(0x00000000be87b7c6), 0x2000000.
[  151.496758] xdma:engine_alignments: engine 00000000c145653f name 0-C2H3-MM alignments=0x00010140
[  151.496766] xdma:engine_alignments: align_bytes = 1
[  151.496768] xdma:engine_alignments: granularity_bytes = 1
[  151.496769] xdma:engine_alignments: address_bits = 64
[  151.496770] xdma:__write_register: engine_init_regs: w reg 0x12f273b52590(0x000000000149a96c), 0xf83e1e.
[  151.496782] xdma:enable_msi_msix: pci_enable_msi()
[  151.496870] xdma:pci_keep_intx_enabled: 0000:04:00.0: clear INTX_DISABLE, 0x406 -> 0x6.
[  151.496894] xdma:irq_msi_setup: Using IRQ#194 with 0x00000000926ecdbd
[  151.496897] xdma:__write_register: channel_interrupts_enable: w reg 0x2000(0x00000000c9d12b24), 0xffffffff.
[  151.496900] xdma:read_interrupts: ioread32(0x000000002d048130) returned 0x00000000 (user_int_request).
[  151.496910] xdma:read_interrupts: ioread32(0x00000000f2dc418f) returned 0x00000000 (channel_int_request)
[  151.496914] xdma:__write_register: user_interrupts_enable: w reg 0x2000(0x00000000c2fe8744), 0x1ffff.
[  151.496917] xdma:read_interrupts: ioread32(0x000000002d048130) returned 0x00000000 (user_int_request).
[  151.496921] xdma:read_interrupts: ioread32(0x00000000f2dc418f) returned 0x00000000 (channel_int_request)
[  151.496925] xdma:probe_one: 0000:04:00.0 xdma0, pdev 0x000000009fa99ad4, xdev 0x0000000055205c8a, 0x00000000926ecdbd, usr 16, ch 4,4.
[  151.496933] xdma:create_xcdev: xcdev 0x00000000cb516f4d, 507:1, (null), type 0x1.
[  151.497064] xdma:create_xcdev: xcdev 0x00000000e12a4c26, 507:10, (null), type 0x3.
[  151.497135] xdma:create_xcdev: xcdev 0x0000000005c93de7, 507:11, (null), type 0x3.
[  151.497202] xdma:create_xcdev: xcdev 0x00000000adff3569, 507:12, (null), type 0x3.
[  151.497263] xdma:create_xcdev: xcdev 0x000000000dd6535f, 507:13, (null), type 0x3.
[  151.497326] xdma:create_xcdev: xcdev 0x00000000dfc70f80, 507:14, (null), type 0x3.
[  151.497376] xdma:create_xcdev: xcdev 0x000000005388ae0d, 507:15, (null), type 0x3.
[  151.497407] xdma:create_xcdev: xcdev 0x00000000d7f617ba, 507:16, (null), type 0x3.
[  151.497439] xdma:create_xcdev: xcdev 0x000000004c927a6b, 507:17, (null), type 0x3.
[  151.497496] xdma:create_xcdev: xcdev 0x00000000502d70a7, 507:18, (null), type 0x3.
[  151.497529] xdma:create_xcdev: xcdev 0x00000000d1dd8eef, 507:19, (null), type 0x3.
[  151.497561] xdma:create_xcdev: xcdev 0x00000000dc29e983, 507:20, (null), type 0x3.
[  151.497613] xdma:create_xcdev: xcdev 0x0000000017baa901, 507:21, (null), type 0x3.
[  151.497645] xdma:create_xcdev: xcdev 0x000000000c31f2d1, 507:22, (null), type 0x3.
[  151.497674] xdma:create_xcdev: xcdev 0x000000007360dc5e, 507:23, (null), type 0x3.
[  151.497706] xdma:create_xcdev: xcdev 0x00000000b43ce344, 507:24, (null), type 0x3.
[  151.497736] xdma:create_xcdev: xcdev 0x00000000e77c2f81, 507:25, (null), type 0x3.
[  151.497768] xdma:create_xcdev: xcdev 0x00000000119324a1, 507:32, (null), type 0x4.
[  151.497802] xdma:create_xcdev: xcdev 0x000000000229aa7d, 507:33, (null), type 0x4.
[  151.497833] xdma:create_xcdev: xcdev 0x00000000eb533c1b, 507:34, (null), type 0x4.
[  151.497863] xdma:create_xcdev: xcdev 0x00000000a3025c1e, 507:35, (null), type 0x4.
[  151.497926] xdma:create_xcdev: xcdev 0x0000000016d2f9a8, 507:36, (null), type 0x5.
[  151.497964] xdma:create_xcdev: xcdev 0x000000008a4ca4b5, 507:37, (null), type 0x5.
[  151.497996] xdma:create_xcdev: xcdev 0x000000001122ef06, 507:38, (null), type 0x5.
[  151.498030] xdma:create_xcdev: xcdev 0x000000005c463f5b, 507:39, (null), type 0x5.

lspci:

04:00.0 Serial controller: Xilinx Corporation Device 9038 (prog-if 01 [16450])
        Subsystem: Xilinx Corporation Device 0007
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 194
        Region 0: Memory at 81e00000 (32-bit, non-prefetchable) [size=64K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 00000000fee00b98  Data: 0000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 10.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s (ok), Width x1 (downgraded)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range BC, TimeoutDis+, NROPrPrP-, LTR-
                         10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-, TPHComp-, ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
                         EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [1c0 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
                LaneErrStat: 0
        Kernel driver in use: xdma

XDMA driver parameters:

(base) ➜  parameters cat /sys/module/xdma/parameters/interrupt_mode
1
(base) ➜  parameters cat /sys/module/xdma/parameters/poll_mode
0
hinxx commented 1 year ago

Nothing jumps out to me, sorry.

Maybe a long shot (or you are already aware of this) but, my buddy who does firmware design pointed out a peculiarity of the IRQ handling that might be interesting to you.

According to the https://docs.xilinx.com/r/en-US/pg195-pcie-dma/User-Interrupts you must keep the usr_irq_req bits stable for long enough until the XDMA kernel driver comes and reads the https://docs.xilinx.com/r/en-US/pg195-pcie-dma/IRQ-Block-User-Interrupt-Request-0x40 register.

Seen in the https://github.com/Xilinx/dma_ip_drivers/blob/9f02769a2eddde008158c96efa39d7edb6512578/XDMA/linux-kernel/xdma/libxdma.c#L1404-L1420.

Looking at the MSI vs. MSI-X user IRQ handling it seems to be different; for MSI-X the user_irq_service() is invoked without reading the above mentioned register.. hope that helps!

wei-zhanpeng commented 11 months ago

I have also encountered the same issue(My FPGA firmware is generated interrupts. I saw through debugging that the usr_irq_req was pulled up, and after a few clock cycles the usr_irq_ack responded by pulling down the usr_irq_req ,but read can't get usr_irq_req ,so are there other functions used to receive user interrupts), with the same configuration . Have you resolved this issue now and could you please let me know where the problem is.thank you!

dmesg

  1 [    1.091980] xdma: loading out-of-tree module taints kernel.
  2 [    1.092047] xdma: module verification failed: signature and/or required key missing - tainting kernel 
  3 [    1.093132] xdma:xdma_mod_init: Xilinx XDMA Reference Driver xdma v2020.2.2
  4 [    1.093134] xdma:xdma_mod_init: desc_blen_max: 0xfffffff/268435455, timeout: h2c 10 c2h 10 sec.
  5 [    1.093632] xdma:xdma_device_open: xdma device 0000:2e:00.0, 0x00000000039ff6bc.
  6 [    1.094740] xdma:map_single_bar: BAR0 at 0xfca10000 mapped at 0x000000000611c709, length=4096(/4096)
  7 [    1.094750] xdma:map_single_bar: BAR1 at 0xfca00000 mapped at 0x00000000dfdc2336, length=65536(/65536)
  8 [    1.094753] xdma:map_bars: config bar 1, pos 1. 
  9 [    1.094755] xdma:identify_bars: 2 BARs: config 1, user 0, bypass -1.          
  10 [    1.095739] xdma:pci_keep_intx_enabled: 0000:2e:00.0: clear INTX_DISABLE, 0x406 -> 0x6. 
  11 [    1.095766] xdma:probe_one: 0000:2e:00.0 xdma0, pdev 0x00000000039ff6bc, xdev 0x00000000d479958d, 
  12 [    1.124953] xdma:cdev_xvc_init: xcdev 0x000000002077df55, bar 0, offset 0x40000. 
  13 [  108.109082] xdma:remove_one: pdev 0x00000000039ff6bc, xdev 0x00000000d479958d, 0x00000000cbf9e42a.
  14 [  108.109087] xdma:xpdev_free: xpdev 0x00000000d479958d, destroy_interfaces, xdev 0x00000000cbf9e42a.
  15 [  108.109752] xdma:xpdev_free: xpdev 0x00000000d479958d, xdev 0x00000000cbf9e42a xdma_device_close.
  16 [  108.161516] xdma:xdma_mod_init: Xilinx XDMA Reference Driver xdma v2020.2.2
  17 [  108.161519] xdma:xdma_mod_init: desc_blen_max: 0xfffffff/268435455, timeout: h2c 10 c2h 10 sec.  
  18 [  108.161574] xdma:xdma_device_open: xdma device 0000:2e:00.0, 0x00000000039ff6bc.
  19 [  108.161907] xdma:map_single_bar: BAR0 at 0xfca10000 mapped at 0x00000000bfdca5b8, length=4096(/4096)
  20 [  108.161917] xdma:map_single_bar: BAR1 at 0xfca00000 mapped at 0x000000009d29d31d, length=65536(/65536)
  21 [  108.161919] xdma:map_bars: config bar 1, pos 1.
  22 [  108.161922] xdma:identify_bars: 2 BARs: config 1, user 0, bypass -1.                             
  23 [  108.161948] xdma:xdma_thread_add_work: 0-H2C0-MM 0x00000000fd1533eb assigned to cmpl status thread 
  24 [  108.161966] xdma:xdma_thread_add_work: 0-C2H0-MM 0x00000000db34fe12 assigned to cmpl status thread 
  25 [  108.162018] xdma:pci_keep_intx_enabled: 0000:2e:00.0: clear INTX_DISABLE, 0x406 -> 0x6.
  26 [  108.162037] xdma:probe_one: 0000:2e:00.0 xdma0, pdev 0x00000000039ff6bc, xdev 0x0000000089cc3ca0, 
  27 [  108.163604] xdma:cdev_xvc_init: xcdev 0x00000000c65e608b, bar 0, offset 0x40000.

lspci

2e:00.0 Serial controller: Xilinx Corporation Device 7028 (prog-if 01 [16450])
  Subsystem: Xilinx Corporation Device 0007
  Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
  Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
  Latency: 0, Cache Line Size: 64 bytes
  Interrupt: pin A routed to IRQ 101
  Region 0: Memory at fca10000 (32-bit, non-prefetchable) [size=4K]
  Region 1: Memory at fca00000 (32-bit, non-prefetchable) [size=64K]
  Capabilities: [40] Power Management version 3
    Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
    Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
  Capabilities: [48] MSI: Enable+ Count=1/1 Maskable- 64bit+
    Address: 00000000fee00000  Data: 0000
  Capabilities: [60] Express (v2) Endpoint, MSI 00
    DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
      ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 75.000W
    DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
      RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
      MaxPayload 256 bytes, MaxReadReq 512 bytes
    DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
    LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited
      ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
    LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
      ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
    LnkSta: Speed 5GT/s (ok), Width x8 (ok)
      TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
    DevCap2: Completion Timeout: Range B, TimeoutDis-, NROPrPrP-, LTR-
      10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
      EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
      FRS-, TPHComp-, ExtTPHComp-
      AtomicOpsCap: 32bit- 64bit- 128bitCAS-
    DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
      AtomicOpsCtl: ReqEn-
    LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
      Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
      Compliance De-emphasis: -6dB
    LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
      EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
  Capabilities: [100 v1] Device Serial Number 00-00-00-00-00-00-00-00
  Kernel driver in use: xdma
  Kernel modules: xdma

xdma driver parameter

cat /sys/module/xdma/parameters/interrupt_mode
0
cat /sys/module/xdma/parameters/poll_mode
0

my usr intr code are as follows

//open usr event
int open_event(char *devicename)
{
  int fd;
  fd=open(devicename,O_RDWR|O_SYNC );
  //enable user intr
  //xdma_user_isr_enable(xdev,MASK_INTR);
  if((fd)==-1){
    printf("open event error\n");
    return -1;
  } 
  else printf("open event success\n");
  return fd;
}
//read event
int read_event(int fd){
  int val;
  int rslt;
  rslt = read(fd,&val,4);
  printf("read event is %d\n",rslt);
  return val;
}
//intr proces
void *event_process(){
  u_int32_t user_irq;
  //read event
  intr_val = read_event(usr_intr_fd);
  //user_irq = read_register(irq_regs->user_int_request); 
    //printf("user_irq = 0x%08x\n", user_irq); 
  printf("intr value is %x\n",intr_val);
  //try get event
  while(1){
    if(intr_val>0){
      sem_post(&sem_rx);
      printf("get intr once\n");
    }
  }
}  

int main(int argc,char *argv[]){
  int ret;
  //sem init ,init 0
  sem_init(&sem_rx,0,0);
  printf("init sem_rx 0\n");
  //open device intr
  usr_intr_fd = open_event("/dev/xdma0_events_0");
  //create intr thread
  ret = pthread_create(&event_thread,NULL,event_process,NULL);
  if(ret==0) printf("thread create success\n");
  else{
    fprintf(stderr,"can't create thread:%s\n",strerror(ret));
    exit(1);
  }
  printids("main thread:");
  //wait intr
  while(1){
    sem_wait(&sem_rx);
    intr_cnt++;
    printf("intr get %d times\n",intr_cnt);
    if(intr_cnt==10) {
      sem_destroy(&sem_rx);
      printf("intr destroy\n");
      break;
    }
  }
}

my environment is

ubuntu20.04   
vivado2018.3
Linux qwe-MS-7C35 5.15.0-86-generic #96~20.04.1-Ubuntu SMP Thu Sep 21 13:23:37 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
americanvain commented 5 days ago

dd bs=4 count=1 if=/dev/

还是卡主了 image

或者您能给一个应用interrupt的上位机例程吗，我的上位机程序应该是没有问题的，cat /dev/interrupt 可以看到中断已经触发，但是上位机该如何检测到这个中断请求，有没有别的办法

卡住是正常的，意味着主机目前在等待FPGA的中断触发。我XDMA中的user_interrupts引脚接了开发板上的按键，此时按下，这个命令就会继续运行。

americanvain commented 5 days ago

问题解决了，只能读4字节，否则协议错误