Unable to read from user box address map using pcimem

[108anup]

I am using following command to read the first address of the user box @ 250MHz. sudo ./pcimem /sys/bus/pci/devices/0000:3b:00.0/resource2 0x100000

I get following error:

/sys/bus/pci/devices/0000:3b:00.0/resource2 opened.
Target offset is 0x100008, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0x100000)
Error at line 111, file pcimem.c (22) [Invalid argument]

From the docs on mmap (https://linux.die.net/man/2/mmap) this error suggests "We don't like addr, length, or offset (e.g., they are too large, or not aligned on a page boundary)." Since this command sudo ./pcimem /sys/bus/pci/devices/0000:3b:00.0/resource2 0x10480 works fine and since pcimem does page alignment, I am guessing the issue is that we are reading a large offset (0x100000 vs 0x10480). Is there a work around to read the large offset?

Based on https://github.com/billfarrow/pcimem/issues/13#issuecomment-790076145, I tried removing the onic kernel module, if that interferes with the mmap, but that did not help.

Additional details:

Note, in the 250MHz user box, I instantiated an axi_lite_register (independent clock mode, with register using axis clock (250 MHz) and axi-lite using 125MHz clock) that returns the value maintained by an axi_stream_size_counter (snooping on tx slave axis interface) on address offset 0. I additionally generated a reset signal in the axis clock domain (used for axi_lite_register) by updating the generic_reset module instance.

I can try running the pcimem command on vanilla open nic shell, but it will likely give the same error as the command fails to read addresses mapped to the 322 MHz box as well which uses the default p2p plugin.

pcimem ➜  sudo ./pcimem /sys/bus/pci/devices/0000:3b:00.0/resource2 0x200000
/sys/bus/pci/devices/0000:3b:00.0/resource2 opened.
Target offset is 0x200000, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0x200000)
Error at line 111, file pcimem.c (22) [Invalid argument]

For reference, reading 0x10480 gives output:

/sys/bus/pci/devices/0000:3b:00.0/resource2 opened.
Target offset is 0x10480, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0x10480)
PCI Memory mapped to address 0x7fd1268e9000.
0x10480: 0x0000A618

Based on my understanding, this address corresponds to SLR1 temperature in sysmon module and A618 translates to around 49 deg C (based on the transfer function in sysmon docs).

[108anup]

Additionally, is there a recommended simulation framework to use with open nic shell? E.g., icarus verilog, verilator don't have good support for system verilog used by many modules in open nic shell.

[108anup]

The largest address I can read is 0xFFFFF. Example:

sudo ./pcimem /sys/bus/pci/devices/0000:3b:00.0/resource2 0xFFFFF "b*1"
/sys/bus/pci/devices/0000:3b:00.0/resource2 opened.
Target offset is 0xfffff, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0xfffff)
PCI Memory mapped to address 0x7f2498e60000.
0xFFFFF: 0xFF

The resource2 size is actually 1MB instead of 4MB, as seen in ls -l: -rw------- 1 root root 1048576 Nov 19 17:26 resource2 This also explains why the largest readable byte is 1048576 - 1 = 2^20 - 1 = 0x100000 - 0x1 = 0xFFFFF.

This can also be seen in lspci -vv:

3b:00.0 Memory controller: Xilinx Corporation Device 903f
        Subsystem: Xilinx Corporation Device 0007
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 199
        NUMA node: 0
        Region 0: Memory at b8600000 (64-bit, non-prefetchable) [size=256K]
        **Region 2: Memory at b8500000 (64-bit, non-prefetchable) [size=1M]**
        Capabilities: <access denied>
        Kernel driver in use: onic

This file shows BAR2 is assumed to be 4 MB: https://github.com/Xilinx/open-nic-shell/blob/223929be072483c7a5c04eb949e20cdfdd6415d2/src/system_config/system_config_address_map.sv#L18

[108anup]

This patch: https://github.com/Xilinx/open-nic-shell/commit/b084599b77a4a1552c766ed83d07a5b6bcd07d50 only updated bar2 size for U280 (src/qdma_subsystem/vivado_ip/qdma_no_sriov_au280.tcl). Changing the bar2 size for U50 works.

Perhaps all the bar2 sizes can be updated in the upstream open-nic-shell. Hence keeping issue open.

[cneely-amd]

@108anup Thanks for finding that. I agree that the bar2 size should be updated.

[cneely-amd]

Additionally, is there a recommended simulation framework to use with open nic shell? E.g., icarus verilog, verilator don't have good support for system verilog used by many modules in open nic shell.

In a much earlier version of open-nic, before it was posted publicly to github, we had support for simulating the middle datapath between CMAC and QDMA (without these two IP), using UVM agents in a sort of hardware software co-design testbench. However, the structure of open-nic-shell was refactored since then and the RTL simulation testbench was not updated for that. I still have some of these early files that haven't been posted.

In terms of simulating the design, I'll comment that Vivado simulator and Mentor's Questa have been tried. The design uses some IP from Vivado's IP catalog. In a lot of cases Vivado's IP use IEEE 1735 encryption, which is supported by the mainstream commercial simulators, but might not be supported by the simulators that you mentioned.

[cneely-amd]

@108anup I modified the bar2 sizes for the other devices (U50, U200, U250, and SOC250) to be consistent with the U280 at 4MB, and I checked in these changes (commit: 7daf2422ed05b410abce6a98c3f01811efd04af4).