A way to control request lengths?

[wilderfield]

Hey, this tool is very nice.

Curious if it can or could be enhanced to support control of the request length.

I painfully discovered that my x86 Ubuntu system was sending MemRd requests formatted as 32 bit addressing, but Length == 2... so requesting 64 bits of data.

My endpoint design (Xilinx PCIE Example Design) only supports 32 bit requests.

So my endpoint just drops the request.

I think for now I am going to have to write my own PCIe User Logic.

[billfarrow]

Hey, The pcimem tool accepts an optional data type on the command line that is applied to both Read and Write operations. Did you experiment with those byte, short, and word accesses and see what pcie transaction hit your fpga implementation ?

I suspect that your problem is in the Linux kernel driver for your platforms pcie controller chip, or maybe the PCI Config parameters.

Let me know if I can help dig into this more. If you find an enhancement to pcimem I will happily roll it in.

[wilderfield]

Hey,

I tried the half word access, and that successfully worked in returning 16 bits of read data.

I think this is a classic case of the platform interpreting the C data type as whatever it wants. In this case it interprets unsigned long to be 64 bits, and the Kernel goes and fetches 2DW.

I think maybe if you used uint32 / uint64 casting in pcimem.c it might be better. I'll play around with that in the next couple of days.

So far I haven't had to touch pcimem.c it has worked beautifully so thanks for contributing that to the community.

Bryan

Sent from my iPhone

On Nov 8, 2015, at 6:12 AM, Bill Farrow notifications@github.com wrote:

Hey, The pcimem tool accepts an optional data type on the command line that is applied to both Read and Write operations. Did you experiment with those byte, short, and word accesses and see what pcie transaction hit your fpga implementation ?

I suspect that your problem is in the Linux kernel driver for your platforms pcie controller chip, or maybe the PCI Config parameters.

Let me know if I can help dig into this more. If you find an enhancement to pcimem I will happily roll it in.

— Reply to this email directly or view it on GitHub.

[billfarrow]

You could be right about it being a C data type issue on 64 bit target architecture. We should convert pcimem.c to use standard types from stdint.h: uint8_t, uint16_t and uint32_t.

[wilderfield]

Bill,

I globally replaced any occurrence of "unsigned long" with "uint32_t".

I added #include

My Xilinx endpoint design is now working with your pcimem.c utility.

I think you are spot on, that the utility could be improved by using the platform agnostic uint*_t data types.

Based on PCIE defenitions... I would propose the following access types:

uint8_t = "b" uint16_t = "w" uint32_t = "dw" uint64_t = "qw"

PCIE defines 4 bytes to be a double word, and 8 bytes to be a quad word.

Wondering if I can hit you up for some related wisdom.

I actually work at Xilinx, and I am working on an application note to create an opensource tool that will generate a basic hardware design for a board controller FPGA.

Essentially its an FPGA to control a bunch of resets/enables, and monitor board level interrupt signals.

From talking to various customers who use FPGAs in their real linux systems... the feedback was that an FPGA driver is preferred to be in user space, since its very custom hardware.

I wanted to use uio_pci_generic from the linux kernel to be the lone kernel mode driver for handling interrupts, and accessing memory.

From my WWW searches, and failed attempts... it seems that uio_pci_generic will not give me any capability to access the memory mapped PCI space.

(I tried using your pcimem.c utility, and passing it /dev/uio0 instead of /sys/bus/pci/... and mmap would fail even though the driver was successfully bound to my FPGA card.)

Does this mean for a user space PCI device drive, I must have a thread using uio_pci_generic for legacy interrupt (INTx) monitoring, and another utility that uses sysfs in the style of your pcimem.c for memory space access?

Also... your pcimem.c code is for merely printing to the screen. Is it very simple to modify it so that it is taking arguments from a calling function, and returning success/read data?

I'm trying to grasp how a user space application uses a user space driver. This is my first time experimenting with this software magic.

I've also heard about VFIO as an alternative to UIO. Wondered if you have any thoughts on that.

Thanks, Bryan

On Mon, Nov 9, 2015 at 5:09 AM, Bill Farrow notifications@github.com wrote:

You could be right about it being a C data type issue on 64 bit target architecture. We should convert pcimem.c to use standard types from stdint.h: uint8_t, uint16_t and uint32_t.

— Reply to this email directly or view it on GitHub https://github.com/billfarrow/pcimem/issues/1#issuecomment-155057383.

[billfarrow]

Hey Bryan, Would you like to go through the process of sending me a pull request for your changes ? If you are not comfortable with how github works I can make the changes myself, I just didn't want to spoil the fun.

I have not written any userspace PCI drivers. I usually develop my drivers "out of tree" modules. It is generally not to hard to keep the drivers compatible with multiple kernel versions.

I had a quick read of the UIO documentation and I think you use the uio_pci_generic driver to bind to the device and do interrupt handling, and use sysfs to mmap the various BARs into your userspace program just like pcimem.c does. What I don't understand is how you implement any type of security with UIO, everything needs to run as root.

https://www.kernel.org/doc/htmldocs/uio-howto/index.html

Bill

Bill

On Mon, Nov 9, 2015 at 2:20 PM, wilderfield notifications@github.com wrote:

Bill,

I globally replaced any occurrence of "unsigned long" with "uint32_t".

I added #include

My Xilinx endpoint design is now working with your pcimem.c utility.

I think you are spot on, that the utility could be improved by using the platform agnostic uint*_t data types.

Based on PCIE defenitions... I would propose the following access types:

unit8_t = "b" unit16_t = "w" unit32_t = "dw" unit64_t = "qw"

PCIE defines 4 bytes to be a double word, and 8 bytes to be a quad word.

Wondering if I can hit you up for some related wisdom.

I actually work at Xilinx, and I am working on an application note to create an opensource tool that will generate a basic hardware design for a board controller FPGA.

Essentially its an FPGA to control a bunch of resets/enables, and monitor board level interrupt signals.

From talking to various customers who use FPGAs in their real linux systems... the feedback was that an FPGA driver is preferred to be in user space, since its very custom hardware.

I wanted to use uio_pci_generic from the linux kernel to be the lone kernel mode driver for handling interrupts, and accessing memory.

From my WWW searches, and failed attempts... it seems that uio_pci_generic will not give me any capability to access the memory mapped PCI space.

(I tried using your pcimem.c utility, and passing it /dev/uio0 instead of /sys/bus/pci/... and mmap would fail even though the driver was successfully bound to my FPGA card.)

Does this mean for a user space PCI device drive, I must have a thread using uio_pci_generic for legacy interrupt (INTx) monitoring, and another utility that uses sysfs in the style of your pcimem.c for memory space access?

Also... your pcimem.c code is for merely printing to the screen. Is it very simple to modify it so that it is taking arguments from a calling function, and returning success/read data?

I'm trying to grasp how a user space application uses a user space driver. This is my first time experimenting with this software magic.

I've also heard about VFIO as an alternative to UIO. Wondered if you have any thoughts on that.

Thanks, Bryan

On Mon, Nov 9, 2015 at 5:09 AM, Bill Farrow notifications@github.com wrote:

You could be right about it being a C data type issue on 64 bit target architecture. We should convert pcimem.c to use standard types from stdint.h: uint8_t, uint16_t and uint32_t.

— Reply to this email directly or view it on GitHub https://github.com/billfarrow/pcimem/issues/1#issuecomment-155057383.

— Reply to this email directly or view it on GitHub https://github.com/billfarrow/pcimem/issues/1#issuecomment-155163136.

[wilderfield]

Hey Bill, I am going to close this, open issue.. I will try out the pull request.

[Mickey-Tsao-NY]

Bill, This pcimem program is wonderful. Exactly what I needed for my Xilinx FPGA PCIe work. Bryan, the uint32_t patch comes just in time for me too. Many thanks to both of you.

[billfarrow]

Pull request has been merged. Awesome !