Library compatibility with xilinx apf kernel driver

[jameshilliard]

What would be required for libaxidma to interface with the xilinx apf kernel driver? I'm a little confused about the difference between the kernel driver in this project and the one xilinx maintains, am I understanding correctly that they do basically the same thing?

[bperez77]

Interesting, I haven't heard of this driver up until this point. It's good to see Xilinx finally putting some effort into adding a userspace bridge driver for DMA for embedded Linux.

I'd need to dig into the interface for the APF driver to figure out what changes would be needed. In principle, the library should only need to change its IOCTL calls to interface with the driver and how it is initialization. Now the tricky part is if the API for the APF driver is designed in a way that would be amenable with the current state of AXI DMA library. If the functions it exposes are similar, then it should be fine.

On a cursory glance, it seems like the APF driver serves a similar purpose to this driver, which is to provide a way for userspace Linux applications to interface with the AXI DMA IP (through Xilinx's backend DMA driver). So, the drivers more or less do the same thing, but they seem to have a fairly different interfaces, at least on an initial glance. While they do the same thing, it ultimately comes down to which interface you prefer.

The other thing to keep in mind is that the APF driver is sitting on the staging directory. This indicates to me that it's not quite yet fully mature, and perhaps still under active development. Though, as far as I can tell, it seems to have most of the required features necessary. Documentation on this driver seems to be pretty sparse. I can't find anything on Xilinx's site about it, other than a mention of it on an SDSoC page and a mention on a bug page. Looking at the commit history, it seems like the driver is only being updated intermittently.

I'd recommend using whichever one is easier for your use case. If you already have an example code with the APF driver, it'd probably be using it. Also, creating a "libapf" as an analogue to "libaxidma" should be fairly straight-forward; you basically just need to map IOCTL interface calls to library functions. The AXI DMA library would be a good template for that.

[jameshilliard]

The other thing to keep in mind is that the APF driver is sitting on the staging directory. This indicates to me that it's not quite yet fully mature, and perhaps still under active development.

The userspace interface appears to have been kept fairly stable for quite a while in their kernel fork at least.

I can't find anything on Xilinx's site about it, other than a mention of it on an SDSoC page and a mention on a bug page.

It appears this may have been intentional by them as it seems they want you to use their closed source library(libsds) and SDSoC toolchain that was designed to interface with this(which is not an option with what I'm trying to do).

I'd recommend using whichever one is easier for your use case.

What would work best for doing DMA without interrupts? I currently haven't had luck getting either to work.

Also, creating a "libapf" as an analogue to "libaxidma" should be fairly straight-forward; you basically just need to map IOCTL interface calls to library functions.

I was attempting to do that but it has quite a few options and operating modes it appears and my understanding of how DMA kernel drivers work is not that great, I haven't been able to figure out how to initialize it.

[bperez77]

The userspace interface appears to have been kept fairly stable for quite a while in their kernel fork at least. It appears this may have been intentional by them as it seems they want you to use their closed source library(libsds) and SDSoC toolchain that was designed to interface with this(which is not an option with what I'm trying to do).

Oh I see, that makes sense then.

What would work best for doing DMA without interrupts? I currently haven't had luck getting either to work.

Unfortunately, I think DMA without interrupts is kind of a non-starter. Linux is pretty much exclusively designed to be interrupt (event) driven, so there isn't good support for DMA polling. In principle, I think it's possible to write a custom DMA polling driver, but the Linux DMA API isn't really designed for that. This will definitely not work with my driver, and it seems also to not be supported by the APF driver as well.

I'm guessing there's a specific reason why you need polling, but I'd really recommend to switching interrupts. The other option is to basically port a Xilinx bare-metal application to C. You can do this by calling mmap on the /dev/mem device to map the AXI DMA control registers into user virtual memory. Then, your bare-metal application should be able to be used (with appropriate changes to any hard-coded addresses of course). I've seen examples of this before, but I can't seem to find them right now.

I was attempting to do that but it has quite a few options and operating modes it appears and my understanding of how DMA kernel drivers work is not that great, I haven't been able to figure out how to initialize it.

A lot of the motivation behind why I wrote this was to provide a simple, yet capable interface to AXI DMA. There have been a fair number of other attempts at a similar driver, but I generally found that they either basically directly ported of the bare-metal interface, or the interface was messy.

Which part of the initialization are you getting stuck on? I could help provide some insight.

[jameshilliard]

Linux is pretty much exclusively designed to be interrupt (event) driven, so there isn't good support for DMA polling. In principle, I think it's possible to write a custom DMA polling driver, but the Linux DMA API isn't really designed for that.

I was planning to implement the polling itself in userspace while just using the ioctl for simple read/write operations into the DMA regions.

You can do this by calling mmap on the /dev/mem device to map the AXI DMA control registers into user virtual memory.

This is basically what I'm currently doing, however there appears to be reliability issues here due to the lack of CPU cache control in userspace.

Which part of the initialization are you getting stuck on? I could help provide some insight.

I'm stuck on configuring the driver, it appears I'm supposed to call the configure ioctl but I'm not sure what to pass to it to have it mmap address regions that I can then read and write to over an ioctl.

[bperez77]

I was planning to implement the polling itself in userspace while just using the ioctl for simple read/write operations into the DMA regions. This is basically what I'm currently doing, however there appears to be reliability issues here due to the lack of CPU cache control in userspace.

Got it, I forgot about that issue, that's a good point. In order to invalidate the relevant cache entries, you would need to write a driver. As far as I know, I don't think it's possible to disable the cache. The other option is to write a driver with similar functionality as /dev/mem, except it sets the no-cached property of the pages it maps.

However, this seems like unnecessarily jumping through hoops just to get pooling to work. If you register a callback function through libaxidma, it will be delivered a Linux real-time signal, so the latency should be relatively low.

I'm stuck on configuring the driver, it appears I'm supposed to call the configure ioctl but I'm not sure what to pass to it to have it mmap address regions that I can then read and write to over an ioctl.

Yeah it's hard to get that right without decent documentation. If you're only looking to allocate DMA buffers, then you should be calling mmap on the character device. I believe that config IOCTL you pointed me at isn't for initializing the driver, it seems to be only for purpose of setting the type of a given DMA device or querying the type. This config is also automatically set when the device tree is loaded, in the function xlnk_load_config_from_dt.

[jameshilliard]

If you register a callback function through libaxidma, it will be delivered a Linux real-time signal, so the latency should be relatively low.

I would need interrupt support for that right?

Yeah it's hard to get that right without decent documentation. If you're only looking to allocate DMA buffers, then you should be calling mmap on the character device.

I figured I could use the dmarequest ioctl and dmasubmit ioctl without userspace mmaping, does that look possible?

This config is also automatically set when the device tree is loaded, in the function xlnk_load_config_from_dt.

I thought it could be set either way since xlnk_set_config() is also called here.

[bperez77]

I would need interrupt support for that right?

Yes and no. The only thing you would need to do is connect the s2mm_introut and similar ports to the IRQ port on the Zynq7 Processing System. Then, you ca regenerate the device tree. On the SW side, the driver and library already handle this for you, and setup a POSIX Real-Time Signal for if you request it. You can register a signal handler to perform some interaction on receipt of the interrupt.

Note, that when the wait option is false for any transfers, no interrupt is delivered, because the call blocks until it receives a completion interrupt.

I figured I could use the dmarequest ioctl and dmasubmit ioctl without userspace mmaping, does that look possible?

I believe that should work, though you of course won't be able to interact with at all from userspace. DMA request should give you back a physical address which you should be able to later user with the submit IOCTL.

I thought it could be set either way since xlnk_set_config() is also called here.

Yep that's correct, though you probably don't need to set any configuration, it should already be done.

You said that you're getting stuck on the set configuration IOCTL. What's the error number that you're seeing in userspace? That should give you some hint on what's going wrong.

[bperez77]

Closing this issue due to inactivity.