cpaDcCompressData failing with CPA_DC_EP_HARDWARE_ERR

[iomartin]

Hello,

I'm working on a project that uses SPDK and qatlib with stateless compression. I started by copying the init and compression functions from the stateless_sample and it worked.

But I need to make some changes to better integrate with my application. One of them is to change how the source/dest buffers (pFlatBuffer->pData) are allocated.

When I stop using qaeMemAllocNUMA() and start using another allocation function (spdk_dma_malloc()), my compression requests (cpaDcCompressData()) start to fail with CPA_DC_EP_HARDWARE_ERR. I'd appreciate some help figuring out what the problem is.

Some other information that might be relevant:

• Both the input and output buffers are 2MB each.
• I've switched to using the appropriate virt_to_phys function.
• spdk_dma_malloc() gives me physically contiguous memory (for requests <= 2MB).
• That function allocates using hugepages, but I don't think this is the problem (I've modified qaeMemAllocNUMA() to use hugepages as well and it worked)
• spdk_dma_malloc() allocates memory by creating a file on /dev/hugepages and then doing an mmap like this:

  mmap(<addr>, 2097152, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE | MAP_FIXED, <fd>, 0)

  Where both <fd> points to the file on /dev/hugepages and <addr> is non-NULL.

• My platform is an Intel(R) Xeon(R) Platinum 8480+ which has 4940 QATs.

[iomartin]

I brought this issue up with the SPDK community. The conclusion was the IOMMU is the issue, the SPDK functions will map the IOMMU for the devices they control, but since the QAT isn't one of them, this causes a translation problem.

I see that qaeMemAllocNUMA maps the IOMMU, so it seems like we would need a way to do the same with externally allocated memory. Some way of calling something like dma_map_slab().

I'm digging through the code to see if I can find a way of doing something like this, but if there is an easier way, please let me know

[iomartin]

I couldn't find any functions in the API to do the mapping, is there one that I missed?

dma_map_slab() is so simple that I can just write my own, but for that I'd need the vfio_container_fd. I could maybe follow to see how it is opened and replicate that, but it seems there is a better solution.

get_vfio_fd() already exists and does just what the name implies, but it is unfortunately not being exported.

Would it be possible to export that function? It seems like this was originally the intent, as this functions is not used by anyone else. Although probably might make more sense to just export dma_map_slab() (or some wrapper around it) directly.

[gcabiddu]

Hi @iomartin - what you reported is correct.

QATlib uses vfio to provide DMA isolation between processes. When running your application you are getting an error since the device is not able to access the buffer provided (the IOMMU is blocking the write).

I guess you are compiling qatlib from sources. If yes, can you try to expose get_vfio_fd() (or dma_map_slab()) and see if you can get it to work with your application?

Here is documentation that explains how to use the VFIO APIs.

I'm going to check internally if we can expose one of those functions in a future release of QATlib.

[iomartin]

Thanks @gcabiddu!

I exported dma_map_slab() and then called it for every allocation I did with SPDK. And it worked! For now I'll use my fork of qatlib but would appreciate it if this can get upstream.

For anyone reading this in the future: One issue I initially had was that I was allocating many small buffers, and when I tried to map those, it would fail. But of course the mappings need to have a size that is a multiple of the page size (this is not a qatlib issue). As a workaround, I always passed 2MB to dma_map_slab() (I'm using hugepages). Naturally this is the wrong thing to do in any serious deployment but it was sufficient for a POC.

[gcabiddu]

Feature request replicated internally - closing ticket.