External memory support

[christosa-rfpro]

Hello. This has been raised before, but are there any plans on supporting extensions like VK_KHR_external_memory_win32 and VK_KHR_external_memory_fd? These are particularly useful for applications that communicate data with, for example, CUDA.

[adam-sawicki-a]

Hello. How would you like this support to look like? You can use VMA with external memory currently with help of multiple features, like:

• Custom pools and VmaPoolCreateInfo::pMemoryAllocationNext which allows you to attach a custom structure to every VkDeviceMemory allocated that VMA will make in this pool.
• Use VmaAllocatorCreateInfo::pTypeExternalMemoryHandleTypes to make VMA automatically adding VkExportMemoryAllocateInfoKHR with specific handle type flags for every allocation made from a specific memory type.
• If the memory shared this way requires special alignment, use vmaCreateBufferWithAlignment or a custom pool with specific VmaPoolCreateInfo::minAllocationAlignment.

[christosa-rfpro]

Hi there, thanks for the reply. I think that setting VmaAllocatorCreateInfo::pTypeExternalMemoryHandleTypes is a nice and simple solution - I shall try that. I may come back adding a few comments here if that's ok.

[christosa-rfpro]

Hello again.

I have managed to make progress with this. I noticed, however, that there's a subtlety on Windows which I'm not sure how to deal with:

Please have a look at the code near line 1545 of the following example, which is part of the NVIDIA CUDA samples: https://github.com/NVIDIA/cuda-samples/blob/master/Samples/5_Domain_Specific/vulkanImageCUDA/vulkanImageCUDA.cu

As you may notice, one needs to set security attributes in a VkExportMemoryAllocateInfoKHR object. I can't seem to find a way of doing so with VMA, though. I believe that this is needed on Windows.

Any ideas?

[adam-sawicki-a]

Hello,

In the sample code you showed, VkMemoryAllocateInfo has pNext chain attached, made of structures VkExportMemoryAllocateInfoKHR and VkExportMemoryWin32HandleInfoKHR. You can make VMA attach similar pNext chain to every Vulkan memory allocation made within a custom pool by creating the pool with specified VmaPoolCreateInfo::pMemoryAllocationNext, as I mentioned above.

[christosa-rfpro]

Yep, I had missed that bit before. Everything is working fine this way. I create a pool per image, though, for each image that needs to be backed by external memory. I guess this is all right. Thanks for the help.

[adam-sawicki-a]

This is all right, but do you really need to create a separate pool for each image? Does each one need different parameters in this pNext chain structures? If not, please remember that you can also create dedicated allocations in custom pools, so if you need these images to each have its own device memory block, maybe you can still use a single pool and allocate them with VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT?

[christosa-rfpro]

Oh, that's good to know, thank you very much. I can see that VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT really makes a difference.