Rough fix for Texture.GetData being horribly slow on Vulkan backend

[kg]

Feedback welcome on how this should actually be fixed. Looking into why GetData is really slow on Vulkan, I think I found a few things:

• GetData does a synced flush which waits for the whole device to become idle. I'm not certain whether this was causing measurable slowdown, but waiting for the device to be fully idle seems wrong and unnecessary.
• SubmitCommands for vulkan does present-related work even when present isn't set.
• If present isn't set, SubmitCommands seems to not actually wait for the in-flight fence of the command buffer it just submitted, which doesn't seem like what we want.
• All the transfer buffers are allocated using the combination of HOST_VISIBLE and HOST_COHERENT, which based on vulkan documentation is the slowest type of memory. This appears to cause the SDL_memcpy in the readback operation to be incredibly slow - presumably every single read is doing an individual DMA from GPU memory. I think what we want is HOST_VISIBLE and HOST_CACHED, plus potentially some manual flushing of regions we just touched.
• Readback potentially uses the fast transfer buffer, which is allocated with preferDeviceLocal=1. For readback, I think we want one of the non-device-local slow transfer buffers, since if I understand this right the GPU will do a big DMA to copy the buffer contents over to CPU-managed memory?

The fixes I tried to apply here:

• When acquiring a transfer buffer for GetData, ensure we don't use the fast transfer buffer. Leave that one for uploads.
• When binding buffer memory, always ask for HOST_CACHED, not HOST_COHERENT. I didn't have to add any explicit flushes anywhere for uploads to still work.
• Only do some of the swapchain-related work if present!=0 in SubmitCommands.
• In SubmitCommands, if present==0, wait for the inFlightFence of the command buffer we just submitted, so that the copy is done by the time GetTextureData wants the data.
• Do a non-sync flush in GetTextureData, since we don't need to wait for the device to be idle, we just need to wait for the copy to finish.

My game is doing one small (320x180x8bpp) GetData per frame at the very start before any drawing happens, on a render target from the previous frame. Without these fixes, D3D11 gets around 1150fps and Vulkan gets ~45fps. With these fixes Vulkan gets ~950fps.

[kg]

Based on discussions on discord, I think the only essential change in here is to make sure transfer buffers are always in host memory, not device memory. Right now it's possible for transfer buffers to end up in device memory, which means writes aren't efficient huge DMAs and reads are especially-inefficient small block transfers.

The present/submit/flush stuff does seem wrong too but I don't think it was the root cause of the performance issues with GetData.

[thatcosmonaut]

Our SDL GPU proposal is approved now, and that Vulkan implementation is enormously superior to FNA3D's, so I'm closing this with the expectation that we will be deleting FNA3D_Driver_Vulkan completely in the near future.