Add a kernel of Vulkan pipeline for VR

[jheo4]

There are a couple of steps to solve to this issue.

• [x] add vulkan to FleXR build system
• [x] create a simple kernel of simple drawing to test FleXR with Vulkan w/o inputs
• [x] Then, create a simple VR kernel with mock inputs; I would help adding mock input kernels

[sb-renderpass]

Do you have a VM or docker image with the dependencies pre-installed to replicate the build environment?

[jheo4]

@sb-renderpass As mentioned, you can pull the docker image jheo4/flexr:focal from the docker hub. The image is for cpu-only because the GPU setting is more complicated because of the dependencies. I set it up with all the dependencies including Vulkan with LLVM pipeline.

In your task, I think it would be fine to develop with the CPU image till now and put some references which can be helpful for setting up.

https://www.pluralsight.com/guides/create-docker-images-docker-hub https://askubuntu.com/questions/1161646/is-it-possible-to-run-docker-container-and-show-its-graphical-application-window

In short, you can pull the container image from my dockerhub repo, and create a container with the connected display. :) Please try and build flexr on it. Since we have a scheduled meeting tomorrow, I can help you more if you face during these processes.

[sb-renderpass]

Progress Update:

• Rendered a basic cube mesh with diffuse lighting.
• Considered assimp but felt too overkill/heavyweight for our simple use case so used fx-gltf instead to load glTF assets.
• Open source assets obtained from official Khronos registry: glTF-Sample-Models/2.0

Next Milestone:

• Add a hardcoded perspective projection camera
• Support textured geometry with texturing shaders
• Support more complex geometry with multiple meshes in a scene

[sb-renderpass]

Progress Update:

• Rendering complex 3D scene with multiple meshes and materials (textures)
• Some basic graphical features like normal mapping, ACES filmic tone mapping, half-lambert shading, mipmapping, 16x anisotropic filtering, and sRGB rendering
• Added more Vulkan abstractions and helper methods (Took some considerable effort coding and debugging to get all this working, especially with Vulkan being so verbose)

Next Milestone:

• Add basic perspective camera (currently using hard-coded matrices)
• Render frame from a pose (position+rotation) so upstream kernel can push poses into this renderer kernel
• Optimize the rendered result readback (currently CPU blocks until GPU is ready) via an asynchronous approach. Not currently slow but for a real-time streaming system we do not want to block the entire pipeline.

Not going to add anymore graphical features as this is a good enough baseline and more features would take more time. More focus on FleXR moving forward. However, in the long-term we could consider adding the following:

• Anti-aliasing: Something like MSAA or FXAA to eliminate "jaggies" while the camera is moving
• Physically Based Rendering (PBR): More realistic and accurate lighting model
• Multiple lights: Different types, colors, positions, maybe even animated
• Shadows:: Static and dynamic (real-time)
• Optimizations: View frustum culling, pre-depth pass, light culling, etc.
• VR specific: Multi-view rendering, lens warping, asynchronous time warp (ATW), etc.

[jheo4]

It looks great. There is much good progress. BTw, I have a quick question. As you add cameras, do you think there is a way to render them by single path? As an example, when you think of VR headsets, there are two eye cameras. Since each camera needs its own rendering instructions, it will require two pathes. I just want to hear from you.

[sb-renderpass]

Yes, and that's what is done in VR pipelines.

There are several ways to have 2 (or more) cameras with a single renderer for VR. One approach is to input a real camera and internally generate 2 virtual cameras at an offset from one another. Then internally render twice but configured to output the 2 rendered frames on the same image (or can be separate too). This can be done fairly easily with instanced rendering. explanation

However, there are a lot improvements to this approach that exploit the redundancy in data between the 2 cameras. For example, some Vulkan extensions like VK_KHR_multiview were specfically designed for this. example