Improve Vulkan support from vkDoom3

[RobertBeckebans]

I'm going to merge the Vulkan code from vkDoom3 over the next weekends.

This will happen in several separated steps:

• [x] I move existing OpenGL stuff in the renderer accordingly to vkDoom3 into the same places and refactor things like the renderer backend RB_ files into classes like idRendererBackend

• [x] Change renderer prog system to generate vkglsl with our custom and new renderprogs like we do with different GLSL targets

• [x] Attach glslang compiler library to generate SPIR-V code from vkglsl shaders

• [x] Merge Vulkan renderer backend from vkDoom3 to work at least with stencil shadows and no HDR

  • [x] Initialize Vulkan device and swapchain and get the loop running
  • [x] Manage Vertex Buffer Objects with Vulkan
  • [x] Manage Uniform Buffer Objects with Vulkan
  • [x] Manage to upload all Images to Vulkan
  • [x] Actually draw stuff with Vulkan

• [x] Get Imgui to work with Vulkan and replace com_drawFPS 2 with it

• [x] Get Video Playback to work with Vulkan

• [ ] Get Shadow Mapping to work with Vulkan

• [ ] Get HDR and Tone Mapping to work with Vulkan

• [ ] Get SSAO to work with Vulkan

• [ ] Get SMAA to work with Vulkan

[KungFuJesus]

Are you using the distro supplied sdk packages or manually installed LunarG sdk?

Distro supplied, though they are much newer than what Dustin had been using for vkDoom3. The same goes for glslang, which is why one of those deprecated includes is commented out. I had some trouble compiling this with GCC 9, a static assert was failing for unordered_map in glslang's code and I couldn't figure out why. You'll also need to add some links that are missing. I started to do it in the CMakeList but quickly realized that it wasn't working. Currently, I'm manually adding the following to CMake's CMAKE_EXE_LINK_FLAGS: -lglslang -lglslang-default-resource-limits -lSPIRV -lxcb. The commenting out of the substitution for the C99 nullptr was needed for me, I'm pretty sure that you can't #ifdef nullptr, as it was trying to use idNullPtr or whatever that surrogate class was and having issues casting to a real nullptr. I imagine the preprocessor doesn't know this keyword like it would a type.

Why are you trying to link to xcb using MSVC?

Sorry, I didn't do any conditional logic on the CMakeList for MSVC (and I probably should have #ifdef _WIN32'd the #define for XCB prior to #include <vulkan/vulkan.h>. I was mostly just trying to make the thing actually work in Linux.

Just use SDL2 and save yourself a lot of trouble.

I mean that was my initial instinct, but a lot of the existing WIN32 vulkan code that is already there is managing the Vulkan context. Refactoring all of this to use SDL would mean a much bigger delta, and realistically I don't think I have a windows installation I can actually test the SDL glue code on to begin with.

[ericwomer]

@KungFuJesus There is a second if USE_VULKAN for clang/gcc that you should add the lib too. Its if(MSVC) ... else() ... if USE_VULKAN ... in here ... endif()

[ericwomer]

@KungFuJesus I got it to compile and link on linux, but its a black screen right now, I don't know if any validation layers are enabled in code or not, and you should have started with RBs vulkan branch.

[KungFuJesus]

@KungFuJesus I got it to compile and link on linux, but its a black screen right now, I don't know if any validation layers are enabled in code or not, and you should have started with RBs vulkan branch.

Yes I probably should have. I turned on validation layers and the issues are more or less related to the idRenderSystem not populating proper framebuffer dimensions. I'll have to hunt that down. In the interim, I may try stashing these changes and perhaps switching over to the vulkan branch.

[ericwomer]

@KungFuJesus I got audio playing in the background, but its still a black screen.

[RobertBeckebans]

You should work with the vulcan branch as it includes the glslang code from an external git repo at a revision that compiles and links the generated GLSL shaders properly. Generating correct SPIR-V code on Linux with a different GLSL compiler than is currently used on Windows might be a bigger issue. Newer versions of glslang failed to compile the shaders and I couldn't get shaderc to link as an alternative. I can look at your branches next week in order to get it compiling on Linux. I haven't finished it yet because the Vulkan backend does not have the features of the OpenGL backend.

[ericwomer]

@KungFuJesus @RobertBeckebans I got it loading the menu and input works, I had to abandon the hard XCB coding, and used SDL2's vulkan implimentation, in the CMakeLists.txt file, if on UNIX and USE_VULKAN is enabled it will set SDL2 as on. but it fails with this message.

RBDoom3BFG: ../renderer/Vulkan/RenderProgs_VK.cpp:1539: void idRenderProgManager::CommitUniforms(uint64): Assertion `image->GetView() != VK_NULL_HANDLE' failed.

I was able to play Doom and Doom2 classic fine without any issues, it just when loading a level in Doom3 BFG it crashes with that error.

[ericwomer]

It also can't handle resizing the window.

[ericwomer]

@KungFuJesus @RobertBeckebans Everything I have done so far is my vulkan-merge-2 branch.

Note Everything on the windows side needs to be brought back, since I applied @KungFuJesus patch first.

Warning Don't merge stuff as its a mix between RB's master and vulkan, and my cmake-fixes plus KungFuJesus's patch.

[ericwomer]

I've been meaning to ask, is there anything we need to do with the base/renderprogs from the git repo? Do we need to copy them over to the /base/ folder?

[RobertBeckebans]

RBDOOM-3-BFG loads the HLSL shaders if they are located in base/renderprogs/ otherwise it loads those that are compiled into the binary - neo/renderer/RenderProgs_embed.h is generated by premake4_embed.bat. This allows to ship new shaders just by shipping the compiled binary.

[RobertBeckebans]

@KungFuJesus @RobertBeckebans I got it loading the menu and input works, I had to abandon the hard XCB coding, and used SDL2's vulkan implimentation, in the CMakeLists.txt file, if on UNIX and USE_VULKAN is enabled it will set SDL2 as on. but it fails with this message.

RBDoom3BFG: ../renderer/Vulkan/RenderProgs_VK.cpp:1539: void idRenderProgManager::CommitUniforms(uint64): Assertion `image->GetView() != VK_NULL_HANDLE' failed.

I was able to play Doom and Doom2 classic fine without any issues, it just when loading a level in Doom3 BFG it crashes with that error.

Maybe you need it to run with r_usePBR 0 in the console or with +set r_usePBR 0 as comandline argument. The vulkan branch has some new Image Based Lighting features that still rely on external textures.

[RobertBeckebans]

And make sure r_useShadowMapping is also set to 0.

[ericwomer]

@RobertBeckebans That was it, but the lighting/rendering is a little funky and black/dark, and the scene (I assume a video) where the two guys start talking when the level starts and after you land is black, but I can here the audio.

[ericwomer]

It seems to be rendering black around the models, so any cut scenes are black.

[KungFuJesus]

I managed to compile and link against shaderc but get a segfault as soon as the level is loaded. Unless maybe I'm missing some external textures like RobertBeckebans mentioned? I switched off those 2 cvars to no avail.

I also have a patch I may set a separate PR for. It adds AVX support for the RenderMatrix multiplications and some of the clipping plane calculations.

On Tue, Dec 31, 2019, 3:15 PM Eric notifications@github.com wrote:

It seems to be rendering black around the models, so any cut scenes are black.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/RobertBeckebans/RBDOOM-3-BFG/issues/398?email_source=notifications&email_token=AA3T3NPWETRSUOGHW5U4PQ3Q3OR5VA5CNFSM4DZLPS52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH4UL7Q#issuecomment-569984510, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA3T3NOWRGGAMGDNDSWX3MTQ3OR5VANCNFSM4DZLPS5Q .

[ericwomer]

Ok, the black around the models is gone, I switched over to using the Vulkan SDK from LunarG's site, since the one on Ubuntu is old, and I didn't recompile until after using RB's switches, which fixed the rendering around models, but there is no transparency, and I couldn't see into the cockpit of the ship dropping me off or the guys in the section where they scan you before you go into the base.

[KungFuJesus]

Also @eriwomer, there's an idPrintf statement I put in there trying to debug the vulkan context bringup - you'll want to comment that out as it's pretty frequent.

[ericwomer]

@KungFuJesus, which one is it?

[ericwomer]

Are you talking about all the extra idLib::PrintF during initialization?

[ericwomer]

@RobertBeckebans @KungFuJesus The black I was experiencing has nothing to do with which library it links to, its weather or not the binary is packaged with the shaders, so best bet is to do

premake4 embed

or

premake5 embed

before compiling.

Maybe we can wrap the debug printing we have now in a console variable, separate from the one that enables the validation layers? Like r_vkEnableVerboseDebugOutput or r_vkVerboseOutput so we don't loose it during a code cleanup?

[KungFuJesus]

@ericwomer I must have added it after I posted the patch, because I can't find it in there

[ericwomer]

@KungFuJesus @RobertBeckebans Enabled window resizing in my vulkan-merge-2 branch. I stops rendering until you stop resizing, but it works. Also switching to and from full-screen works, but using [alt] + [enter] keys resizes the game to full-screen but at the resolution it was when you hit those key combinations, causing some fun, but incorrect, rendering. It also causes your desktop to switch to that resolution, so hitting [alt] + [enter] doesn't fix it.

[ericwomer]

If running in renderdoc and you go to resize the window, it causes a crash. Something in specific locations causes the game to crash, and according to renderdoc, vulkan to stop presenting, but the game does not close, you have to manually close it with [alt] + [F4].

[KungFuJesus]

One thing I've noticed with this so far is switching out of and back into the window makes the framerate permanently tank. There also seems to be an issue with the shaders for the fireballs from imps.

[KungFuJesus]

There also seems to be some texture flickering and antialiasing doesn't seem to take effect.

Here's a weird texture flicker effect:

Here's the antialiasing not working (SMAA 1x). I think the multisample methods do this as well.

[RobertBeckebans]

I merged your Linux Vulkan contributions into my master branch. Good job! I managed to get it working on Kubuntu 19.10 with the open source drivers for my AMD RX 580.

[KungFuJesus]

@RobertBeckebans Perhaps I'm using a bad combination of the shaders / shader compilers, but imp fireballs and the propeller fan blades in the first cut scene still seem to have a black halo around them.

The framerate also seems to tank when you leave the window and come back to it, still. I'm using a 980 Ti, but I managed to make this work on a meager Haswell-U series chip by commenting out one of the Vulkan extensions that are being enabled. So far I haven't seen anything consequential when doing so. The named extension that seems to be causing trouble is "depthBounds". This is likely due to somewhat incomplete Vulkan support for Haswell, but as far as you know, is there any place it's actually being used?

[RobertBeckebans]

@KungFuJesus The black halos are still caused by an issue that it fails to copy the framebuffer image to the internal "_currentRender" image. That one is the input for the heatHaze shaders which usually look up it with some normal distortion.
I disabled rendering of all those effects in the master with forcing r_skipPostProcess 0 through the code. The driver needs to support the depth bound testing because it is required by the stencil shadows.

[10010101001]

Crashes at launch Hardware: AMD A9 9425, AMD R5 OS: Ubuntu 20.04 LTS

Creating Vulkan Instance...
Enumerating physical devices and their properties...
WARNING: vallium/llvmpipe is not a conformant vulkan implementation, testing use only.
found 3 devices
Iterating over gpu 0
have gpuInfo_t ref at 0x55ba69b8f5c0
set gpu.device
Getting queue family props...
Getting available vulkan extensions...
Found device[0] Vendor: Unknown (0x10005)
Iterating over gpu 1
have gpuInfo_t ref at 0x55ba69b8fbc0
set gpu.device
Getting queue family props...
Getting available vulkan extensions...
Found device[1] Vendor: AMD
Iterating over gpu 2
have gpuInfo_t ref at 0x55ba69b901c0
set gpu.device
Getting queue family props...
Getting available vulkan extensions...
Found device[2] Vendor: AMD
Selecting physical device...
Selected device 'llvmpipe (LLVM 10.0.1, 256 bits)'
Device[0] : Vendor: Unknown (0x10005)
Creating logical device and queues...
../src/gallium/frontends/vallium/val_device.c:832: VK_ERROR_FEATURE_NOT_PRESENT
Unknown command 'vid_restart'
Sys_Error: VK: VK_ERROR_FEATURE_NOT_PRESENT - vkCreateDevice( vkcontext.physicalDevice, &info, NULL, &vkcontext.device )

[danieleftodi]

Great to here, is the current parts of the refactoring process from last year going to be the starting point or will you refactor the refactor, also is there any way you could blog about your process to sort of drag us along with your learning experience? It would really help, and I think before I was just upset because it seemed most of the developing community (namely Epic and some of their employees) just considered Vulkan as mobile only and didn't take it serious as a desktop api and blew it off, and before I found out about you working on CattleAndCrops I thought you fell into that category also. I am more upset that I don't have the programming skill to learn 3D engine programming let-a-lone Vulkan to even help out with this project.

a game engine has many "lego-parts" ... not just the "3D engine" ... it too is just one tiny part of the sum off all ports in a "game engine" :-)

I'm new like you ... and have began in any way I can, to just hack at it ... and keep hacking at it :-) ... this is the super-nice part of F/OSS ... we all learn in our own temp and manner ... we just must have a "I can do" attitude and try to ask for help when ever we get stuck :-)