RenderDoc capture fails and gpu data is tampered

[manhnt9]

Description

May relate to #677

1. RenderDoc fails to capture:

• Hit F12, app quits
• Use Queue Capture, app also quits

Terminal log after launching, F12 and Queue Capture produce no logs.

./qrenderdoc 
Log     - QRenderDoc initialising.
Irrlicht Engine version 1.8.3-baw
Linux 4.15.0-38-generic #41-Ubuntu SMP Wed Oct 10 10:59:38 UTC 2018 x86_64
Creating X window...
Log     - Got remote handshake: GPU_Mesh [2877]
Using renderer: OpenGL 4.5.0
GeForce GTX 1050 Ti/PCIe/SSE2: NVIDIA Corporation
GLSL version: 45.0
Loaded texture: /home/manh/Desktop/gpu_buffer/skydome.jpg
Log     - Used API: OpenGL (Not presenting & supported)
Log     - Used API: OpenGL (Presenting & supported)

2. GPU buffer data

When running with RenderDoc (no frame capturing), the app should render the cube on screen but it doesn't. Cube still appears when running outside RenderDoc such as from Terminal.

Repro steps

gpu_buffer.zip

Extract above file, Open RenderDoc and run GPU_Mesh executabe.

Source code: https://github.com/buildaworldnet/IrrlichtBAW/tree/master/examples_tests/03.GPU_Mesh (uploaded version has been removed std::cin sky choice and changed shader/texture path for simplicity)

Environment

• RenderDoc build: v1.1
• Operating System: Ubuntu 18.04
• API: OpenGL
• GPU: GTX 1050Ti
• Driver: 390.87

[devshgraphicsprogramming]

The worst part is that it actually stops the "straight forward" pipeline from working.

Persistently Mapped Buffer -> OpenGL copy buffers -> immutable device local unmappable unupdatable buffer

This can also be duplicated with example 05 (runs as standalone executable without needing any textures) https://github.com/buildaworldnet/IrrlichtBAW/blob/ad42894989fc9c45bf1f9eca659dea089d3dec0c/examples_tests/05.SpherePoints/bin/SpherePoints

This is worrying, as not only the capturing of a buffer is broken, but whole program operation while running through RenderDoc

I use

RenderDoc nightly from 12 Nov Ubuntu 18.04 Nvidia Driver 410 on 1060 Mobile

[baldurk]

Buffer mapping, particularly persistent mapped buffers, requires interception to capture in most cases so it's not too surprising that there can be bugs with some paths based on how many possible combinations and interactions there are in GL.

I'm trying to build the IrrlichtBAW project on windows so I can debug, just running into some compile problems that I'm fixing up as I go (I'll let you know what I come up with separately in case you find it useful).

[devshgraphicsprogramming]

In that case we flush explicitly so you should know what we've written.

The interesting thing is that we now use a WRITE_ONLY buffer for uploads whereas before we used read/write buffers for everything.

[baldurk]

Thanks, I think I understand the problem. I've got it building now so I'm investigating.

For reference these are the problems I ran into while building on VS2015. Some of these fixes might be invalid but you might know how to fix better:

• include/irr/core/alloc/address_allocator_traits.h(80): supportsArbitraryOrderFrees isn't available in one template instantiation where AddressAlloc=irr::scene::IMeshSceneNodeInstanced::InstanceDataAddressAllocator.
• include/irr/core/alloc/address_allocator_traits.h(139): the std::declval<U&>().template multi... syntax really confused the compiler. Removing the template worked but I'm sure that's not the right fix.
• include/irr/core/alloc/AlignedBase.h(49): in DummyForConditional defining void as most_aligned_type broke things - std::alignment_of(void) is not allowed it seems. I changed it to char and it worked.
• include/irr/core/alloc/HeterogenousMemoryAddressAllocatorAdaptor.h(95): HeterogenousMemoryAddressAllocatorAdaptor inheriting AddressAllocator as private broke later in ResizableHeterogenousMemoryAllocator::multi_free_addr that expected to access AddressAllocator::reserved_size which was inaccessible. Changing to public inheritance fixed it.
• include/irr/core/EventDeferredHandler.h(93): constructing singleWaitTimePt fails to cast between the custom time_point and time_point<system_clock>. I changed <system_clock> to <Clock, Duration> and it worked OK.
• include/irr/video/HostDeviceMirrorBufferAllocator.h(47): the memcpy line tries to add to a void* which isn't valid as it's an unsigned pointer, casting lastAllocation.first to char * fixed it.
• include/irr/video/StreamingTransientDataBuffer.h(165): The operator= needs a return *this; on the end which is missing.
• source/Irrlicht/CMeshSceneNodeInstanced.cpp: A few cases an array is created on the stack with a dynamic size, which I think is a C-only extension that some compilers support but isn't in C++. I just replaced these with a leaking new[] so a proper fix would be needed.
• source/Irrlicht/COpenGLDriver.cpp(425): The call to _IRR_DELETE_ARRAY is missing a count parameter, but AuxContexts seemed to just be allocated with new[] so I changed this to delete[].
• src/irr/video/GPUMemoryAllocatorBase.cpp(14): The definition of GPUMemoryAllocatorBase::min_alignment is missing a noexpect from the header.
• A lot of places used assert which wasn't found. I added an include of assert.h in irrMath.h which seemed to catch them all.
• The source/Irrlicht/CMakeLists.txt was missing the cpp files in src/irr/video which are needed to build. It also refers to CGPUTransientBuffer.cpp which doesn't exist. I don't know if maybe you use another build system, I only saw the cmake one.
• Also not fatal, I removed the SSL dependency really easily as it's only used inside CBAWFile.cpp. I guess though this is maybe a hard dependency when not used with simple example tests.

[baldurk]

I believe this is a bug in the Irrlicht code:

struct IDriver declares a function virtual void flushMappedMemoryRanges with a default empty implementation. class COpenGLDriver declares it as flushBufferRanges which doesn't override the interface function - meaning the empty base class implementation is run and the map is never flushed in the first place.

Fixing the presumed typo in COpenGLDriver causes the example to work correctly and capture fine in RenderDoc with all data coming through for me. I'd recommend using the override specifier liberally - particularly with virtual interfaces being implemented - to turn problems like this into compile errors.

[devshgraphicsprogramming]

oh crap, sorry for wasting your time!

[manhnt9]

Glad that it's resolved. Thank you @baldurk.