vktrace with trim causes vkreplay to segfault

[eero-t]

Setup:

• Ubuntu 16.04
• Mesa & VulkanTools git versions from last week

Use-case:

• Working DOTA2 trace, which replays fine with vkreplay
• Run vkreplay with vktrace set to do trimming for a frame range within the trace, for example: ./vktrace -o test.vktrace -tr frames-466-502 -p ./vkreplay -a "-o dota2.vktrace"

Result is that vkreplay crashes when the trim closes:

Info: Trim stopping now at frame: 502
*** Error in `../vktrace/vkreplay': free(): invalid size: 0x00000000ae205e80 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f8ef720e7e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7f8ef721737a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f8ef721b53c]
layersvt/../vktrace/libVkLayer_vktrace_layer.so(+0x36bc9)[0x7f8ef5354bc9]
layersvt/../vktrace/libVkLayer_vktrace_layer.so(+0x3714a)[0x7f8ef535514a]
layersvt/../vktrace/libVkLayer_vktrace_layer.so(+0x239a3)[0x7f8ef53419a3]
layersvt/../vktrace/libVkLayer_vktrace_layer.so(__HOOKED_vkQueuePresentKHR+0x368)[0x7f8ef5336518]
loader/libvulkan.so.1(vkQueuePresentKHR+0x3b)[0x7f8ef7f4c7a1]

And the resulting trimmed trace file doesn't work:

vkreplay error: Skipping vkCmdBindDescriptorSets() due to invalid remapped VkDescriptorSet.
Segmentation fault (core dumped)
...
Program received signal SIGSEGV, Segmentation fault.
emit_binding_table (cmd_buffer=cmd_buffer@entry=0x43811f0, stage=stage@entry=MESA_SHADER_VERTEX, bt_state=bt_state@entry=0x4381628) at ../../../src/intel/vulkan/genX_cmd_buffer.c:1566
1566          uint32_t offset = set->layout->binding[binding->binding].descriptor_index;
(gdb) bt
#0  emit_binding_table (cmd_buffer=cmd_buffer@entry=0x43811f0, stage=stage@entry=MESA_SHADER_VERTEX, bt_state=bt_state@entry=0x4381628) at ../../../src/intel/vulkan/genX_cmd_buffer.c:1566
LunarG/VulkanTools#1  0x00007ffff5fc8e19 in flush_descriptor_sets (cmd_buffer=0x43811f0) at ../../../src/intel/vulkan/genX_cmd_buffer.c:1774
LunarG/VulkanTools#2  gen9_cmd_buffer_flush_state (cmd_buffer=cmd_buffer@entry=0x43811f0) at ../../../src/intel/vulkan/genX_cmd_buffer.c:2010
LunarG/VulkanTools#3  0x00007ffff5fcbbb5 in gen9_CmdDraw (commandBuffer=0x43811f0, vertexCount=6, instanceCount=1, firstVertex=0, firstInstance=0) at ../../../src/intel/vulkan/genX_cmd_buffer.c:2121
LunarG/VulkanTools#4  0x00007ffff798ffe6 in vkCmdDraw () from /home/testrunner/work/VulkanTools/build/loader/libvulkan.so.1
LunarG/VulkanTools#5  0x0000000000424356 in vkReplay::replay(vktrace_trace_packet_header*) ()

Everything works fine if I trace the whole replay without trimming, and that trace can also be replayed fine, the new trace is just somewhat larger than the original trace.

Tracing DOTA2 directly works also fine, but if I set trim hotkey, DOTA2 will crash even before I enable tracing, if I just let it idle in the startup menu. It may be related to above problem.

[eero-t]

Note: trim works without segfaults for simpler things like cube trace replay.

Taking a trimmed trace of Talos Principle trace replay doesn't crash, but like with DOTA2, the resulting trace doesn't work (depending on the trim range, the trimmed trace may show errors or not, but it never renders anything resembling what should be on screen).

(Earlier LunarG/vktrace#43 issue mentions rendering issues also in simpler smoketest case.)

[eero-t]

Tested today's Mesa & yesterday's VulkanTools.

vkreplay doesn't anymore crash when trimming stops.

The trimmed DOTA2 file still doesn't work though:

$ vkreplay -o trimmed.vktrace
... lots of validation errors ...
IMAGE(ERROR): object: 0x0 type: 0 location: 3318 msgCode: 180357110: vkCreateImageView() format VK_FORMAT_B8G8R8A8_UNORM differs from image 8 format VK_FORMAT_B8G8R8A8_SRGB.  Formats MUST be IDENTICAL unless VK_IMAGE_CREATE_MUTABLE_FORMAT BIT was set on image creation. The spec valid usage text states 'If image was not created with the VK_IMAGE_CREATE_MUTABLE_FORMAT_BIT flag,' (https://www.khronos.org/registry/vulkan/specs/1.0/html/vkspec.html#VUID-VkImageViewCreateInfo-image-01019)
vkreplay error: Failed to remap VkSampler.
vkreplay error: Skipping vkUpdateDescriptorSets() due to invalid remapped VkSampler.
vkreplay error: Failed to remap VkSampler.
vkreplay error: Skipping vkUpdateDescriptorSets() due to invalid remapped VkSampler.
vkreplay error: Failed to remap VkSampler.
vkreplay error: Skipping vkUpdateDescriptorSets() due to invalid remapped VkSampler.
vkreplay error: Failed to remap VkSampler.
vkreplay error: Skipping vkUpdateDescriptorSets() due to invalid remapped VkSampler.
vkreplay error: Failed to remap VkSampler.
vkreplay error: Skipping vkUpdateDescriptorSets() due to invalid remapped VkSampler.
vkreplay error: Failed to remap VkSampler.
vkreplay error: Skipping vkUpdateDescriptorSets() due to invalid remapped VkSampler.
vkreplay error: Failed to remap VkSampler.
vkreplay error: Skipping vkUpdateDescriptorSets() due to invalid remapped VkSampler.
vkreplay error: Failed to remap VkSampler.
vkreplay error: Skipping vkUpdateDescriptorSets() due to invalid remapped VkSampler.
vkreplay error: Failed to remap VkSampler.
vkreplay error: Skipping vkUpdateDescriptorSets() due to invalid remapped VkSampler.
vkreplay error: Failed to remap VkSampler.
vkreplay error: Skipping vkUpdateDescriptorSets() due to invalid remapped VkSampler.
vkreplay error: Failed to remap VkSampler.
vkreplay error: Skipping vkUpdateDescriptorSets() due to invalid remapped VkSampler.
vkreplay error: Failed to remap VkSampler.
vkreplay error: Skipping vkUpdateDescriptorSets() due to invalid remapped VkSampler.
DS(ERROR): object: 0x5278 type: 23 location: 1187 msgCode: 61: Descriptor set 0x5278 encountered the following validation error at vkCmdDrawIndexed() time: Descriptor in binding #0 at global descriptor index 0 is being used in draw but has not been updated.
Segmentation fault (core dumped)

Segfault is NULL pointer access inside (Mesa) driver.