search

ValveSoftware / Fossilize

A serialization format for various persistent Vulkan object types.
MIT License
585 stars 47 forks source link

Use after free when compiling GPL pipelines #235

Closed pendingchaos closed 1 year ago

pendingchaos commented 1 year ago

When replaying control.56e940357de1ece7.foz from our fossil-db (with RADV_DEBUG=nocache RADV_FORCE_FAMILY=gfx1100 and --num-threads 1), it will eventually crash:

fossilize-replay: ../../../../../../../mesa/src/vulkan/runtime/vk_object.h:101: vk_object_base_assert_valid: Assertion `base == NULL || base->type == obj_type' failed.
Fossilize ERROR: Graphics pipeline crashed or hung: 4265a17c6a6a3944. Rerun with: --graphics-pipeline-range 1284 1285.

This is:

From the ASan output, I decided it's Fossilize bug.

Output when running with ASan ``` ================================================================= ==78062==ERROR: AddressSanitizer: heap-use-after-free on address 0x62000014c4fc at pc 0x7f82a0e6eb32 bp 0x7f8285bbf270 sp 0x7f8285bbea30 READ of size 4 at 0x62000014c4fc thread T3 #0 0x7f82a0e6eb31 in __interceptor_memcpy (/usr/lib64/libasan.so.8+0x6eb31) (BuildId: e5f0a0d511a659fbc47bf41072869139cb2db47f) #1 0x7f8298141462 in blob_copy_bytes ../../../../../../../mesa/src/util/blob.c:301 #2 0x7f8298141946 in blob_read_uint32 ../../../../../../../mesa/src/util/blob.c:324 #3 0x7f8298ade447 in decode_type_from_blob ../../../../../../../mesa/src/compiler/glsl_types.cpp:3081 #4 0x7f8298a52bda in read_variable ../../../../../../../mesa/src/compiler/nir/nir_serialize.c:365 #5 0x7f8298a549cb in read_var_list ../../../../../../../mesa/src/compiler/nir/nir_serialize.c:443 #6 0x7f8298a6daf1 in nir_deserialize ../../../../../../../mesa/src/compiler/nir/nir_serialize.c:2073 #7 0x7f829785c1b0 in radv_pipeline_import_retained_shaders ../../../../../../../mesa/src/amd/vulkan/radv_pipeline_graphics.c:2360 #8 0x7f829785ce53 in radv_pipeline_load_retained_shaders ../../../../../../../mesa/src/amd/vulkan/radv_pipeline_graphics.c:2392 #9 0x7f8297862d50 in radv_graphics_pipeline_compile ../../../../../../../mesa/src/amd/vulkan/radv_pipeline_graphics.c:2702 #10 0x7f8297879010 in radv_graphics_pipeline_init ../../../../../../../mesa/src/amd/vulkan/radv_pipeline_graphics.c:3975 #11 0x7f829787ae61 in radv_graphics_pipeline_create ../../../../../../../mesa/src/amd/vulkan/radv_pipeline_graphics.c:4074 #12 0x7f829787caf0 in radv_CreateGraphicsPipelines ../../../../../../../mesa/src/amd/vulkan/radv_pipeline_graphics.c:4217 #13 0x42b9fa in ThreadedReplayer::worker_thread(unsigned int) (/home/pendingchaos/valve-dev/Fossilize/build/cli/fossilize-replay+0x42b9fa) (BuildId: 8e68b05e509c476827f7d2574c59e4d64c6d9f5c) #14 0x7f82a0ae31f2 in execute_native_thread_routine (/lib64/libstdc++.so.6+0xe31f2) (BuildId: 65f174befecd93031976ef77c0c77fdcea95fdc8) #15 0x7f82a08ae946 in start_thread (/lib64/libc.so.6+0x8c946) (BuildId: c9f62793b9e886eb1b95077d4f26fe2b4aa1ac25) #16 0x7f82a093486f in __clone3 (/lib64/libc.so.6+0x11286f) (BuildId: c9f62793b9e886eb1b95077d4f26fe2b4aa1ac25) 0x62000014c4fc is located 1148 bytes inside of 3600-byte region [0x62000014c080,0x62000014ce90) freed by thread T0 here: #0 0x7f82a0ed7fb8 in __interceptor_free.part.0 (/usr/lib64/libasan.so.8+0xd7fb8) (BuildId: e5f0a0d511a659fbc47bf41072869139cb2db47f) #1 0x7f829787c800 in radv_destroy_graphics_lib_pipeline ../../../../../../../mesa/src/amd/vulkan/radv_pipeline_graphics.c:4195 #2 0x7f82977f5d34 in radv_pipeline_destroy ../../../../../../../mesa/src/amd/vulkan/radv_pipeline.c:97 #3 0x7f82977f5fa1 in radv_DestroyPipeline ../../../../../../../mesa/src/amd/vulkan/radv_pipeline.c:126 #4 0x4300e7 in std::_Function_handler(std::vector >*, std::unordered_map, std::equal_to, std::allocator > > const&, std::unordered_map, std::equal_to, std::allocator > >&, std::vector >&, std::vector > const&, unsigned int)::{lambda()#2}>::_M_invoke(std::_Any_data const&) (/home/pendingchaos/valve-dev/Fossilize/build/cli/fossilize-replay+0x4300e7) (BuildId: 8e68b05e509c476827f7d2574c59e4d64c6d9f5c) previously allocated by thread T3 here: #0 0x7f82a0ed81e5 in __interceptor_realloc.part.0 (/usr/lib64/libasan.so.8+0xd81e5) (BuildId: e5f0a0d511a659fbc47bf41072869139cb2db47f) #1 0x7f829813f9ef in blob_finish_get_buffer ../../../../../../../mesa/src/util/blob.c:136 #2 0x7f829785b2e4 in radv_pipeline_retain_shaders ../../../../../../../mesa/src/amd/vulkan/radv_pipeline_graphics.c:2319 #3 0x7f829785e8c7 in radv_graphics_shaders_compile ../../../../../../../mesa/src/amd/vulkan/radv_pipeline_graphics.c:2523 #4 0x7f8297863b2a in radv_graphics_pipeline_compile ../../../../../../../mesa/src/amd/vulkan/radv_pipeline_graphics.c:2764 #5 0x7f829787c298 in radv_graphics_lib_pipeline_init ../../../../../../../mesa/src/amd/vulkan/radv_pipeline_graphics.c:4153 #6 0x7f829787c572 in radv_graphics_lib_pipeline_create ../../../../../../../mesa/src/amd/vulkan/radv_pipeline_graphics.c:4176 #7 0x7f829787ca30 in radv_CreateGraphicsPipelines ../../../../../../../mesa/src/amd/vulkan/radv_pipeline_graphics.c:4215 #8 0x42b9fa in ThreadedReplayer::worker_thread(unsigned int) (/home/pendingchaos/valve-dev/Fossilize/build/cli/fossilize-replay+0x42b9fa) (BuildId: 8e68b05e509c476827f7d2574c59e4d64c6d9f5c) Thread T3 created by T0 here: #0 0x7f82a0e48956 in pthread_create (/usr/lib64/libasan.so.8+0x48956) (BuildId: e5f0a0d511a659fbc47bf41072869139cb2db47f) #1 0x7f82a0ae32c8 in std::thread::_M_start_thread(std::unique_ptr >, void (*)()) (/lib64/libstdc++.so.6+0xe32c8) (BuildId: 65f174befecd93031976ef77c0c77fdcea95fdc8) SUMMARY: AddressSanitizer: heap-use-after-free (/usr/lib64/libasan.so.8+0x6eb31) (BuildId: e5f0a0d511a659fbc47bf41072869139cb2db47f) in __interceptor_memcpy Shadow bytes around the buggy address: 0x62000014c200: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x62000014c280: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x62000014c300: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x62000014c380: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x62000014c400: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd =>0x62000014c480: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd[fd] 0x62000014c500: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x62000014c580: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x62000014c600: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x62000014c680: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x62000014c700: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==78062==ABORTING ```
HansKristian-Work commented 1 year ago

I'm able to reproduce and building Fossilize with asan does expose something:

=================================================================
==22336==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000060d40 at pc 0x55ee1bd1aa2c bp 0x7fd55e9fd660 sp 0x7fd55e9fd650
WRITE of size 8 at 0x603000060d40 thread T2
    #0 0x55ee1bd1aa2b in ThreadedReplayer::complete_work_item(PipelineWorkItem const&) const (/home/maister/git/Fossilize/cmake-build-debug/cli/fossilize-replay+0xe1a2b) (BuildId: a81709a321a5faea2d28f812eb7865121a652e4a)
    #1 0x55ee1bd1b69d in ThreadedReplayer::run_creation_work_item_graphics_iteration(PipelineWorkItem const&, VkPipelineCache_T*, bool) (/home/maister/git/Fossilize/cmake-build-debug/cli/fossilize-replay+0xe269d) (BuildId: a81709a321a5faea2d28f812eb7865121a652e4a)
    #2 0x55ee1bd1b833 in ThreadedReplayer::run_creation_work_item_graphics(PipelineWorkItem const&) (/home/maister/git/Fossilize/cmake-build-debug/cli/fossilize-replay+0xe2833) (BuildId: a81709a321a5faea2d28f812eb7865121a652e4a)
    #3 0x55ee1bd1cad6 in ThreadedReplayer::run_creation_work_item(PipelineWorkItem const&) (/home/maister/git/Fossilize/cmake-build-debug/cli/fossilize-replay+0xe3ad6) (BuildId: a81709a321a5faea2d28f812eb7865121a652e4a)
    #4 0x55ee1bd1d772 in ThreadedReplayer::worker_thread(unsigned int) (/home/maister/git/Fossilize/cmake-build-debug/cli/fossilize-replay+0xe4772) (BuildId: a81709a321a5faea2d28f812eb7865121a652e4a)
    #5 0x55ee1bde8350 in void std::__invoke_impl<void, void (ThreadedReplayer::*)(unsigned int), ThreadedReplayer*, unsigned int>(std::__invoke_memfun_deref, void (ThreadedReplayer::*&&)(unsigned int), ThreadedReplayer*&&, unsigned int&&) (/home/maister/git/Fossilize/cmake-build-debug/cli/fossilize-replay+0x1af350) (BuildId: a81709a321a5faea2d28f812eb7865121a652e4a)
    #6 0x55ee1bde8179 in std::__invoke_result<void (ThreadedReplayer::*)(unsigned int), ThreadedReplayer*, unsigned int>::type std::__invoke<void (ThreadedReplayer::*)(unsigned int), ThreadedReplayer*, unsigned int>(void (ThreadedReplayer::*&&)(unsigned int), ThreadedReplayer*&&, unsigned int&&) (/home/maister/git/Fossilize/cmake-build-debug/cli/fossilize-replay+0x1af179) (BuildId: a81709a321a5faea2d28f812eb7865121a652e4a)
    #7 0x55ee1bde80ac in void std::thread::_Invoker<std::tuple<void (ThreadedReplayer::*)(unsigned int), ThreadedReplayer*, unsigned int> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>) (/home/maister/git/Fossilize/cmake-build-debug/cli/fossilize-replay+0x1af0ac) (BuildId: a81709a321a5faea2d28f812eb7865121a652e4a)
    #8 0x55ee1bde7feb in std::thread::_Invoker<std::tuple<void (ThreadedReplayer::*)(unsigned int), ThreadedReplayer*, unsigned int> >::operator()() (/home/maister/git/Fossilize/cmake-build-debug/cli/fossilize-replay+0x1aefeb) (BuildId: a81709a321a5faea2d28f812eb7865121a652e4a)
    #9 0x55ee1bde7f81 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (ThreadedReplayer::*)(unsigned int), ThreadedReplayer*, unsigned int> > >::_M_run() (/home/maister/git/Fossilize/cmake-build-debug/cli/fossilize-replay+0x1aef81) (BuildId: a81709a321a5faea2d28f812eb7865121a652e4a)
    #10 0x7fd5726e1942 in execute_native_thread_routine /usr/src/debug/gcc/gcc/libstdc++-v3/src/c++11/thread.cc:104
    #11 0x7fd5724aa9ea  (/usr/lib/libc.so.6+0x8c9ea) (BuildId: 023ea16fd6c04ef9cf094507024e6ecdb35e02ca)
    #12 0x7fd57252e71b  (/usr/lib/libc.so.6+0x11071b) (BuildId: 023ea16fd6c04ef9cf094507024e6ecdb35e02ca)

I'll investigate.

HansKristian-Work commented 1 year ago

Fixed on master.