thedmd / imgui-node-editor

Node Editor built using Dear ImGui
MIT License
3.7k stars 545 forks source link

Segfault in ImGui_ImplVulkan_RenderDrawData with pcmd->UserCallback = 0xfffffffffffffffe #267

Open azonenberg opened 11 months ago

azonenberg commented 11 months ago

I'm getting intermittent crashes in the ngscopeclient filter graph editor (using my fork at https://github.com/ngscopeclient/imgui-node-editor although most of the changes are build/warning related and I haven't found any way they could be causing problems).

I'm attempting to develop a reproducible test case but have so far not been successful.

What I do know is that the crash is due to the sentinel value ImDrawCallback_ImCanvas being called - which as far as I know should not ever happen.

Thread 1 "ngscopeclient" received signal SIGSEGV, Segmentation fault.
--Type <RET> for more, q to quit, c to continue without paging--c
0xfffffffffffffffe in ?? ()
(gdb) bt
#0  0xfffffffffffffffe in  ()
#1  0x0000555555c4a548 in ImGui_ImplVulkan_RenderDrawData(ImDrawData*, VkCommandBuffer_T*, VkPipeline_T*) (draw_data=0x612000005c68, command_buffer=0x621000074550, pipeline=0x61c0000188d0)
    at /ceph/fast/home/azonenberg/code/scopehal-apps/src/imgui/backends/imgui_impl_vulkan.cpp:566
#2  0x0000555556b145c1 in VulkanWindow::Render() (this=0x620000009080) at /ceph/fast/home/azonenberg/code/scopehal-apps/src/ngscopeclient/VulkanWindow.cpp:519
#3  0x0000555556869443 in MainWindow::Render() (this=0x620000009080) at /ceph/fast/home/azonenberg/code/scopehal-apps/src/ngscopeclient/MainWindow.cpp:397
#4  0x0000555556bab310 in main(int, char**) (argc=2, argv=0x7fffffffdf78) at /ceph/fast/home/azonenberg/code/scopehal-apps/src/ngscopeclient/main.cpp:127
(gdb) frame 2
#2  0x0000555556b145c1 in VulkanWindow::Render (this=0x620000009080) at /ceph/fast/home/azonenberg/code/scopehal-apps/src/ngscopeclient/VulkanWindow.cpp:519
519                     ImGui_ImplVulkan_RenderDrawData(main_draw_data, *cmdBuf);
(gdb) print cmdBuf
$1 = (vk::raii::CommandBuffer &) @0x603000069a90: {static objectType = vk::ObjectType::eCommandBuffer, static debugReportObjectType = vk::DebugReportObjectTypeEXT::eCommandBuffer, m_device = {
    static objectType = vk::ObjectType::eDevice, static debugReportObjectType = vk::DebugReportObjectTypeEXT::eDevice, m_device = 0x62d000258450}, m_commandPool = {static objectType = vk::ObjectType::eCommandPool, 
    static debugReportObjectType = vk::DebugReportObjectTypeEXT::eCommandPool, m_commandPool = 0x616000036cd0}, m_commandBuffer = {static objectType = vk::ObjectType::eCommandBuffer, 
    static debugReportObjectType = vk::DebugReportObjectTypeEXT::eCommandBuffer, m_commandBuffer = 0x621000074550}, m_dispatcher = 0x6210000b9100}
(gdb) print main_draw_data
$2 = (ImDrawData *) 0x612000005c68
(gdb) frame 1
#1  0x0000555555c4a548 in ImGui_ImplVulkan_RenderDrawData (draw_data=0x612000005c68, command_buffer=0x621000074550, pipeline=0x61c0000188d0) at /ceph/fast/home/azonenberg/code/scopehal-apps/src/imgui/backends/imgui_impl_vulkan.cpp:566
566                         pcmd->UserCallback(cmd_list, pcmd);
(gdb) print pcmd
$3 = (const ImDrawCmd *) 0x61c0000108b8
(gdb) print cmd_list
$4 = (const ImDrawList *) 0x6190001c6460
(gdb) print pcmd
$5 = (const ImDrawCmd *) 0x61c0000108b8
(gdb) print pcmd->UserCallback
$6 = (ImDrawCallback) 0xfffffffffffffffe
azonenberg commented 11 months ago

My nodes are doing custom draw commands in them so perhaps that's related, the sentinel is not being found in the expected state?

(gdb) frame 1
p#1  0x0000555555c4a548 in ImGui_ImplVulkan_RenderDrawData (draw_data=0x612000005c68, command_buffer=0x621000074550, pipeline=0x61c0000188d0) at /ceph/fast/home/azonenberg/code/scopehal-apps/src/imgui/backends/imgui_impl_vulkan.cpp:566
566                         pcmd->UserCallback(cmd_list, pcmd);
(gdb) print cmd_i
$7 = 1
(gdb) print cmd_list->CmdBuffer.Size
$8 = 11
azonenberg commented 11 months ago

I now have a semi reproducible test case, at least if you're able to build and run ngscopeclient.

I added the following code to imgui_canvas.cpp after the check for the sentinel:

        //DEBUG: Search the *entire* draw list for the sentinel command
        for(int i=0; i<m_DrawList->CmdBuffer.size(); i++)
        {
            if(m_DrawList->CmdBuffer[i].UserCallback == ImDrawCallback_ImCanvas)
            {
                fprintf(stderr, "found and removed sentinel at offset %d, expected %d or %d\n",
                    i,
                    m_DrawListCommadBufferSize,
                    m_DrawListCommadBufferSize-1);

                m_DrawList->CmdBuffer.erase(m_DrawList->CmdBuffer.Data + i);
            }
        }

The fault occurs while mousing over the Divide_1 node in the graph editor, at which point I get

found and removed sentinel at offset 1, expected 3 or 2

For whatever reason, it appears critical that the right hand "data" output port is partially clipped by the edge of the canvas. If the entire port text is on or off screen, the fault does not occur.

fault-example

azonenberg commented 11 months ago

In general, the fault is triggered by mousing over any filter node that has output ports which are partially clipped by the edge of the canvas. Instrument channels do not appear to trigger it regardless of clip position.

thedmd commented 10 months ago

@azonenberg

ImDrawCallback_ImCanvas is a sentinel value and should not be left in draw command after canvas is done. ImGui internal plumbing changed and that exposed some issues.

There were fixes around that issue some time ago. Can you check if master or develop does work for you?

pthom commented 7 months ago

Hello,

This is probably related to https://github.com/thedmd/imgui-node-editor/issues/282 Could you test if the PR https://github.com/thedmd/imgui-node-editor/pull/285 fixes the issue?