Multi-viewports: coordinate handling when embedding imgui window in non-imgui-aware parent window

[schwaaa]

Version/Branch of Dear ImGui:

Version: v1.87 WIP Branch: docking

Back-end/Renderer/Compiler/OS

Back-ends: imgui_impl_win32.cpp + imgui_impl_glfw.cpp + imgui_impl_opengl3.cpp Compiler: vs12 Operating System: win11

My Issue/Question:

I am looking at using imgui to manage the UI of plugins that are hosted within a non-imgui-aware applications.

Each plugin process may present multiple os-level windows (hwnds, nsviews, etc) , each of which needs to be embedded in a parent os-level window provided by the hosting application.

This all works ok in multi-viewport mode, but there are mouse issues. After the platform window is parented to the host application window, the various ImGuiWindow rects are stored as client coordinates within the host platform window, but the mouse coordinates are in screen coordinates.

Note that mouse hover/capture does work properly in this scenario in non-multi-viewport mode, because the mouse coordinates provided by GLFW are not translated to screen coordinates, so everything is accounted in host platform window client coordinates. But in this mode it's not practical for the plugin to present multiple platform windows.

Screenshots/Video

In the screen capture you can see that the mouse is not processed at all until the imgui window is moved to the top left of the screen, after which it's clear the code is comparing client to screen coordinates.

Thanks!

[ocornut]

Hard to guess what may be going wrong without a simple repro or more details. When multi-viewports is enabled, all coordinates should match OS coordinates.

[schwaaa]

After the initial creation of the imgui window, this code parents the platform window to new_rec->parent, which is the non-imgui-aware host window.

  GLFWwindow *glfw_win=(GLFWwindow*)new_rec->viewport->PlatformHandle;
  HWND new_hwnd=(HWND)glfwGetWin32Window(glfw_win);
  if (glfw_win && new_hwnd) // assert
  {
    SetParent(new_hwnd, new_rec->parent);
    long style=GetWindowLong(new_hwnd, GWL_STYLE);
    style &= ~WS_POPUP;
    style |= WS_CHILDWINDOW;
    SetWindowLong(new_hwnd, GWL_STYLE, style);
  }

After which, ImGuiWindow.Pos is in the non-imgui-aware host window coordinates, not screen coordinates. As noted above, this does work properly in non-multi-viewport mode, because the mouse is not translated to screen coordinates either.

If the answer is not to embed imgui windows in non-imgui-aware windows, then imgui isn't the right tool for this job, which would be too bad, because it would be a great tool for prototyping these plugins.

[ocornut]

There’s always a solution, but you should look at the code the backend uses to pass mouse coordinates to imgui.

[ocornut]

then imgui isn't the right tool for this job,

I think there's no reason this cannot be made to work.

My point is you have access to the entire code and you can debug it and find what the issue is and find what the fix will be and then share it. I cannot magically do it for you, lacking enough context and a minimal repro. It's a bit tiring that people expect their unusual use case to get fixed for them and imply "this isn't the right tool" instead of thinking in term of "let's develop for the solution".

[schwaaa]

I meant no offense. What I was trying to say was that if this context is too far outside the expected use case, it may not be worth working on. Given your response, I'll continue to work on it.

[schwaaa]

OK, it turns out the problem is simply that glfwSetWindowPos and glfwGetWindowPos, just like Get/SetWIndowPos on win32, are not symmetrical. Set is in parent client coordinates and Get is in screen coordinates. But imgui accounts everything in screen coordinates.

It doesn't appear that glfw even has a concept of child windows, and other backends may not as well, so the only way to handle this would be via the imgui caller. We'd need to add a flag that imgui can use to translate SetNextWindowPos and SetWindowPos to screen coordinates before passing them to the backend. If this solution sounds reasonable to you, please let me know if you think it's stylistically preferable as an optional parameter to those functions, or as an imgui state variable that means "this platform window is a child."

[ocornut]

Thanks for looking. Couldn't this be done automatically, by having the backend code query the flag of a window to tell if it is a child and then patch things out correctly? Without relying on a parameter or extra manually fed info.

I also wonder if something should be reported/fixed upstream in GLFW but happy to include whichever workaround works in the meantime.

EDIT May also mean that win32/SDL could do with a similar patch, but we can handle one at a time.

[schwaaa]

ImGui_ImplWin32_SetWindowPos could query the window to see if it is a child and adjust the coordinates, just as a typical win32 api caller would.

But GLFW does not appear to have any concept of child windows. The only way to handle it in ImGui_ImplGlfw_SetWindowPos would be with an OS-dependent define that looks at the raw platform handle.

So, three choices:

1. OS-dependent behavior in the GLFW backend
2. Make the caller (who is the one who embedded the window) responsible for passing the childness flag to imgui
3. Extend GLFW itself to expose window childness

[ocornut]

OS-dependent code is fine, we already have that sort of stuff in GLFW/SDL backends to handle multi-viewport betters.

Both (1) and (3) are fine. (1) is obviously simpler and the only thing we can do short-term.

(3) can be helpful down the line, considering how distant are GLFW releases the sooner the better but it needs to makes sense cross-platform wise. Maybe for that specific problem the better solution on GLFW side may be to actually change SetWindowPos/GetWindowPos behavior or add new API. I'm not sure.

[schwaaa]

With those minor mods this is now generally functional on windows and mac. Plugins can generate multiple ImGui windows with separate viewports, and embed them in a non-ImGui-aware host application.

https://github.com/schwaaa/clap-plugin

[afxgroup]

I have a similar problem with SDL2 and OpenGL ES2. The X axis is different from some pixels. The strange thing is that in full screen everything works correctly. But the SDL2 coordinates are correct. If i print them when passed to window:

printf("SDL : %d:%d\n", sdlEvent.motion.x, sdlEvent.motion.y);

They are correct

[schwaaa]

I am now implementing this on linux and I'd like to clarify the semantics for window positioning when the imgui window is not a top-level window. Should ImGui::SetNextWindowPos() always be in screen coordinates, or relative to the parent window if there is one? I think the latter former would be more expected.

[ocornut]

All positions are absolute.