How to place an successfully rendered opengl scene at background into an imgui window?

[sirlis]

Version/Branch of Dear ImGui:

Version: Latest Branch: docking

Back-end/Renderer/Compiler/OS

Back-ends: imgui_impl_glfw.cpp + imgui_impl_opengl3.cpp Compiler: gcc-13.1.0 Operating System: windows

My Issue/Question:

I followed the https://learnopengl.com/ step by step to produce a 3D scene. Now I want to place it into a imgui window.

After some code work I managed to arrange the window but I don't know how to put the background rendered scene into the imgui window (e.g. the Render Panel )

I read FAQs and get to know that I can render the scene into a texture and use the ImGui::Image() or ImGui::GetWindowDrawList()->AddImage() function to draw it onto the window. But I am confused about how to achieve render to texture according to my existing codes, especially that my model, shader, etc., is constructed in classes.

What is actually rendered:

What I expected to see (I fake it, want the scene rendered fullscreen in the render panel):

Related Codes

I write a Sphere class like this:

class Sphere
{
private:
    vector<Vertex> vertices;
    std::vector<int> indices;
    GLuint VBO, VAO, EBO;
    float radius = 1.0f;
    int uCount = 50;
    int vCount = 50;

public:
    ~Sphere()
    {
        glDeleteVertexArrays(1, &VAO);
        glDeleteBuffers(1, &VBO);
        glDeleteBuffers(1, &EBO);
    }

    Sphere(float r, int sectors, int stacks)
    {
        Vertex vertex;
        int i = 0, j = 0;
        unsigned short usNo = 0;
        radius = r;
        uCount = sectors;
        vCount = stacks;

        /* GENERATE VERTEX ARRAY */
        float x, y, z;    // vertex position
        float tx, ty;     // vertex texture coordinate
        float nx, ny, nz; // vertex normal vector
        float xy;         // vertex position on xy plane

        float uStep = (float)(2 * M_PI / uCount);
        float vStep = (float)(M_PI / vCount);
        float uAngle, vAngle;

        for (i = 0; i <= vCount; ++i)
        {
        //...
        }
        /* GENERATE VERTEX ARRAY */

        /* GENERATE INDEX ARRAY */
        int k1, k2;
        for (i = 0; i < vCount; ++i)
        {
        //...
        }
        /* GENERATE INDEX ARRAY */

        /* GENERATE VAO-EBO */
        glGenVertexArrays(1, &VAO);
        glGenBuffers(1, &VBO);
        glGenBuffers(1, &EBO);
        // Bind the Vertex Array Object first, then bind and set vertex buffer(tx) and attribute pointer(tx).
        glBindVertexArray(VAO);

        glBindBuffer(GL_ARRAY_BUFFER, VBO);
        glBufferData(GL_ARRAY_BUFFER, (unsigned int)vertices.size() * sizeof(Vertex), &vertices[0], GL_DYNAMIC_DRAW);

        glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, EBO);
        glBufferData(GL_ELEMENT_ARRAY_BUFFER, (unsigned int)indices.size() * sizeof(unsigned int), indices.data(), GL_DYNAMIC_DRAW);

        // set the vertex attribute pointers
        // vertex Positions
        glEnableVertexAttribArray(0);   
        glVertexAttribPointer(0, 3, GL_FLOAT, GL_FALSE, sizeof(Vertex), (void*)0);
        // vertex normals
        glEnableVertexAttribArray(1);   
        glVertexAttribPointer(1, 3, GL_FLOAT, GL_FALSE, sizeof(Vertex), (void*)offsetof(Vertex, Normal));
        // vertex texture coords
        glEnableVertexAttribArray(2);   
        glVertexAttribPointer(2, 2, GL_FLOAT, GL_FALSE, sizeof(Vertex), (void*)offsetof(Vertex, TexCoords));
        // vertex tangent
        glEnableVertexAttribArray(3);
        glVertexAttribPointer(3, 3, GL_FLOAT, GL_FALSE, sizeof(Vertex), (void*)offsetof(Vertex, Tangent));
        // vertex bitangent
        glEnableVertexAttribArray(4);
        glVertexAttribPointer(4, 3, GL_FLOAT, GL_FALSE, sizeof(Vertex), (void*)offsetof(Vertex, Bitangent));

        glBindBuffer(GL_ARRAY_BUFFER, 0);
        glBindVertexArray(0);
        /* GENERATE VAO-EBO */
    }
    void Draw()
    {
        glBindVertexArray(VAO);
        glDrawElements(GL_TRIANGLES,
                       (unsigned int)indices.size(),
                       GL_UNSIGNED_INT,
                       (void *)0);
        glBindVertexArray(0);
    }
};

Then I wrote a Shader class like this:

class Shader
{
public:
    unsigned int ID;
    // constructor generates the shader on the fly
    // ------------------------------------------------------------------------
    Shader(void)
    {
    }
    Shader(const char *vertexPath, const char *fragmentPath)
    {
        // 1. retrieve the vertex/fragment source code from filePath
        std::string vertexCode;
        std::string fragmentCode;
        std::ifstream vShaderFile;
        std::ifstream fShaderFile;
        // ensure ifstream objects can throw exceptions:
        vShaderFile.exceptions(std::ifstream::failbit | std::ifstream::badbit);
        fShaderFile.exceptions(std::ifstream::failbit | std::ifstream::badbit);
        try
        {
            // open files
            vShaderFile.open(vertexPath);
            fShaderFile.open(fragmentPath);
            std::stringstream vShaderStream, fShaderStream;
            // read file's buffer contents into streams
            vShaderStream << vShaderFile.rdbuf();
            fShaderStream << fShaderFile.rdbuf();
            // close file handlers
            vShaderFile.close();
            fShaderFile.close();
            // convert stream into string
            vertexCode = vShaderStream.str();
            fragmentCode = fShaderStream.str();
        }
        catch (std::ifstream::failure &e)
        {
            std::cout << "ERROR::SHADER::FILE_NOT_SUCCESFULLY_READ: " << e.what() << std::endl;
        }
        const char *vShaderCode = vertexCode.c_str();
        const char *fShaderCode = fragmentCode.c_str();
        // 2. compile shaders
        unsigned int vertex, fragment;
        // vertex shader
        vertex = glCreateShader(GL_VERTEX_SHADER);
        glShaderSource(vertex, 1, &vShaderCode, NULL);
        glCompileShader(vertex);
        checkCompileErrors(vertex, "VERTEX");
        // fragment Shader
        fragment = glCreateShader(GL_FRAGMENT_SHADER);
        glShaderSource(fragment, 1, &fShaderCode, NULL);
        glCompileShader(fragment);
        checkCompileErrors(fragment, "FRAGMENT");
        // shader Program
        ID = glCreateProgram();
        glAttachShader(ID, vertex);
        glAttachShader(ID, fragment);
        glLinkProgram(ID);
        checkCompileErrors(ID, "PROGRAM");
        // delete the shaders as they're linked into our program now and no longer necessery
        glDeleteShader(vertex);
        glDeleteShader(fragment);
    }

    // activate the shader
    // ------------------------------------------------------------------------
    void use() const
    {
        glUseProgram(ID);
    }
//......

After that , at initialization:

//...
GLFWwindow *MainWindow = glfwCreateWindow(SCREEN_WIDTH, SCREEN_HEIGHT, "DSScene_OpenGL", nullptr, nullptr);
glfwMakeContextCurrent(MainWindow);
// 2 frames between swap, to avoid satellite blinking
// caused by not synchronously updated camera position and satellite position
// that makes the satellite rendered in and out of the camera FOV frequently
glfwSwapInterval(2);
// callback
glfwSetErrorCallback(error_callback);
glfwSetKeyCallback(MainWindow, key_callback);
glfwSetFramebufferSizeCallback(MainWindow, framebuffer_size_callback);
glfwSetCursorPosCallback(MainWindow, mouse_callback);
glfwSetScrollCallback(MainWindow, scroll_callback);
glfwSetMouseButtonCallback(MainWindow, mouse_button_callback);

Sphere Earth(EarthRadius, 20, 20);
Shader EarthShader("res/shader/Earth.vs", "res/shader/Earth.fs");
//...

and the code in the main render loop is like:

while (!glfwWindowShouldClose(MainWindow))
{
    //......
    ImGui::Begin("Render Panel");

    //......
    camera.Focus(EarthPos);
    glm::dmat4 view(1.0);
    glm::dmat4 projection(1.0);
    view = camera.GetViewMatrix();
    projection = glm::perspective((GLdouble)glm::radians(camera.Fov),
                                  (GLdouble)SCREEN_WIDTH_CURRENT / (GLdouble)SCREEN_HEIGHT_CURRENT,
                                  NearPlane,
                                  FarPlane);
    glm::dmat4 modelview(1.0);
    EarthShader.use();
    EarthShader.setVec3("lightPosition", SunLightPos);
    EarthShader.setVec3("lightColor", SunLightColor);
    EarthShader.setFloat("farPlane", FarPlane);
    EarthShader.setVec3("viewPos", camera.Position);
    glActiveTexture(GL_TEXTURE0);
    glBindTexture(GL_TEXTURE_2D, texture_earth);
    EarthShader.setInt("texture_day", 0);
    glActiveTexture(GL_TEXTURE1);
    glBindTexture(GL_TEXTURE_2D, texture_earth_night);
    EarthShader.setInt("texture_night", 1);
    glActiveTexture(GL_TEXTURE2);
    glBindTexture(GL_TEXTURE_2D, texture_earth_cloud);
    EarthShader.setInt("texture_cloud", 2);
    glActiveTexture(GL_TEXTURE3);
    glBindTexture(GL_TEXTURE_2D, texture_earth_watermask);
    EarthShader.setInt("texture_watermask", 3);
    glActiveTexture(GL_TEXTURE4);
    glBindTexture(GL_TEXTURE_2D, texture_earth_normal);
    EarthShader.setInt("texture_height", 4);
    EarthShader.setVec3("material.ambient", 0.2f, 0.2f, 0.2f);
    EarthShader.setVec3("material.diffuse", 1.0, 1.0, 1.0);
    EarthShader.setVec3("material.specular", 0.25, 0.25, 0.25);
    EarthShader.setFloat("material.shininess", 32.0);
    EarthShader.setVec3("light.ambient", 1.0, 1.0, 1.0);
    EarthShader.setVec3("light.diffuse", 1.0, 1.0, 1.0);
    EarthShader.setVec3("light.specular", 1.0, 1.0, 1.0);
    glm::dmat4 model_earth(1.0);
    model_earth = glm::translate(model_earth, EarthPos);
    model_earth = glm::rotate(model_earth, EarthRotAngle, glm::dvec3(0.0, 0.0, 1.0)); // self rotation
    EarthShader.setMat4("model", model_earth);
    EarthShader.setMat4("view", view);
    EarthShader.setMat4("projection", projection);
    EarthShader.setFloat("farPlane", FarPlane);
    Earth.Draw();
    //......

    ImGui::End()
    ImGui::Render()
    //....
}

And finally, the framebuffer size callback :

// glfw: whenever the window resizes, this callback is called
void framebuffer_size_callback(GLFWwindow *window, int width, int height)
{
    if (height == 0)
        height = 1;
    SCREEN_WIDTH_CURRENT = width;
    SCREEN_HEIGHT_CURRENT = height;
    glViewport(0, 0, width, height);
}

Standalone, minimal, complete and verifiable example: (see https://github.com/ocornut/imgui/issues/2261)

Sorry that the project is too big that I don't know how to provide a minimal example. I think the above code snipps may be enough.

[sakiodre]

You can render the scene into a texture and use the ImGui::Image() or ImGui::GetWindowDrawList()->AddImage() function to draw it onto the window

[sirlis]

You can render the scene into a texture and use the ImGui::Image() or ImGui::GetWindowDrawList()->AddImage() function to draw it onto the window

Thanks for the suggestion, that is exactly the part I don't know how to achieve. I read the FAQs and get to know the two functions, but I don't know how to transform my existing rendered scene into a texture and called by ImGui::GetWindowDrawList()->AddImage(). That's why I provide so many code snipps, thinking someone may show me the way.

[sakiodre]

I'm not familiar with OpenGL but I found this, along with plenty of other examples: https://gamedev.stackexchange.com/questions/140693/how-can-i-render-an-opengl-scene-into-an-imgui-window

Have you tried that? What particular part are you having trouble with?

[sirlis]

I'm not familiar with OpenGL but I found this, along with plenty of other examples: https://gamedev.stackexchange.com/questions/140693/how-can-i-render-an-opengl-scene-into-an-imgui-window

Have you tried that? What particular part are you having trouble with?

Thanks! I will try it immediately. I wonder if I can leave u a message if any further progress is made.

[sakiodre]

Sure, I'm open to learning OpenGL as well so leave your discord or telegram and I'll send a message

[sirlis]

Sure, I'm open to learning OpenGL as well so leave your discord or telegram and I'll send a message

Thank you. I am working on downloading and using discord (plz search for : lihongjue as my username, and I don't know if it is since I am new to discord). Currently I succesfully put my opengl rendered scene into the correct panel.

I used:

ImGui::Begin("GameWindow");
// Using a Child allow to fill all the space of the window.
// It also alows customization
if (ImGui::BeginChild("GameRender"))
    // Get the size of the child (i.e. the whole draw size of the windows).
    ImVec2 wsize = ImGui::GetWindowSize();
    // Because I use the texture from OpenGL, I need to invert the V from the UV.
    ImGui::Image((void*)(intptr_t)tex, wsize, ImVec2(0, 1), ImVec2(1, 0)); //<---warning here
    ImGui::EndChild();
}
ImGui::End();

In addition, there are still some bugs: 1) the earth is not rendered at the center of the scene, which is should be. And there is deformation, and blurry (UNSOLVED, I raised another issue #6894 about this); 2) mouse drag/keybord callbacks only linstens to the old main window instead of the current imgui panel window (solved).

The second bug is solved by disable the callbacks (maybe disable only the framebuffersizecallback will work):

    glfwSetErrorCallback(error_callback);
    glfwSetKeyCallback(MainWindow, key_callback);
    glfwSetFramebufferSizeCallback(MainWindow, framebuffer_size_callback);
    glfwSetCursorPosCallback(MainWindow, mouse_callback);
    glfwSetScrollCallback(MainWindow, scroll_callback);
    glfwSetMouseButtonCallback(MainWindow, mouse_button_callback);

[ocornut]

Please note that the part about rendering your scene into a texture is not really an Dear ImGui question but nevertheless is covered in our wiki for many graphics librairies: https://github.com/ocornut/imgui/wiki/Image-Loading-and-Displaying-Examples

[sirlis]

Please note that the part about rendering your scene into a texture is not really an Dear ImGui question but nevertheless is covered in our wiki for many graphics librairies: https://github.com/ocornut/imgui/wiki/Image-Loading-and-Displaying-Examples

Thank you. There is one issue, when I draw a texture into a imgui window, and when the window is floating, double click it's title bar will cause error, like the last image in the above response.

Assertion failed: (g.CurrentWindowStack.Size == 1) && "Mismatched Begin/BeginChild vs End/EndChild calls: did you forget to call End/EndChild?"

However, I checked and confirmed that I had matched Begin/BeginChild in the following code:

if (ImGui::Begin("Render Panel"))
{
    ImVec2 pos = ImGui::GetCursorScreenPos();

    // Using a Child allow to fill all the space of the window.
    // It also alows customization
    ImGui::BeginChild("GameRender");
    // Get the size of the child (i.e. the whole draw size of the windows).
    ImVec2 wsize = ImGui::GetWindowSize();
    SCREEN_WIDTH_CURRENT = wsize.x;
    SCREEN_HEIGHT_CURRENT = wsize.y;
    // Because I use the texture from OpenGL, I need to invert the V from the UV.
    // glViewport(0, 0, wsize.x, wsize.y);
    ImGui::Image((ImTextureID)FBO_textureColorBuffer, wsize, ImVec2(0, 1), ImVec2(1, 0));
    ImGui::EndChild();
    ImGui::End();
}

---

Update

after test, if initialized with e.g. ImGui::Begin("Render Panel", &window_renderpanel_visible, 0); will have no problem.

if initialized with e.g. ImGui::Begin("Render Panel"); will have program crash problem.

[ocornut]

You ImGui::End() call is misplaced.

[sirlis]

You ImGui::End() call is misplaced.

Got it. Thank you !

[Seneral]

I used two other methods, and the one I ended up using is, in my opinion, superior to the officially suggested one, with the exception that you can't easily decouple the game and UI framerate and resolution as easily as with the image/FrameBuffer method. For full detail, see: https://gamedev.stackexchange.com/a/207560/174820

Essentially, you use a callback to render the code at exactly the right point in the UI (in your window code, before the UI that's supposed to overlay it, and it all works fine. The advantage over the image method is that you don't render parts that might be outside the window, and there's less hassle and overhead, so multiple simple viewports are much more feasible.

Here's some example code using Eigen (superfluous) with glfw, for OpenGL, but it should work similarly for any setup:

// Needs to be accessible in the callback, but specific to this callback
static ImVec2 renderSize;
static float renderScale;
static ImRect viewRect;
auto viewWin = ImGui::GetCurrentWindowRead();
renderScale = ImGui::GetIO().DisplayFramebufferScale.x;
renderSize = viewWin->Viewport->Size * renderScale;
viewRect = viewWin->InnerRect;

ImDrawList* draw_list = ImGui::GetWindowDrawList();
draw_list->AddCallback([&](const ImDrawList* dl, const ImDrawCmd* dc)
{
    auto view = viewRect.ToVec4() * renderScale;
    auto clip = dc->ClipRect * renderScale;
    glViewport(view.x, renderSize.y-view.w, view.z-view.x, view.w-view.y);
    glScissor(clip.x, renderSize.y-clip.w, clip.z-clip.x, clip.w-clip.y);
    glClearColor(0.2f, 0.0, 0.2f, 0.0);
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

    // Enable Depth for 3D scene
    glDepthMask(GL_TRUE);
    glClearDepth(0);
    glDepthFunc(GL_GEQUAL);
    glEnable(GL_DEPTH_TEST);

    // TODO: Render 3D scene
}, nullptr);
draw_list->AddCallback(ImDrawCallback_ResetRenderState, nullptr);

I'd recommend to put something like this in the official docs, or at least a reference to it as a better approach in the section here. Yes it's out of scope, but a common user case that I haven't seen solved this well before.

[Seneral]

See my reply here for my current solution that I'll be using in my code from now on.

[gkoreman]

Thanks @Seneral! This is exactly what I was looking for. The key part I was missing was including imgui_internal.h for the GetCurrentWindowRead() function.

I'm doing something slightly different to get the framebuffer height. I get the ImGui::GetWindowViewport() and send it to my callback. I can then get the DrawData and use DrawData.DisplaySize.y * DrawData.FramebufferScale.y. I also use the DrawData.FramebufferScale instead of pixelRatio.

I imagine using the DrawData instead of glfwGetFramebufferSize() may be more resilient to using multiple viewports, although I haven't got that far yet.

[Seneral]

Thanks for the improvements @gkoreman - my code had many errors due to hastily extracting it from my project, sorry about that. I'll edit it. I personally don't use multiple viewports feature because wayland (linux compositor) doesn't support such an use case yet and that's what I use lol

One warning though, DrawData is NULL for the first frame, so better use ImGui::GetWindowViewport().Size and one of ImGui::GetIO().DisplayFramebufferScale, ImGui::GetWindowDpiScale() or ImGui::GetWindowViewport().DpiScale - I don't know if DPI scale is the same

Edit 2: But be careful, current window/viewport WILL be NULL in the draw function, since it's not executed between a BeginFrame and Render!

[Seneral]

Ok, updated the code and verified that it works just the same, at least on X11 and Wayland. Though scaling only worked properly on wayland, when in theory it should also work on X11, but I guess something is not implemented there. No matter, as long as wayland works:)

[Seneral]

The code is now available here: https://gist.github.com/Seneral/b4b34a283539938869cd10b2d065a88c And it works quite nicely to integrate OpenGL views or smaller renders like icons into ImGui. You can also optionally apply the patch and then you can render only those views without updating and rendering the whole UI.