how to superimpose gr graphics on gr3 scene?

[nilsbecker]

i could not find an answer in the docs and since i also found no other support channel mentioned, i post my question as a an issue here -- please refer me to another place if it's not appropriate.

here it goes: i have a working gr3 scene which is rendered in a window controlled by pyGLFW. some relevant code snippets:

class Scene(object):
    def __init__(self,...):
         if not glfw.init():
            raise Exception("could not initialize glfw")
        glfw.window_hint(glfw.SAMPLES, 8)
        self.window = glfw.create_window(width, height, title, None, None)
        glfw.make_context_current(self.window)
        ...
    def draw(self):
        width, height = glfw.get_framebuffer_size(self.window)
        glViewport(0, 0, width, height)
        _set_gr3_camera()   # from global _camera
        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)
        gr3.drawimage(0, width, 0, height,
                      width, height,
                      gr3.GR3_Drawable.GR3_DRAWABLE_OPENGL)
        glfw.swap_buffers(self.window)
    def show(self):
        glfw.post_empty_event()
        while not glfw.window_should_close(self.window):
            self.clear()
            sleep(1./_update_rate)
            glfw.poll_events()
            for el in self.elements: el.update()
            self.draw()
            if _quit_pressed: break
        glfw.terminate()
        gr3.terminate()

now i would like to add some 2D text on top of the scene, so i tried to define an additional element like this:

class HUD(object):
    def __init__(self, csv_file, position, fontheight=0.03):
        self.x, self.y = position
        self.reader = csv.reader(csv_file)
        self.file = csv_file
        gr.setcharheight(fontheight)

    def read(self):
        self.file.seek(0)
        return list(self.reader)

    def update(self):
        s = '\n'.join([':'.join(l) for l in self.read()])
        gr.text(self.x, self.y, s)
        gr.updatews()

this was supposed to read text from a csv file and display it (an instance of HUD is added to the list elements of elements in the gr3 scene above. however this does not seem to render anything; in fact the very last line creates another window with just the text rendered.

how can i tell gr to use the same gl context / window and put text in front of all the gr3 3D scene?

[FlorianRhiem]

Hey @nilsbecker,

while GR does have a glplugin, that also creates a window instead of rendering in an existing context. Probably the easiest way to render GR content on top of your GLFW window would be to use OpenGL directly to draw a texture over the area where your GR content should be, and then update that texture with the pixmap from the memory output. If you create a numpy array as shown in the example there, you can then pass that array's content straight to glTexImage2D as GL_RGBA8 GL_UNSIGNED_BYTE data.

[nilsbecker]

Hi, thanks for the hint. I'm trying to follow it. So far, in HUD.__init__ i now allocate a numpy array as in the example. in HUD.update i will re-read the file from disk, call gr.text and then call gr.beginprint to draw to that memory array. i don't know where and how i have to call glTexImage2D to add the texture to the GL window. on every update? once in the beginning? the API here is also a bit intimidating...

[FlorianRhiem]

Yes, if you don't have experience with OpenGL this would have a very steep learning curve. I assumed you would be using OpenGL as you used GLFW to create the image. Was GLFW simply chosen as the easiest way to create a window with an OpenGL context? If so, using something like Qt might be an easier alternative for using GR and GR3. And if you do not need the user to be able to interact with the window, you don't need either, as GR3 can draw directly to GR/GKS.

[nilsbecker]

i don't have a lot of experience with opengl, but i have been using some opengl functions for managing drawing in the current project; partly by analogy to examples i found in pyglfw/mogli/gr. the user interacts with the scene by using the mouse to rotate and zoom, and a few key presses for toggling view modes. for this, glfw seemed the easiest solution. that's all done and working. the missing piece is just a non-interactive text display, for running statistics of the simulation that is rendered. (simplest case: display the current speed of the simulation in sweeps/s)

[FlorianRhiem]

Ah, okay. I will prepare an example for this.

[nilsbecker]

i found this answer: https://stackoverflow.com/questions/24262264/drawing-a-2d-texture-in-opengl/24266568 which seems to go part of the way to what i need. still, an explicit example using glfw would be much appreciated. maybe that's useful also for the example gallery on the project page?

[FlorianRhiem]

I've prepared an example that shows a minimal GR3 scene and displays a text using GR and OpenGL. There is a lot of boilerplate here for setting up OpenGL to draw the textured full screen quad. The example uses GLSL 1.2 as somewhat of a lowest common denominator, similar to what's done in GR3.

If you need help with the example, just let me know.

"""
Demo for using pyGLFW, PyOpenGL, GR and GR3 together.

The main structure is taken from the example at:
https://github.com/FlorianRhiem/pyGLFW/blob/master/README.rst
"""
import datetime
import os

import glfw
import gr
import gr3
import numpy as np
from OpenGL.GL import *

# Disable regular GKS output
os.environ['GKS_WSTYPE'] = '100'

def main():
    # Initialize the library
    if not glfw.init():
        return

    # Create a windowed mode window and its OpenGL context
    window = glfw.create_window(640, 480, "Demo", None, None)
    if not window:
        glfw.terminate()
        return

    # Make the window's context current
    glfw.make_context_current(window)

    # Initialize gr3 so it uses the context of the GLFW window
    gr3.init()

    # Prepare a minimal GR3 scene
    gr3.drawspheremesh(1, [0, 0, 0], [1, 1, 1], [1])

    # Image to be allocated at the correct size later
    gr_image = np.zeros((0, 0, 4), np.uint8)
    gr_image_pointer = gr_image.ctypes.data

    # Prepare the GR texture for later
    gr_texture = glGenTextures(1)
    glActiveTexture(GL_TEXTURE0)
    glBindTexture(GL_TEXTURE_2D, gr_texture)
    # No filtering should be necessary as the texture is create in the correct size
    glTexParameter(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST)
    glTexParameter(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST)

    # Prepare a GLSL 1.2 shader program for drawing a textured surface
    vertex_shader = glCreateShader(GL_VERTEX_SHADER)
    glShaderSource(vertex_shader, """#version 120
    attribute vec2 in_Vertex;

    varying vec2 vf_TexCoord;

    void main() {
        vf_TexCoord = vec2(0.5 + in_Vertex.x * 0.5, 0.5 - in_Vertex.y * 0.5);
        gl_Position = vec4(in_Vertex, 0.0, 1.0);
    }
    """)
    glCompileShader(vertex_shader)
    assert glGetShaderiv(vertex_shader, GL_COMPILE_STATUS)

    fragment_shader = glCreateShader(GL_FRAGMENT_SHADER)
    glShaderSource(fragment_shader, """#version 120
    uniform sampler2D u_Texture;

    varying vec2 vf_TexCoord;

    void main() {
        gl_FragColor = texture2D(u_Texture, vf_TexCoord);
    }
    """)
    glCompileShader(fragment_shader)
    assert glGetShaderiv(fragment_shader, GL_COMPILE_STATUS)

    program = glCreateProgram()
    glAttachShader(program, vertex_shader)
    glAttachShader(program, fragment_shader)
    glLinkProgram(program)
    assert glGetProgramiv(program, GL_LINK_STATUS)

    # Set the texture uniform value
    glUseProgram(program)
    uniform_location = glGetUniformLocation(program, "u_Texture")
    glUniform1i(uniform_location, 0)

    # Prepare a VBO for drawing and get the attrib location
    vbo = glGenBuffers(1)
    glBindBuffer(GL_ARRAY_BUFFER, vbo)
    glBufferData(GL_ARRAY_BUFFER, np.array([(-1, -1), (1, -1), (-1, 1), (1, 1)], np.float32), GL_STATIC_DRAW)
    attrib_location = glGetAttribLocation(program, "in_Vertex")

    # Loop until the user closes the window
    while not glfw.window_should_close(window):
        framebuffer_width, framebuffer_height = glfw.get_framebuffer_size(window)
        glViewport(0, framebuffer_width, 0, framebuffer_height)

        # Draw the GR3 scene
        gr3.drawimage(0, framebuffer_width, 0, framebuffer_height, framebuffer_width, framebuffer_height, gr3.GR3_Drawable.GR3_DRAWABLE_OPENGL)

        # Ensure the GR image will be the correct size to match the current framebuffer size
        if (framebuffer_width, framebuffer_height) != (gr_image.shape[1], gr_image.shape[0]):
            gr_image = np.zeros((framebuffer_height, framebuffer_width, 4), np.uint8)
            gr_image_pointer = gr_image.ctypes.data

        # Draw with GR into the gr_image numpy array
        gr.beginprint('!{}x{}@{:x}.mem'.format(framebuffer_width, framebuffer_height, gr_image_pointer))
        gr.settextcolorind(2)
        gr.text(0.25, 0.25, "Hello World from GR at {}".format(datetime.datetime.now().strftime("%H:%m:%S")))
        gr.endprint()

        # Update and draw the GR texture using the GR image
        glActiveTexture(GL_TEXTURE0)
        glBindTexture(GL_TEXTURE_2D, gr_texture)
        glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA8, framebuffer_width, framebuffer_height, 0, GL_RGBA, GL_UNSIGNED_BYTE, gr_image)

        # Enable blending and then draw the 4 vertices of the VBO to get a full screen quad
        glEnable(GL_BLEND)
        glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA)
        glUseProgram(program)
        glBindBuffer(GL_ARRAY_BUFFER, vbo)
        glVertexAttribPointer(attrib_location, 2, GL_FLOAT, False, 0, None)
        glEnableVertexAttribArray(attrib_location)
        glDrawArrays(GL_TRIANGLE_STRIP, 0, 4)
        glDisable(GL_BLEND)

        # Swap front and back buffers
        glfw.swap_buffers(window)

        # Poll for and process events
        glfw.poll_events()

    glfw.terminate()

if __name__ == "__main__":
    main()

[nilsbecker]

i'll work my way through it and post any questions. thanks a lot!

[nilsbecker]

ok, by following the example and a good deal of copy-paste, i got it working now! indeed the learning curve is steep. in particular if one is not used to 'modern openGL' the need for writing shader programs is surprising at first. the thing that i understood last was that vbo stores the corners of the viewport and that it is used to draw two triangles to put the texture on. the opengl api is so stateful that it's quite difficult to know what's going on at first.

there is one blemish in my current implementation: reading the small csv file with the text to be rendered within the main loop seems to take too long sometimes; then the screen redraw is held up and a white flash appears. the file is also simultaneously being written to by the simulation, so that might also be what's blocking it. is there a way i can prevent blocking the rendering while updating the contents of gr_image?

[FlorianRhiem]

Once gr.endprint has returned, the content of gr_image should be updated. The framebuffers should only be swapped (for displaying the content of the current framebuffer) when you call glfw.swap_buffers, so I don't quite understand what's causing the white screen/flash.

If the content displayed by GR doesn't need to change very frequently, you should consider updating the gr_image less than once each frame. You could, for example, perform a check whether the content changed and only update it then. As long as the texture has been written once (with glTexImage), it will keep its content until updated.

[nilsbecker]

Once gr.endprint has returned, the content of gr_image should be updated. The framebuffers should only be swapped (for displaying the content of the current framebuffer) when you call glfw.swap_buffers, so I don't quite understand what's causing the white screen/flash.

yes, i call glTexImage2D directly after endprint. i also don't quite understand. but the flashes disappear when i give a constant string to gr.text, so i assume it has to do with delays in reading. i tried using asynchronous file i/o by adapting this answer but that did not seem to change anything.

[FlorianRhiem]

A few ways of speeding up the drawing and data transfer, would be to:

• only draw the area that you need to draw, instead of the full window, e.g. if there is only text in one corner of the window you don't need a large texture for the whole screen, and
• only re-create the gr_image and update the texture when the text changes, or
• only update the text every nth-frame or every second or something like that

[nilsbecker]

yes, thanks for the hints. i tried your third suggestion: only every 10th frame, glTexImage2D is called. this had an interesting effect: the text does get updated more slowly, about twice per second, but every now and then i get a white screen for half a second. this would indicate that gr_image is somehow uninitialized sometime when glTexImage2D is called. maybe i am slowing down the update at the wrong point and should reduce the frequency of gr.text?

[FlorianRhiem]

Can you produce a minimal working example of these issues, perhaps based on the example I provided? A white screen sounds like the gr_image contains only white, but I don't quite see how that would be produced.

[nilsbecker]

i have a version now that seems to be working without white flashes. i do both, gr.text and glTexImage2D only every 10th frame. i paste a relevant snippet (not a MWE, sorry)

def draw(self):
        width, height = glfw.get_framebuffer_size(self.window)
        glViewport(0, 0, width, height) 
        # first the 3D scene
        _set_gr3_camera()   # from global _camera
        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)
        gr3.drawimage(0, width, 0, height,
                      width, height,
                      gr3.GR3_Drawable.GR3_DRAWABLE_OPENGL)
        # now the HUD
        if self.hud:
            self.ticker += 1
            if not (height, width) == self.text_mem.shape:
                # reset if window resized. height = outer dimension!
                self.text_mem = np.zeros((height, width, 4), dtype=np.uint8)
            x, y = self.hud.position
            # write text to texel array
            if not (self.ticker % 10):
                self.ticker = 0
                gr.beginprint('!{}x{}@{:x}.mem'.format(width, height, self.text_mem.ctypes.data))
                gr.text(x, y, self.hud.s)
                gr.endprint()
                glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA8, width, height, 0,
                    GL_RGBA, GL_UNSIGNED_BYTE, self.text_mem)
            # glActiveTexture(GL_TEXTURE0) # we do this in initialization already
            # glBindTexture(GL_TEXTURE_2D, self.gr_texture)  # dito
            # now draw, blending together 3D and 2D
            glEnable(GL_BLEND)
            glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA)
            glUseProgram(self.program)
            glBindBuffer(GL_ARRAY_BUFFER, self.vbo)
            glVertexAttribPointer(self.attrib_location, 2, GL_FLOAT, False, 0, None)
            glEnableVertexAttribArray(self.attrib_location)
            glDrawArrays(GL_TRIANGLE_STRIP, 0, 4) # Z path = 4 vertices -> 2 triangles
            glDisable(GL_BLEND)
            # update texture
        # execute drawing
        glfw.swap_buffers(self.window)

for me this is good enough, so unless you need it i would not try to produce another example. i checked that when i remove the if self.ticker check, the flashes reappear, though.

[FlorianRhiem]

If it works, that's good enough for me :) Can this issue be closed then?

[nilsbecker]

yes. a nitpick: the FONT_COURIER is not rendered as a monospace font, which makes updated digits shift around. it would be nice if GR had one or more monospaced fonts (in addition to courier maybe some standard console font?) would it make sense to make a feature request for that?

[FlorianRhiem]

I think that was fixed very recently, in commit 84bf0c58f14e46f27d4323c67ba4c237b0068b2d. You can download the latest development build of GR from the website or from the downloads directory if you want to use the development version where this is fixed.

[nilsbecker]

i think i found the issue. sometimes indeed file reading was delayed/blocked/confused, possibly by simultaneous access to the same file from elsehwere. this resulted in an empty string being returned and written to self.hud.s above. it turns out gr.text does not like an empty string input and this results in a white screen. (is this a bug?)

similarly, when setting self.hud.s = " ", i get warning messages "GKS: invalid bitmap size" but this time, the screen updates ok.

i can fully eliminate flickering by skipping gr.text when self.hud.s is empty.

[FlorianRhiem]

Ah, I think I know what's happening, then. Without any content at all, GKS (the system below GR) will usually produce a blank (white, not transparent) page. In the case of memory output the cairoplugin usually uses the cairo graphics libary, but when the page is empty what happens instead boils down to this line from the cairoplugin.c:

memset(mem, 255, height * width * 4);

When you pass a non-empty string, the page is not considered empty and instead that string is rendered, though a string containing no printable characters (like " ") will produce a a 0x0 bitmap (as it has no content) and a warning is printed as a result of that, though it shouldn't have any negative side effects.

Skipping the GR part when there is no content avoids those issues and should lead to a better performance as well, as the texture drawing will be avoided, too