Native windows event handling

[sadiuk]

Description

This issue covers native window event handling support on several operating systems, including Windows, Linux(X11, Wayland) and Android. It also contains suggestions for possible lightweight cross-platform libraries which can be used in Nabl to simplify this process.

Native Implementations

Windows

Window event handling on Windows works in a very straightforward manner: you define 2 things: 1) An event loop - the loop which catches all the messages the application should receive? translates them and calls the window procedure. 2) A window procedure - the callback which will be called by the event loop with the proper event type and handle it.

So the actual code that will allow you to handle events looks like this:

#include "Windows.h"

Define a window procedure:

LRESULT stdcall MainWindowProc(HWND hwnd, uint uMsg, WPARAM wParam, LPARAM lParam)
{
    switch(uMsg)
    {
        case WM_CLOSE: // User closes the window
            PostQuitMessage(0);
            break;
        case WM_SIZING: // User resizes the window
            //Handle resizing
            break;
    }
    return DefWindowProc(hwnd, uMsg, wParam, lParam);
}

And an event loop in the main function:

int __stdcall WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR lpCmdLine, int nShowCmd)
{
    HWND hwnd;
    // Create window
    // ...
    // ...
    ShowWindow(hwnd, SW_SHOW);
    UpdateWindow(hwnd);

    MSG msg; // the message we'll receive

    while(GetMessage(&msg, nullptr, 0, 0))
    {
        TraslateMessage(&msg);
        DispatchMessage(&msg);
    }
}

Linux

X11

The X11 event handling looks pretty similar to how it works on Windows. However, here you do not define a callback(like the MainWindowProc), but do all the dirty work iside the main function.

#include <X11/Xlib.h>
#include <X11/Xutil.h>
#include <X11/Xos.h>

First you define stuff like window, display etc for the X11 window application to work.

Display *dis = XOpenDisplay((char *)0);;

unsigned long black, white;
black=BlackPixel(dis,screen);
white=WhitePixel(dis, screen); 

Window win = XCreateSimpleWindow(dis,DefaultRootWindow(dis),0,0, 200, 300, 5, white, black);

Then you need an event loop to wait for events and handle them. Since, unlike on Windows, the event types are not as specific (e.g. ConfigureNotify event means that the window could be either moved, or resized, or the border width was changed or all 3 at once), you either need to store the previous window state, or update all possible properties: Option 1:

XEvent event;
while(1)
{
    XNextEvent(dis, &event);
    //Since 
    if(event.type == ConfigureNotify)
    {
        XResizeWindow(dis, win, event.width, event.height);
        XMoveWindow(dis, win, event.x, evennt.y);
        //... All possible event handlings
        //... 
        //...
    }
}

Option 2:

XEvent event;
while(1)
{
    XNextEvent(dis, &event);
    //Since 
    if(event.type == ConfigureNotify && event.width != prevWidth || event.height != prevHeight)
    {
        // Update prevWidth, prevHeight
        XResizeWindow(dis, win, event.width, event.height);
        //...
        //...
    }
}

Wayland

With wayland the situation is not as pleasant as with previous systems. Here you do not get nice events that can tell you what exactly happened to your window, but define listeners for simple events like mouse pointer moving, clicking, entering/leaving the window surface, etc. All the other stuff should be handled manually.

Let's go through main ideas behing window event handling. Before we connect all the stuff in main, lets create a listener for a pointer:

// Mouse enter the window surface
static void pointer_handle_enter(void *data, struct wl_pointer *pointer,
                     uint32_t serial, struct wl_surface *surface,
                     wl_fixed_t sx, wl_fixed_t sy)
{
    fprintf(stderr, "Pointer entered surface %p at %d %d\n", surface, sx, sy);
}

// Mouse leave the window surface
static void pointer_handle_leave(void *data, struct wl_pointer *pointer,
                     uint32_t serial, struct wl_surface *surface)
{
    fprintf(stderr, "Pointer left surface %p\n", surface);
}

// Mouse move
static void pointer_handle_motion(void *data, struct wl_pointer *pointer,
                      uint32_t time, wl_fixed_t sx, wl_fixed_t sy)
{
    printf("Pointer moved at %d %d\n", sx, sy);
}

// Mouse click
static void pointer_handle_button(void *data, struct wl_pointer *wl_pointer,
                      uint32_t serial, uint32_t time, uint32_t button,
                      uint32_t state)
{
    printf("Pointer button click\n");

    // !!!! This is where you can perform window resizing, moving and stuff.

    if (button == BTN_LEFT && state == WL_POINTER_BUTTON_STATE_PRESSED)
    wl_shell_surface_move(shell_surface,
                  seat, serial);

}

// This one seems to be a one-axis gesture like the left-to-right swipe
static void pointer_handle_axis(void *data, struct wl_pointer *wl_pointer,
                    uint32_t time, uint32_t axis, wl_fixed_t value)
{
        printf("Pointer handle axis\n");
}

static const wl_pointer_listener pointer_listener = {
    pointer_handle_enter,
    pointer_handle_leave,
    pointer_handle_motion,
    pointer_handle_button,
    pointer_handle_axis,
};

Then there goes a LOT of code that registers all those listeners, and then we bind all the stuff in the main: Note: for simplicity i removed a lot of code and error handling so this code will not actually work.

int main()
{
    wl_display* display = wl_display_connect(NULL);

    // Some other stuff, unrelated to this issue....

    wl_surface* surface = wl_compositor_create_surface(compositor);

    // In Wayland there should be a shell surface on top of the actuall surface that catches pointer events.
    wl_shell_surface shell_surface = wl_shell_get_shell_surface(shell, surface);
    wl_shell_surface_set_toplevel(shell_surface);

    wl_shell_surface_add_listener(shell_surface,
                  &shell_surface_listener, NULL);

    init_egl();
    create_window();

    // The event loop itself
    while (wl_display_dispatch(display) != -1) {
    ;
    }

    wl_display_disconnect(display);
}

A complete tutorial on wayland events is here A simple program program that tracks when a window is moved can be found here.

Android

Android provides C++ programming capabilities via the JNI (Java native interface) - basically a set of java-like functions in C++. Window (and other kinds of) events are processesed by defining two event handlers in the entry point and registering them:

#include <EGL/egl.h>
#include <GLES/gl.h>

#include <android/sensor.h>
#include <android/log.h>
#include <android_native_app_glue>

void android_main(struct android_app* state) {
    engine engine;

    memset(&engine, 0, sizeof(engine));
    state->userData = &engine;
    state->onAppCmd = applicationCommandHandler;
    state->onInputEvent = inputHandler;
    engine.app = state;
}

Where the applicationCommandFunc is the callback which processes events like window initialization, window termination etc and the inputHandler is the function which recieves and processes input events like key presses. Here I've posted possible implementations of those two:

void applicationCommandHandler(android_app *app, int32_t cmd)
{
    switch (cmd)
    {
    case APP_CMD_INIT_WINDOW:
        initialize();
        break;
    case APP_CMD_TERM_WINDOW:
        finalize();
        break;
    default:
        break;
    }
}

int32_t inputHandler(android_app *app, AInputEvent *inputEvent)
{
    if (AInputEvent_getType(inputEvent) == AINPUT_EVENT_TYPE_KEY &&
        AKeyEvent_getKeyCode(inputEvent) == AKEYCODE_BACK)
    {
        ANativeActivity_finish(app->activity);
        return 1;
    }

    if (AInputEvent_getType(inputEvent) == AINPUT_EVENT_TYPE_MOTION)
    {
        // TODO: Handle touch event...
        return 1;
    }

    return 0;
}

[devshgraphicsprogramming]

Ok so on Windows we set up a callback, and on X11 we poll.

Does the XNextEvent have a version that sleeps our thread until the next event? (this is often an issue with implementing efficient poll-based systems)

Also it seems like X11 doesn't actually molest our window (window is not resized behind out back, we get to schedule it ourselves) whereas on Windows this seems to notify us of the event already taking place?

@Crisspl its an important distinction because under Vulkan a resize would invalidate a surface&swapchain I think?

[sadiuk]

Does the XNextEvent have a version that sleeps our thread until the next event? (this is often an issue with implementing efficient poll-based systems)

Yes, XNextEvent does block a thread in case the event queue is empty.

Also it seems like X11 doesn't actually molest our window (window is not resized behind out back, we get to schedule it ourselves) whereas on Windows this seems to notify us of the event already taking place?

No, not really. In both Windows and X11 you should handle them manually, the difference is that Windows has a DefWindowProc function which handles the events you didn't handle, or only handled partially. So basically you can just use DefWindowProc for event handling and have additional event processing done in switch cases:

LRESULT stdcall MainWindowProc(HWND hwnd, uint uMsg, WPARAM wParam, LPARAM lParam)
{
    switch(uMsg)
    {
        case WM_CLOSE: // User closes the window
            m_state = CLOSED;
                        LOGI("Window closed");
            break;
    }
    return DefWindowProc(hwnd, uMsg, wParam, lParam); // Here the WM_CLOSE will be handled on OS-level
}

[devshgraphicsprogramming]

No, not really. In both Windows and X11 you should handle them manually, the difference is that Windows has a DefWindowProc function which handles the events you didn't handle, or only handled partially. So basically you can just use DefWindowProc for event handling and have additional event processing done in switch cases:

Ok now go and read the Vulkan spec (with extensions) and talk to people knowledgeable with Vulkan about how to handle changes to windows (AFAIK resizing a window will invalidate a swapchain, a rotation will reduce performance on android).

I basically need to know of the best way to handle this.

[sadiuk]

@Devsh-Graphics-Programming/seniors updated

[sadiuk]

Ok, so i found some issues along with the ways to handle them: 1) Window resizing. Vulkan does provide an interface to check whether the swapchain is still valid for use by calling vkAcquireNextImageKHR, which returns either VK_ERROR_OUT_OF_DATE_KHR when the swapchain is not usable any more or VK_SUBOPTIMAL_KHR when it still can be used for presenting, but the surface properties are likely not gonna match correctly. These two options depend on the platform. So the code for handling that might look like this:

VkResult result = vkAcquireNextImageKHR(device, swapChain, UINT64_MAX, imageAvailableSemaphores[currentFrame], VK_NULL_HANDLE, &imageIndex);

if (result == VK_ERROR_OUT_OF_DATE_KHR) {
   waitForResizeEvent();
   recreateSwapChain();
}

The thing is that on wayland you don't get either of those errors, so you should recreate swapchain even if you didn't get any error, but received a window resize event. As far as i have seen, the swapchain recreation should not be a big performance issue.

2) Window minimization. Well, we don't need to render anything when the window is minimized, so if the window is minimized, we wait untill it is in the foreground and recreate the swapchain.

3) Android orientation change. When the device orientation is changed, the application must react to this by rotating the output image. According to Vulkan Best Practices For Mobile Developers, there are 3 ways of handling the rotation: 1) Rotation by the Display Processing Unit. This one is the most optimal, but not all devices are currently supporting it. 2) Rotation by the compositor. This one is the slowest and has additional system-level costs such as extra memory bandwidth, or even GPU processing if the compositor uses the GPU as the rotation engine. 3) Rotation by the application itself by rendering into a window surface which is oriented to match the physical orientation of the display panel (pre-rotation).

The first one is a perfect option, but looks like there is a risk that the device will not support it. The third one is probably the one to go with. It contains 5 main steps:

• Destroy the Vulkan framebuffers and the swapchain.
• Re-create the swapchain using the old swapchain dimensions, i.e. the swapchain dimensions do not change. Update the preTransform field in VK_STRUCTURE_TYPE_SWAPCHAIN_CREATE_INFO_KHR so that it matches the currentTransform field of the VkSurfaceCapabilitiesKHR returned by the new surface. This communicates to Android that it does not need to rotate the scene.
• Re-create the framebuffers
• Adjust the MVP matrix.
• Render the scene.

[Crisspl]

On Windows event receiving thread and window creation thread must be the same thread. So we're doing dedicated thread for all windows that will:

• run event gathering loop (turns out callback is not enough, you need an actual loop on Windows)
• dispatches requests from different threads (the only request type is window creation, and maybe window destruction)

Integration with external frameworks will be done with pushing virtual events into our input event channels.

[Crisspl]

X11: https://stackoverflow.com/questions/6402476/multithreaded-x11-application-and-opengl We need to make sure, our Xlib calls will not interact in a weird way with Xlib calls done by our EGL implementation. A possible solution might be to use XLockDisplay/XUnlockDisplay instead of mutexes in EGL impl. Then display locking is global, and not just EGL-local. Downside of this is a must to call XInitThreads which must be very first Xlib call and we cant say what will happen first: creation of a window or initialization of EGL.

[devshgraphicsprogramming]

Downside of this is a must to call XInitThreads which must be very first Xlib call and we cant say what will happen first: creation of a window or initialization of EGL.

my old IrrlichtBaW way of doing multithreaded OpenGL used XInitThreads with great success.

P.S. Can't we just call XInitThreads multiple times and not worry who calls it first? (EGL will call XInitThreads first thing, and so will our window creation)

[Crisspl]

Can't we just call XInitThreads multiple times and not worry who calls it first? (EGL will call XInitThreads first thing, and so will our window creation)

if thats legal, then problem solved

[devshgraphicsprogramming]

if thats legal, then problem solved

Ask the X11 mailing list ;)

[devshgraphicsprogramming]

if thats legal, then problem solved

Ask the X11 mailing list ;)

did any of you @Crisspl @sadiuk do this?

P.S. @sadiuk close this issue after implementing Linux and Android stuff in the ui and system namespaces.

[devshgraphicsprogramming]

@Nihon11 look at this as a reference