Making zahnrad touch screen friendly.

[richi]

I am right now toying around building an interface for iOS and Android and potentially other touch screen devices. Getting a rough port up running on iOS was pretty easy, thanks to the clean structure of zahnrad. I have discovered only one major show stopper and this is the mouse pointer centric approach. On a touch screen there is no concept of hovering for example. And most widgets (those that do not involve dragging) do fire on touch up and not down.

As a workaround I am sending a motion event right before a touch down.

zr_clear(&gui.ctx);

zr_input_begin(&gui.ctx);
zr_input_motion(&gui.ctx, location.x, location.y);
zr_input_end(&gui.ctx);

zr_input_begin(&gui.ctx);
zr_input_button(&gui.ctx, b, location.x, location.y, zr_true);
zr_input_end(&gui.ctx);

run_demo(&gui);

and again a motion event right after a touch up

zr_clear(&gui.ctx);

zr_input_begin(&gui.ctx);
zr_input_button(&gui.ctx, b, location.x, location.y, zr_false);
zr_input_end(&gui.ctx);

zr_input_begin(&gui.ctx);
zr_input_motion(&gui.ctx, 0, 0);
zr_input_end(&gui.ctx);

run_demo(&gui);

This does work surprisingly well, but of course feels quite hackish and does not solve the touch up vs. down issue.

Another issue is the hit test. On a touch screen it is rather difficult to hit small widgets and the solution usually is to extend the hit area virtually to at least 40 x 40 pixel.

I see a huge potential for zahnrad on touch devices.

What do you think?

Cheers, richi

[vurtun]

Hey, wow amazing work.

I am right now toying around building an interface for iOS and Android and potentially other touch screen devices.

That would be amazingly helpful especially to hear about things that do not work out as well for touch devices. Currently zahnrad also does not support multiple (mouse/touch) pointer which should hinder porting as well. Looks like I definitely have to work or rework the input to better support touchscreen devices.

I have discovered only one major show stopper and this is the mouse pointer centric approach. On a touch screen there is no concept of hovering for example. And most widgets (those that do not involve dragging) do fire on touch up and not down.

The fire on touch up instead if touch down is in hindsight probably the better approach even for standard desktop UI so I will rewrite all button interaction, which hopefully should not be to hard. About hovering I don't really understand while touch screens probably do not use hovering does it cause any problems?

Another issue is the hit test. On a touch screen it is rather difficult to hit small widgets and the solution usually is to extend the hit area virtually to at least 40 x 40 pixel.

Height and width of widgets can be easily controlled by each zr_layout_xxx function (each one has a row height parameter which is 30 in the demo most of the time but can also any other number).

zr_begin(&context, "Demo", zr_rect(0, 0, window_width, window_height),0);
{
    /* Button with size: (width: 80, height: 40) pixel   */
    zr_layout_row_static(&ctx, 40, 80, 1);
    if (zr_button_text(&ctx, "button", ZR_BUTTON_DEFAULT)) {
        /* event handling */
    }
}
zr_end(ctx);

The only problem is font height which can often depend on the renderbackend and the height of the font loaded. It is probably a good idea to create a custom demo or window inside the demo for touchscreens with bigger widgets and bigger font. So far great work and I am really interested on how far you can go and find.

[richi]

while touch screens probably do not use hovering does it cause any problems?

First zr_input_is_mouse_hovering_rect is used in nearly all state changing functions like zr_button_behaviorfor example. There is no 'down' detection without a 'motion' up front (mouse.prev). Second it simply looks ugly :) because the widgets change their state and therefore the color etc.. For example the last touched button does have a different color until another one is pressed. Maybe moving the hack to zr_input_button could be a quick (but still somewhat hackish) solution:

void
zr_input_button(struct zr_context *ctx, enum zr_buttons id, int x, int y, int down)
{
    struct zr_mouse_button *btn;
    struct zr_input *in;
    ZR_ASSERT(ctx);
    if (!ctx) return;
    in = &ctx->input;
    if (in->mouse.buttons[id].down == down) return;

    if (is_touch_screen && down)
    {
       in->mouse.pos = zr_vec2(x, y);
       in->mouse.delta = zr_vec2(0, 0);
       zr_vec2_mov(in->mouse.prev, in->mouse.pos);
    }

    btn = &in->mouse.buttons[id];
    btn->clicked_pos.x = (float)x;
    btn->clicked_pos.y = (float)y;
    btn->down = down;
    btn->clicked++;

    if (is_touch_screen && !down)
    {
        in->mouse.pos = zr_vec2(0, 0);
        in->mouse.delta = zr_vec2(0, 0);
        zr_vec2_mov(in->mouse.prev, in->mouse.pos);
    }
}

Height and width of widgets can be easily controlled

Absolutely, but every now and than, one does need smaller widgets and for those one could increase the hit rectangle to at least 44 x 44. Virtually so to speak, just for the hit test. If things overlap the one nearest to the down point is the winner.

So far great work and I am really interested on

Gorgeous, let's start the engines!

[richi]

The fire on touch up instead if touch down is in hindsight probably the better approach even for standard

Great, that would make one finger scrolling easier:

 if (nothing_draggable_is_hit && down && motion)
 {
       if (has_scrollbar)
           do_scrolling;
}

[vurtun]

Edit: forget what I said I missunderstood the problem. Have to think about a possible solution.

[richi]

No problem at all.

[vurtun]

Quick idea you can do this

zr_input_begin(&gui.ctx);
if (touch_up_event) {
    zr_input_motion(&gui.ctx, location.x, location.y);
    zr_input_button(&gui.ctx, b, location.x, location.y, zr_false);
} else if (touch_down_event) {
    zr_input_motion(&gui.ctx, location.x, location.y);
    zr_input_button(&gui.ctx, b, location.x, location.y, zr_false);
}
zr_input_end(&gui.ctx);

and solve the different color while hovering by just changing all the hovered colors to the same color as the default color. Then you do not have to do the zeroing of the mouse position at the end of the frame.

[richi]

Of course, one could wrap it like that:

void
zr_input_touch(struct zr_context *ctx, int count, int* x, int* y, int down)
{
    // ignoring multiple touches for now
    zr_input_motion(&gui.ctx, x[0], y[0]);
    zr_input_button(&gui.ctx, ZR_BUTTON_LEFT, x[0], y[0], down);
}

But to be honest, not zeroing the mouse position at touch up sounds like asking for trouble to me. Is there a technical reason why you want to avoid it? The thing is, on a touch screen there is no relation between downs, no delta no nothing.

[vurtun]

Absolutely, but every now and than, one does need smaller widgets and for those one could increase the hit rectangle to at least 44 x 44. Virtually so to speak, just for the hit test. If things overlap the one nearest to the down point is the winner.

Ah OK that should be possible but I don't know if it should be an relative value like 'add x pixel padding to every clickable bounding box' or to use an absolute minimum with 'every widget has to have at least N x N sized bounding box'. I will probably just add an absolute value to zr_style that will be added as padding for each widget.

But to be honest, not zeroing the mouse position at touch up sounds like asking for trouble to me. Is there a technical reason why you want to avoid it? The thing is, on a touch screen there is no relation between downs, no delta no nothing.

OK I probably have to back up and first clarify the problem how I understood. As far as I understand the condition checking if something is hovered is a problem with touchscreens. So a possible solution from you was to use zr_input_motion before clicking/touch on zr_button_button. The problem then was that because hovering results in a different color scheme the button kept having another color. To fix that you added zeroing on button click up to fix that issue. That is how far understood the problem.

So my idea to the color problem was to set all ZR_COLOR_XXX_HOVERED colors to the same color as their corresponding default colors to stop having a different color while being hovered. The reason for that is that I want this library to be as independent from the used platform as possible and not add any functions or conditions that are specific to any platform (touch or otherwise) if possible (at least your proposed zr_input_button function depended on a condition if touchscreen). So I directly do not care if it is being zeroed or not but rather that the resulting API is as platform independent as possible.

[richi]

but rather that the resulting API is as platform independent as possible.

I fully understand your approach and I guess that is what makes zahnrad as great as it is. But I think it will get very difficult without any "is_touch_screen" flag or zr_touch functions at all. Certain things are very different using touch screen or mouse. Another example:

After doing things like minimizing or maximizing a window, the next frame will be somewhat garbled. That does not cause much grieve running at 60 fps on a desktop machine. But on mobile machines one can not afford such a tight loop. What one does is cacheing the ui tree (preferably on the GPU) after a user input and using this cache during the next frames until the user generates more input. Right now i have to always build the tree twice after each user input to avoid such artefacts. Not a big deal, but would it be possible to detect such a situations and signal the need for a second run with a dirty flag? Something like:

zr_input_begin()
 do_input...
zr_input_end()

while (zr_frame_is_dirty()) {
    do_demo();
}

If you want to try how it feels on a touch screen device, just modify the demo and send only up and down events and only generate a new frame after such an event.

But no question at all, one should keep the differences as small as possible.

[richi]

I will probably just add an absolute value to zr_style that will be added as padding for each widget.

That would be perfect!

[vurtun]

If you want to try how it feels on a touch screen device, just modify the demo and send only up and down events and only generate a new frame after such an event.

Oh you mean event based updating. All example/ demos are event based. I don't know how apple does input but maybe you can take away some things from the demos.

But I think it will get very difficult without any "is_touch_screen" flag or zr_touch functions at all.

In the end do what needs to be done, but in general if there are multiple solutions to a problem and one does not require any platform dependent code it is probably best to use it. So don't try to force it but if it means that a little bit more work has to be done by the user to get things running I prefer platform independent code over usability for a small fraction of users.

After doing things like minimizing or maximizing a window, the next frame will be somewhat garbled. That does not cause much grieve running at 60 fps on a desktop machine

Do you require minimizing? I thought smartphones and pads don't have it?

[vurtun]

Not a big deal, but would it be possible to detect such a situations and signal the need for a second run with a dirty flag?

This is probably quite hard to detect, but I have to think about it.

[richi]

I don't know how apple does input but maybe you can take away some things from the demos.

The idea was to only use "up" and "down" events just to get a better feeling for the problems on a touch screen :)

Do you require minimizing? I thought smartphones and pads don't have it?

Well, it was just meant as an example, but certain Android versions do actually have it and it is of course great functionality.

This is probably quite hard to detect, but I have to think about it.

No need for a hurry.

Sorry for opening a can of worms. :+1:

[vurtun]

The idea was to only use "up" and "down" events just to get a better feeling for the problems on a touch screen :)

Ah OK my bad.

Not a big deal, but would it be possible to detect such a situations and signal the need for a second run with a dirty flag?

OK I am not 100% certain but it seems highly possible that minimizing and maximizing does not require another GUI tree pass since everything can be resolved in zr_layout_begin. I will make some changes and report back the results.

[vurtun]

I was able to remove the additional required frame for minimizing/maximizing. There are still some widgets or systems that require at least one additional frame that cannot be simply changed, but there is a high possibility that those are not needed for touch screen (popups).

[richi]

Looks promising, but there are some issues.

If you change a few lines in the file demo/x11/xlib.c you will have a pretty good touch screen simulator running on a mobile device and you will see:

while (running) {
    /* Input */
    XEvent evt;
    started = timestamp();
    zr_input_begin(&gui.ctx);
    while (XCheckWindowEvent(xw.dpy, xw.win, xw.swa.event_mask, &evt)) {
        if (XFilterEvent(&evt, xw.win)) continue;
        if (evt.type == KeyPress)
            input_key(&xw, &gui.ctx, &evt, zr_true);
        else if (evt.type == KeyRelease)
            input_key(&xw, &gui.ctx, &evt, zr_false);
        else if (evt.type == ButtonPress) {
            input_button(&gui.ctx, &evt, zr_true);
    // important, only generate UI after up or down
            break;
        }
        else if (evt.type == ButtonRelease) {
            input_button(&gui.ctx, &evt, zr_false);
    // important ,only generate UI after up or down
            break;
        }
        else if (evt.type == MotionNotify)
            input_motion(&gui.ctx, &evt);
        else if (evt.type == Expose || evt.type == ConfigureNotify)
            resize(&xw, xw.surf);
        else if (evt.type == KeymapNotify)
            XRefreshKeyboardMapping(&evt.xmapping);
    }
    zr_input_end(&gui.ctx);

// important, only generate UI after up or down
    if (evt.type != ButtonPress || evt.type != ButtonRelease)
      continue;

    /* GUI */
    running = run_demo(&gui);

[richi]

should be:

// important, only generate UI after up or down if (evt.type != ButtonPress && evt.type != ButtonRelease) continue;

[vurtun]

Thanks I will try it out

[richi]

Cool! For a second stage you could include the motion events as well, but please try it first with up and down only. With the latest commit my event handling has to look like the following to make it work:

if (down) {
    zr_clear(&gui.ctx);
    zr_input_begin(&gui.ctx);
    zr_input_motion(&gui.ctx, location.x, location.y);
    zr_input_end(&gui.ctx);
    run_demo(&gui);

    zr_clear(&gui.ctx);
    zr_input_begin(&gui.ctx);
    zr_input_button(&gui.ctx, ZR_BUTTON_LEFT, location.x, location.y, zr_true);
    zr_input_end(&gui.ctx);
    run_demo(&gui);

    send_to_GPU();

} else if (up) {
    zr_clear(&gui.ctx);
    zr_input_begin(&gui.ctx);
    zr_input_motion(&gui.ctx, location.x, location.y);
    zr_input_end(&gui.ctx);
    run_demo(&gui);

    zr_clear(&gui.ctx);
    zr_input_begin(&gui.ctx);
    zr_input_button(&gui.ctx, b, location.x, location.y, zr_false);
    zr_input_end(&gui.ctx);
    run_demo(&gui);

    zr_clear(&gui.ctx);
    zr_input_begin(&gui.ctx);
    zr_input_motion(&gui.ctx, 0, 0);
    zr_input_end(&gui.ctx);
    run_demo(&gui);

    send_to_GPU();

} else if (moved) {
    zr_clear(&gui.ctx);  
    zr_input_begin(&gui.ctx);
    zr_input_motion(&gui.ctx, location.x, location.y);
    zr_input_end(&gui.ctx);
    run_demo(&gui);

    send_to_GPU();
}

A little crowded, but it works :)

What causes a all the confusion is probably the fact that I can not afford to run the event loop for the UI in polling mode to generate and send the GUI tree for every frame on those small machines, At least not if the UI gets a little heavier. Unfortunately it is necessary to cache as much data as possible with the GPU. Sending huge amounts of data from CPU to GPU for every frame is too expensive.

[vurtun]

For

if (down) {
    zr_clear(&gui.ctx);
    zr_input_begin(&gui.ctx);
    zr_input_motion(&gui.ctx, location.x, location.y);
    zr_input_end(&gui.ctx);
    run_demo(&gui);

    zr_clear(&gui.ctx);
    zr_input_begin(&gui.ctx);
    zr_input_button(&gui.ctx, ZR_BUTTON_LEFT, location.x, location.y, zr_true);
    zr_input_end(&gui.ctx);
    run_demo(&gui);
}

Is there a reason why you cannot just do:

if (down) {
    zr_clear(&gui.ctx);
    zr_input_begin(&gui.ctx);
    zr_input_motion(&gui.ctx, location.x, location.y);
    zr_input_button(&gui.ctx, ZR_BUTTON_LEFT, location.x, location.y, zr_true);
    zr_input_end(&gui.ctx);
    run_demo(&gui);
}

?

[vurtun]

I tested a little bit. So far what I noticed is that dragging stuff (window scaling, properties) at least for me in that demo does not work. In addition like you said buttons are still hovered after I clicked which can be easily fixed by just setting gui.ctx.colors[ZR_COLOR_BUTTON_HOVER] = zr_gba(50,50,50,255). The repeater button does not work at all at least for X11 which I currently don't know why. Anything that I missed?

[richi]

This is difficult to explain I have hacked together an X11 based simulator with a non polling UI event loop:

https://gist.github.com/richi/a7d9fdeff20157e4a64d#file-demo-c

Replacing main in /demo/x11/xlib.c with this two functions gives an exact simulation of the touch screen behaviour. Including my strange input dance :)

[vurtun]

Ok I tried it and it does not work for me at all since the performance drops quite hard even after I converted to a blocking loop. But I removed some iteration and it still seems to work:

if (evt.type == ButtonPress) {
    printf("down %d, %d\n", evt.xbutton.x, evt.xbutton.y);

    zr_clear(&gui.ctx);
    zr_input_begin(&gui.ctx);
    zr_input_motion(&gui.ctx, evt.xbutton.x, evt.xbutton.y);
    zr_input_button(&gui.ctx, ZR_BUTTON_LEFT, evt.xbutton.x, evt.xbutton.y, zr_true);
    zr_input_end(&gui.ctx);
    running = run_demo(&gui);

    send_to_GPU(&gui, xw);
    is_down = zr_true;
}
else if (evt.type == ButtonRelease) {
    printf("up %d, %d\n", evt.xbutton.x, evt.xbutton.y);

    zr_clear(&gui.ctx);
    zr_input_begin(&gui.ctx);
    zr_input_motion(&gui.ctx, evt.xbutton.x, evt.xbutton.y);
    zr_input_button(&gui.ctx, ZR_BUTTON_LEFT, evt.xbutton.x, evt.xbutton.y, zr_false);
    zr_input_end(&gui.ctx);
    run_demo(&gui);

    zr_clear(&gui.ctx);
    zr_input_begin(&gui.ctx);
    zr_input_motion(&gui.ctx, 0, 0);
    zr_input_end(&gui.ctx);
    running = run_demo(&gui);

    send_to_GPU(&gui, xw);
    is_down = zr_false;
}
else if (is_down && evt.type == MotionNotify) {
    printf("moved %d, %d\n", evt.xmotion.x, evt.xmotion.y);
    zr_clear(&gui.ctx);
    zr_input_begin(&gui.ctx);
    zr_input_motion(&gui.ctx, evt.xmotion.x, evt.xmotion.y);
    zr_input_end(&gui.ctx);
    running = run_demo(&gui);
    send_to_GPU(&gui, xw);
}

I was not able to remove the zeroing part tough.

[richi]

But I removed some iteration and it still seems to work:

removing iteration 1 in down does unfortunately break sliders, removing iteration 1 in up looks good.

Great!

Ok I tried it and it does not work for me at all

My bad, Wrong event flags and a missing XSetGraphicsExposures(xw.dpy, xw.surf->gc, 0). I checked it only on my machine, and my X11 experience is ages old. Now I checked it on Linux as well. I 've updated the gist:

https://gist.github.com/richi/a7d9fdeff20157e4a64d#file-demo-c

Hopefully it does now work as advertised on your machine as well.

[vurtun]

My bad, Wrong event flags and a missing XSetGraphicsExposures(xw.dpy, xw.surf->gc, 0).

Works perfectly now, thanks!

removing iteration 1 in down does unfortunately break sliders, removing iteration 1 in up looks good. Great!

Every saved iteration is probably quite beneficial for all energy depended devices. I tested removing the first pass in the down path but instead of breaking the slider it broke the dragging behavior in zr_property. I found out why and the problem lies inside zr_input and it does not require an additional frame so I was able to remove the first pass of button press. The reason is that the zr_input needs another frame to calculate the mouse movement delta which only requires a call to zr_input_end and not a complete pass.

if (evt.type == ButtonPress || evt.type == ButtonRelease || evt.type == MotionNotify) {
    if (evt.type == ButtonPress) {
        printf("down %d, %d\n", evt.xbutton.x, evt.xbutton.y);
        zr_clear(&gui.ctx);
        zr_input_begin(&gui.ctx);
        zr_input_motion(&gui.ctx, evt.xbutton.x, evt.xbutton.y);
        zr_input_end(&gui.ctx);

        zr_input_begin(&gui.ctx);
        zr_input_motion(&gui.ctx, evt.xbutton.x, evt.xbutton.y);
        zr_input_button(&gui.ctx, ZR_BUTTON_LEFT, evt.xbutton.x, evt.xbutton.y, zr_true);
        zr_input_end(&gui.ctx);

        running = run_demo(&gui);
        send_to_GPU(&gui, xw);
        is_down = zr_true;
    }
    else if (evt.type == ButtonRelease) {
        printf("up %d, %d\n", evt.xbutton.x, evt.xbutton.y);
        zr_clear(&gui.ctx);
        zr_input_begin(&gui.ctx);
        zr_input_button(&gui.ctx, ZR_BUTTON_LEFT, evt.xbutton.x, evt.xbutton.y, zr_false);
        zr_input_end(&gui.ctx);
        run_demo(&gui);
#if 1
        zr_clear(&gui.ctx);
        zr_input_begin(&gui.ctx);
        zr_input_motion(&gui.ctx, 0, 0);
        zr_input_end(&gui.ctx);
        running = run_demo(&gui);
#endif
        send_to_GPU(&gui, xw);
        is_down = zr_false;
    }
}

[richi]

Works great on the real thing als well!

[vurtun]

Sounds good. The only thing that does not seem work at all is the repeater button, which requires repeatedly being called while the button is being held down. I also tried to add additional optional touch padding but it will be a little bit more complex than I thought and I don't have a lot of time the next few weeks to it probably will not be added in quite some time.