not-fl3 / miniquad

Cross platform rendering in Rust
Apache License 2.0
1.59k stars 174 forks source link

Precise input #117

Open chayleaf opened 4 years ago

chayleaf commented 4 years ago

Currently input events are handled on update time. It isn't suitable for e.g. rhythm games. It can be implemented by recording event time when it happens. As framerate control isn't planned, I believe this is a necessary feature.

not-fl3 commented 4 years ago

Related macroquad issue: https://github.com/not-fl3/macroquad/issues/1

chayleaf commented 4 years ago

I checked sokol code, and it seems there's no easy way to handle input separately from drawing, at least because on Windows the thread blocks while waiting for the new frame, and events aren't being polled during that time. I think it should work correctly on wasm because the events are called from js directly. With that in mind, the easiest way to implement that would be to rely on the existing input system in wasm (and possibly some other platforms?) and creating a separate thread for polling input on other platforms. I'm not sure how it can fit in this library, so I will not be creating a pull request for adding that functionality, but I'll leave this as a hint for other people who want high precision input.

not-fl3 commented 4 years ago

Any good articles on high-level input events? I am curious what kind of games is sensitive to 0.01ms input lag caused by waiting for the end of a current frame for an event processing and what are the common practices to solve this, but cant really find anything on internet :(

chayleaf commented 4 years ago

One such example is rhythm games. Most actions in rhythm games need to be absolutely precise, 5ms offset is very noticable (I say it as a rhythm game player), and a (possible) 17ms offset at 60fps is completely inaccurate. In rhythm games, each hit is usually judged for accuracy, and human's rhythm sense can be perfected a lot, so the judgements can be pretty strict on high levels, but since they're so strict, even minimal delays can affect the judgement accuracy. For example, in the game osu!, at highest judgement level without further difficulty increasing modifiers (which a lot of players can handle well) the hitwindow for a perfect hit is around 20ms.

Other examples include shooters, where precise movement and reflexes are important as well, and other similar games.

Usually it's solved by either handling input in a separate thread or uncapping/increasing the framerate. In games with no such options players "solve" it by buying monitors with higher refresh rates.

not-fl3 commented 4 years ago

The problem - with current library design Context is going to be locked for the whole frame callback, so even introducing threads will not be really helpful. Right now I see only one solution - export some low-level primitives from sapp-* to allow building custom, platform-dependent event loop.

However, I still did not found any articles on common practices though and will think more about this.

nokola commented 4 years ago

I think low-level input is a critical for successful game library, or even if used for apps (e.g. for painting app.) There's something about the controls that feels "snappy" when the input is processed correctly. I have worked high-precision input for my Unity app and in Windows, will look some more into input in miniquad to understand how sokol handles it.

Context locked is not really a problem imo - as long as there's some place (separate thread) to accumulate input and have all the input (e.g. 30 touch points) that happened since last frame be available in the next frame. That way the app can respond independent of frame rate.

Copying typical high precision input in game loop from not-fl3/macroquad#1

frame_loop {

     while (input_event = input.poll()) { // get single input event
          // process input event:
          // imagine each input event has "type" e.g. mousedown/move/mouseup and 
          // absolute_time when it happened
         // ... some process code here ...
         // ... in some cases , e.g. Android we may have 100+ input evets to process ...
     }

    draw_frame();
}
nokola commented 4 years ago

Sending for information - example of high precision touch+move input (sampled as it comes) and low precision (sampled at Unity frame) on Android phone showing the significant difference.

The red dots are the input points and the blue lines are just connecting lines.

image

not-fl3 commented 4 years ago

I think low-level input is a critical for successful game library, or even if used for apps (e.g. for painting app.) There's something about the controls that feels "snappy" when the input is processed correctly. I have worked high-precision input for my Unity app and in Windows, will look some more into input in miniquad to understand how sokol handles it.

Context locked is not really a problem imo - as long as there's some place (separate thread) to accumulate input and have all the input (e.g. 30 touch points) that happened since last frame be available in the next frame. That way the app can respond independent of frame rate.

Copying typical high precision input in game loop from not-fl3/macroquad#1

frame_loop {

     while (input_event = input.poll()) { // get single input event
          // process input event:
          // imagine each input event has "type" e.g. mousedown/move/mouseup and 
          // absolute_time when it happened
         // ... some process code here ...
         // ... in some cases , e.g. Android we may have 100+ input evets to process ...
     }

    draw_frame();
}

But this is exactly how it works right now, isnt it?

nokola commented 4 years ago

this is exactly how it works right now, isnt it?

By "it" above do you mean miniquad, macroquad, or some sokol internal? I looked around the source of miniquad/macroquad but I didn't find the pattern above.

I found these patterns:

miniquad:

on_frame {
     draw_frame();
}
on_mouse_down() { ... } // may be fired asynchronously but will not be processed if frame is being drawn, and doesn't have timestamp for event
on_mouse_up() { ... }

and in macroquad:

frame_loop {
     if (mouse_down()) { ... } // will be processed with low precision - if mouse is moved multiple times in a frame, the movement in the frame is lost
     draw_frame();
     next_frame.await;
}

Maybe I'm missing something, please let me know!

Edit: Added again the first frame loop but with better comments to highlight the difference:

frame_loop {

     while (input_event = input.poll()) { // get *multuple* input events
         // can process multiple input events since last frame - e.g. 55 touches
         // each input event has a timestamp so that code can update physics/app state correctly as if the event was processed at the exact time
     }

    draw_frame();
    next_frame.await;
}
not-fl3 commented 4 years ago

In miniquad on most platforms main loop looks something like this

loop {
  while let Some(event) = os.poll_event() {
    context.on_whatever();
  }
  context.draw_frame();
}

Which is really similar to

frame_loop {     
     while (input_event = input.poll()) { // get single input event
          // process input event:
          // imagine each input event has "type" e.g. mousedown/move/mouseup and 
          // absolute_time when it happened
         // ... some process code here ...
         // ... in some cases , e.g. Android we may have 100+ input evets to process ...
     }

    draw_frame();
}

If I understood correctly, @pavlukivan wants events being pulled simultaneously with the draw_frame call and event callbacks being fired from some other thread.

For @nokola's examples, accurate events timestamps would be enough. The OS (at least windows and linux) does not include a timestamp to the event.

However I still think that both cases are kind of niche and it looks that for that kind of problems special platform-dependent event loop will be required anyway. Nothing better than provide platform-dependent primitives for building such an event loops for each platform comes to my mind.

nokola commented 4 years ago

In miniquad on most platforms main loop looks something like this

loop {
  while let Some(event) = os.poll_event() {
    context.on_whatever();
  }
  context.draw_frame();
}

Just checking - do you mean in sokol.h?

For @nokola's examples, accurate events timestamps would be enough.

Yes. @pavlukivan - what about you? Would timestamps be enough, or would separate thread be better?

@not-fl3 - do you see the high-precision input perhaps as separate library? (inputquad) Or some modification that we (one of us or whoever is interested) could do in miniquad?

Related notes for high precision input:

  1. Android: I looked for Android and this is the function to get touch event time when it happens https://developer.android.com/ndk/reference/group/input#group___input_1ga7e13fbf3cff0700b0b620284ebdd3a33

  2. Windows: I know Windows has high-precision events for stylus, and for mouse the docs say "doesn't matter" https://docs.microsoft.com/en-us/windows/win32/dxtecharts/taking-advantage-of-high-dpi-mouse-movement

  3. iOS: https://developer.apple.com/documentation/uikit/touches_presses_and_gestures/handling_touches_in_your_view/getting_high-fidelity_input_with_coalesced_touches

  4. WASM: ??

I guess more testing needed on actual device and performance tuning.

not-fl3 commented 4 years ago

Just checking - do you mean in sokol.h?

Windows: https://github.com/not-fl3/miniquad/blob/master/native/sapp-windows/external/sokol/sokol_app.h#L4605 linux/x11: https://github.com/not-fl3/miniquad/blob/master/native/sapp-linux/src/lib.rs#L2748 wasm: makes no sense, browser javascript engine is single-threaded anyway + event loop is handled inside the js vm android: https://github.com/not-fl3/miniquad/blob/master/native/sapp-android/external/sokol/sokol_app.h#L1671

chayleaf commented 4 years ago

I do think timestamps is enough, I even wrote "It can be implemented by recording event time when it happens" in the original post, and it basically is how I implemented my custom polling

not-fl3 commented 4 years ago

I do think timestamps is enough, I even wrote "It can be implemented by recording event time when it happens" in the original post, and it basically is how implemented my custom polling

Awesome!

It looks like high precision timings for events are enough. We can do it! I can see it as an optional parameter in Conf and a separate event loop for high-precision input. This will spawn additional thread and will collect events during the frame. But the event callbacks will be called after the frame callback finished. Something similar to DirectInput approach - all the API will be the same, but events will have "timestamp" data.