Add support of keyboard and mouse

[sergystepanov]

WIP This PR adds mouse and keyboard controls for aplicable Libretro cores. As simple as it may sound, the main idea is to capture browser Keyboard and Mouse API events and then feed them in a digestible form to the Libretro frontend. Unfortunately, there are several problems related to the original design choices of cloud-game. Initially, it was assumed that each player would have just one controller device occupying a single controller port in terms of the underlying Libretro API. This way, it would be pretty easy to distinguish controller events of each player, knowing that you can safely forward player control events to their dedicated port without the fear of interfering with other players. However, that was quite a wrong assumption, thinking that one player occupies just one device port. In reality, many cores allow the use of several different controller devices, in theory mapped to an unlimited number of ports (in practice, only up to 9 ports can be used), or even multiplex multiple devices (merge controller events) on a single port, as in the case of the DOSBox Pure core.

Keyboard in Libretro

To use a keyboard with Libretro, two callback functions from the core are required:

1. Keyboard Callback Function: RETRO_ENVIRONMENT_SET_KEYBOARD_CALLBACK

   • Set when the core pushes out the environment variable with the same name.
   • This callback should be called once a user presses or releases a keyboard key.
   • It's important to note that the keyboard controls in Libretro do not have a distinct port argument in the callback function. This limitation makes it impossible to separate player keyboards in cores like DosBox, even though DosBox polls the state of pressed keys for each port separately in the polling callback.
   • The callback function has the following parameters:
     • bool down: Indicates if the key is pressed.
     • unsigned keycode: Internal Libretro representation of the pressed key.
     • uint32_t character: Ignored.
     • uint16_t key_modifiers: Special key modifiers such as shift, alt, ctrl, meta, num/caps/scroll-locks.

2. Device Callback Function: RETRO_DEVICE_KEYBOARD

   • It is polled after each emulation tick in the core_input_state polling function.
   • Each key that was previously pressed by the keyboard callback function must be left in the pressed state or released (by setting the device callback function result to 1 or 0).

Mouse in Libretro

Mouse controls in Libretro are polled similarly to the keyboard, using the core_input_state function with the device name RETRO_DEVICE_MOUSE. The behavior of the callback depends on the id parameter:

• Button States:

  • When the id parameter corresponds to a mouse button, the callback returns the state of that button (pressed or released).
  • In this PR, only the states of the Left, Right, and Middle buttons are implemented.

• Cursor Position:

  • When the id parameter corresponds to the cursor position, the callback returns the position of the mouse cursor.
  • It should be noted that in the case of DOSBox, the X and Y positions of the mouse are not absolute, but rather the relative deltas of X and Y coordinates between two emulation ticks.

Scale cursor speed to native resolution

Since raw pointer movement is not supported in all browsers or platforms, when you stream a display which size is different from a display on which you use mouse, the mouse movement (DPI) should be approximated accordingly. Used algorithm takes coordinates (movement delta) from the virtual display (cursor over the browser video element) and scales them to the default DosBox VGA display size of 640x480. Because the scale ratio value is a floating point number and we can't use subpixel coordinates on the server side, we need to take into account accumulated floating point errors when rounding coordinates to the destination screen (Libretro framebuffer). See an example.

[guozanhua218]

WIP This PR adds mouse and keyboard controls for aplicable Libretro cores. As simple as it may sound, the main idea is to capture browser Keyboard and Mouse API events and then feed them in a digestible form to the Libretro frontend. Unfortunately, there are several problems related to the original design choices of cloud-game. Initially, it was assumed that each player would have just one controller device occupying a single controller port in terms of the underlying Libretro API. This way, it would be pretty easy to distinguish controller events of each player, knowing that you can safely forward player control events to their dedicated port without the fear of interfering with other players. However, that was quite a wrong assumption, thinking that one player occupies just one device port. In reality, many cores allow the use of several different controller devices, in theory mapped to an unlimited number of ports (in practice, only up to 9 ports can be used), or even multiplex multiple devices (merge controller events) on a single port, as in the case of the DOSBox Pure core.

Keyboard in Libretro

To use a keyboard with Libretro, two callback functions from the core are required:

1. Keyboard Callback Function: RETRO_ENVIRONMENT_SET_KEYBOARD_CALLBACK

   • Set when the core pushes out the environment variable with the same name.
   • This callback should be called once a user presses or releases a keyboard key.
   • It's important to note that the keyboard controls in Libretro do not have a distinct port argument in the callback function. This limitation makes it impossible to separate player keyboards in cores like DosBox, even though DosBox polls the state of pressed keys for each port separately in the polling callback.

   • The callback function has the following parameters:

     • bool down: Indicates if the key is pressed.
     • unsigned keycode: Internal Libretro representation of the pressed key.
     • uint32_t character: Ignored.
     • uint16_t key_modifiers: Special key modifiers such as shift, alt, ctrl, meta, num/caps/scroll-locks.

2. Device Callback Function: RETRO_DEVICE_KEYBOARD

   • It is polled after each emulation tick in the core_input_state polling function.
   • Each key that was previously pressed by the keyboard callback function must be left in the pressed state or released (by setting the device callback function result to 1 or 0).

Mouse in Libretro

Mouse controls in Libretro are polled similarly to the keyboard, using the core_input_state function with the device name RETRO_DEVICE_MOUSE. The behavior of the callback depends on the id parameter:

• Button States:

  • When the id parameter corresponds to a mouse button, the callback returns the state of that button (pressed or released).
  • In this PR, only the states of the Left, Right, and Middle buttons are implemented.

• Cursor Position:

  • When the id parameter corresponds to the cursor position, the callback returns the position of the mouse cursor.
  • It should be noted that in the case of DOSBox, the X and Y positions of the mouse are not absolute, but rather the relative deltas of X and Y coordinates between two emulation ticks.

Scale cursor speed to native resolution

Since raw pointer movement is not supported in all browsers or platforms, when you stream a display which size is different from a display on which you use mouse, the mouse movement (DPI) should be approximated accordingly. Used algorithm takes coordinates (movement delta) from the virtual display (cursor over the browser video element) and scales them to the default DosBox VGA display size of 640x480. Because the scale ratio value is a floating point number and we can't use subpixel coordinates on the server side, we need to take into account accumulated floating point errors when rounding coordinates to the destination screen (Libretro framebuffer). See an example.

Could it support virtual keyboard and virtual mouse features which support by retro on mobile? and If run on PC platform, could it supports native mouse and keyboard events?