Support directly using Operating System (hardware) cursors

[ericoporto]

Describe the problem Nowadays with monitors that uses different refresh rates, some with 144Hz, most still 60Hz, players may be used to their own computer refresh rates. While they may not care about the actual game refresh rate, the input system may imply a different feedback, in this case the mouse cursor may be perceived disconnected from their input if they are used to it.

Suggested change The feature here is to provide the possibility of using OS cursor directly instead of rendering in engine.

• probably leverage SDL_CreateColorCursor
  • This may fail with returning a nullptr, and SDL_error will message "Cursors are not currently supported", this may be a config error in the particular platform if the person builds from source and not turn this on, usually only in very old SDL2 in old Linux packages.
• possible optionally (game or winsetup or per platform? )
• applying custom cursors so it works indepently from the game
• mouse cursor though will probably render in between pixels
• needs to be scaled like the game
  • possibly using an AA aware method if the game has bilinear filtering set (but it seems we don't have this with alpha in bitmap form)
• make sure it's destroyed if game crashes so that regular cursor doesn't change, this looks like is taken care by SDL2, but just need to recheck

Additional context

• https://steamcommunity.com/app/1346360/discussions/0/3416557114763335804/?tscn=1655196272
• https://www.adventuregamestudio.co.uk/forums/index.php?topic=59842.msg636647042#msg636647042

[ivan-mogilko]

So, to clarify, this is about what is also known as "hardware cursors", that should not depend on the game's own render?

[ericoporto]

Yes! Did AGS supported those in the past?

Anyway, not sure if this makes sense or not but it was the lower scope alternative I could think to solve the problem reported in the "additional context" without using the render and threads.

---

• Engine/device/mousew32.cpp: probably add something here for showing/hiding cursor and passing image for cursor
• Engine/ac/mouse.cpp: code for setting the mouse cursor is here
• probably a test is due if just showing the system mouse there works, since it looks like there's position control and acceleration handling in the engine
• this is a point and click engine at heart, so mouse is used in a million places, need to think through if this is actually a path that makes sense to solve the reported issue.

---

Made a terrible version here: https://github.com/ericoporto/ags/tree/test-hardware-cursor

It has memory leaks, the Red and Blue channels are swapped and the cursor doesn't scale. The following questions arisen when I played around it.

• how to scale the mouse cursor bitmap and where to place this interface, possibly if the window scales, the mouse cursor has to update it's graphics in this case
• wrap an allegro bitmap as an SDL_Surface instead. Where this code should be placed?
• somewhere AGS makes the cursor invisible, basically if OS cursor is used, it needs to be visible and the in-game cursor needs to be hidden. Add an interface for this in mouse and a game config?

[morganwillcock]

I don't think it would be reliable to use a hardware cursor in all situations. I don't think they can have an arbitrary size (it will depend on the underlying display system as to what works and what doesn't) and I've seen some systems where their display gets corrupted. You would probably end up needing seperate implementation paths for each platform for it to be usable without the common parts becoming too restrictive.

Nowadays with monitors that uses different refresh rates, some with 144Hz, most still 60Hz, players may be used to their own computer refresh rates. While they may not care about the actual game refresh rate, the input system may imply a different feedback, in this case the mouse cursor may be perceived disconnected from their input if they are used to it.

To me this is purely an engine design problem rather than a prompt to implement hardware cursors.

[ericoporto]

I've seen some systems where their display gets corrupted

I have literally never seen a system using a mobile Nvidia quadro card where the cursor DOESN'T get occasionally corrupted.

You would probably end up needing seperate implementation paths for each platform for it to be usable without the common parts becoming too restrictive.

In my head it would be just desktop platforms, and there would be an option to use it or not in winsetup/acsetup.cfg.

To me this is purely an engine design problem rather than a prompt to implement hardware cursors.

We can totally decide this is not a good idea and go the way of adjusting the renderer to allow this with our current in engine cursor! Or any other way! I just did not felt like opening a bug report at the time, because this is not a bug for me, it's a new feature.

We can just close this as won't fix and then figure another approach if we want to cater to that user story - I like Jira concepts.

[wchristian]

To me this is purely an engine design problem rather than a prompt to implement hardware cursors.

(Not sure if meant this way, if not disregard my post, but this reads as: "The in-engine-rendered cursor should be improved.")

Not entirely. In my experience, and doing external testing with cameras and such while working on a game and deciding on proper timing loop implementation: It is almost impossible to achieve the very low hardware cursor input latency with in-engine rendering due to limitations within implementations like OpenGL or DirectX themselves. For this reason alone the vast majority of modern cursor-driven games uses the hardware cursor rather rendering it on their own. You can verify this by checking whether the games you care about render the cursor beyond the window chrome, or only inside it.

[morganwillcock]

For this reason alone the vast majority of modern cursor-driven games uses the hardware cursor rather rendering it on their own. You can verify this by checking whether the games you care about render the cursor beyond the window chrome, or only inside it.

I imagine such games are already processing events and able to keep their display up-to-date at vsync speed. If just adding hardware cursors on top of the existing AGS update loop (that is linked to the defined game speed) it would be superficially smoother but the input is effectively de-synced from engine.

I guess I'm saying that the original description of the problem is actually the opposite of the way it reads:

While they may not care about the actual game refresh rate, the input system may imply a different feedback, in this case the mouse cursor may be perceived disconnected from their input if they are used to it.

I would say the problem is that the input is connected with the engine but the engine does not differentiate between rendering/input and the game logic. The user may feel their input is slow or lossy in terms of inputs events, but that is because that is exactly how the engine works. How it feels is representative of what is going on. Implementing hardware cursors on the current engine would cause the input and game to be disconnected.

[messengerbag]

Yes, I would argue that the proposal is fundamentally impossible within the paradigm of the engine, because the appearance of the cursor (and potentially other logic to do with the mouse position) is under the control of the game script. Therefore, you cannot decouple the cursor rendering from the game loop without introducing bugs and artifacts (such as the cursor failing to highlight when you sweep it over a hotspot, for example).

I see how it can be feasible in game engines that are closely customized for a particular game, where you might implement this logic in the engine itself, but AGS is a general-purpose engine that allows game makers a great deal of flexibility, primarily through the scripting language. So details of how the UI behaves are coded in the game script, and therefore cannot update faster than the game loop.

[wchristian]

@morganwillcock You're correct about the connected/disconnected matter, however it is important to keep in mind that that only matters if the mouse input is used to drive a physical representation, like rolling a ball or rotating a camera. However the mouse cursor often does not have such a connection, such as in adventure games, strategy games, visual novels, etc. :)

@messengerbag It's not just feasible in customized game engines, it's the default in many engines that are currently the market default, unity and friends. If it's truly a hard technical limitation y'all might want to consider whether you truly want to stay behind on that matter.

[ericoporto]

@wchristian In feature requests we usually go over some implementation details, if it breaks backwards compatibility or not, api design and tradeoffs with other solutions for the same problem. If you look at Contributing (README), you will see that the current plan was to "add improvements to existing functionality" and develop new things in a different branch. AGS 3 has been maintaining binary compatibility with almost 20 years of games, so there's many things that may seem trivial from outside, but are much more complex to execute. Different game engines are happily, different.

ags_hardware_cursor_experiment.zip This is only an experiment ok? Just copy this exe in the directory game you want to test this, and double click it to run with hardware cursor. In case it complains you are missing SDL2.dll, just download it here.

---

So looking on the problem at hand, I recorded and slowed down a video of my experiment, it looks like there will be a delay between the mouse cursor position on screen and what the game developer expected, always at least one frame behind - this confirms the expectation from how the game logic works.

The user may feel their input is slow or lossy in terms of inputs events, but that is because that is exactly how the engine works. How it feels is representative of what is going on. Implementing hardware cursors on the current engine would cause the input and game to be disconnected.

I cannot tell any solution in ags3 scope for this other than providing a "slightly" disconnected experience and let the game developer decide if they want to make such option available or not. To let the game developer be in full control this could be provided only through the script API (possibly bool System.HardwareCursor)

Say they feel their game works in such approach - meaning it's disconnected, but such disconnection is tolerable because your game elements are not moving crazily fast (like Starcraft), or you don't heavily use drag and drop in the game interaction - which also feels like the game elements slide a bit with the mouse, unless such feature is done by altering the game cursor in a seamless way, like using a dynamic surface to combine the cursor and game object graphic and apply it to the cursor.

About the code issues,

• managed to fix the rgb channels,
• found a solution for the scaling
• removed the allegro modification, and added GfxUtil::CreateSDL_SurfaceFromBitmap in Engine/gfx/gfx_util.{h|cpp}
• Found a new issue, it appears there's an offset between the hardware cursor and the in-game cursor, but I can't yet tell from where it originates ok, fixed by correctly scaling the hotspot position
• Noticed that some game modifies the game cursor graphics many times during the frame but it's only intended for the last cursor from frame to appear, but since the hardware cursor is instantly applied, this may result in flickering between different graphics in this cases.
  • This kinda reinforces my beliefe that this is not good to apply to previous games, but the flickering can be solved by only applying the last value set in the frame, if it changed from the previous frame, but it's not trivial in the code organization how to accomplish this.

With the amount of open PRs I have in the air I am not feeling confident to tackle this right now, and would prefer to go back to what I was already working on.

[ivan-mogilko]

AGS 3 has been maintaining binary compatibility with almost 20 years of games

To clarify, it's not binary compatibility, but API and game logic compatibility.

It might be still possible to maintain this compatibility having larger changes in the engine, similarily to how we (try to) maintain it after changing backend libraries from allegro4 to SDL2, and rewrote audio handling. Of course, if the required changes are large, then it should not be part of 3.6.0, but the next version (as 3.6.0 is already in the late beta).

The questions here are 1) what is the wanted behavior, 2) what would have to be changed to achieve this behavior. After that is established and documented, we may look and investigate whether it's possible to keep old games working and how (or not). In the worst case this change will go strictly into ags4.

Does the system cursor fix the original problem though? I guess that's the first thing to confirm, because otherwise trying to fix any side issues would be a waste of time.

---

This kinda reinforces my beliefe that this is not good to apply to previous games, but the flickering can be solved by only applying the last value set in the frame, if it changed from the previous frame, but it's not trivial in the code organization how to accomplish this.

I don't think that's exactly a problem with old games, as same behavior may be met in the newly made ones. There are already number of optimizations in the engine, where a accumulated or final change is only applied once the scripts were finished; so mouse cursor may be another one.

There's update function that starts script callbacks, there are functions that run these callbacks directly. Applying new cursor may be done either after all scripts have finished in update, or after each callback exits (e.g. see post_script_cleanup). Also, the final mouse update is here, this is where it does last update before rendering the cursor: https://github.com/adventuregamestudio/ags/blob/master/Engine/ac/draw.cpp#L2562

[wchristian]

@ericoporto

Appreciate the explanation. :)

ags_hardware_cursor_experiment.zip This is only an experiment ok? Just copy this exe in the directory game you want to test this, and double click it to run with hardware cursor.

Appreciate this incredibly much. I gave it a lengthy test run in "Old Skies" and it works perfectly and is exactly how it should be, except for a few tiny matters of ux adaptation. (The coffee cup cursor is a tad small on a large resolution.)

I recorded and slowed down a video of my experiment, it looks like there will be a delay between the mouse cursor position on screen and what the game developer expected, always at least one frame behind - this confirms the expectation from how the game logic works.

This is in fact how it works in almost all games ever. Even modern Starcraft 2 has this issue. Below are two screencaps of when the (green) cursor is on the unit, and when the tooltip appears. Other things also don't work well such as animated cursors. Modern games do however also do workarounds to ameliorate this, for exampling by modifying when in the update loop cursor positions are sampled, and when those updated positions are made available to the rendering logic.

@ivan-mogilko

Does the system cursor fix the original problem though?

Yes, emphatically. Erico's test does exactly what i hoped for. I recommend trying it out.

[ericoporto]

(The coffee cup cursor is a tad small on a large resolution.)

@wchristian, Oh, that is just because I am not hiding the mouse when I should have, the bluecup means the mouse is hidden.

I updated the link on the previous post so it's fixed now.

---

the final mouse update is here

Thanks, I used that and had no more flickering!

The questions here are 1) what is the wanted behavior, 2) what would have to be changed to achieve this behavior.

@ivan-mogilko I am still thinking about this. Will update when I have something figured.

[ericoporto]

ok, thought a bit more

1) what is the wanted behavior

From this ticket, it seems it would be to be able to play AGS games using hardware cursors, independently if they were built with this in mind or not. It looks like other than position being desynced for a frame there is no downside, assuming the platform supports it. Also this is a feature that makes sense only on desktop platforms.

If we want this to be in the developer hands instead, than we can move this for ags4 and think about adding a flag in game or Mouse for this. The opposite of ags4 is not necessarily 3.6.0, it can be 3.6.1 or what is the next 3 release.

2) what would have to be changed to achieve this behavior

I think adding a command line flag/config entry would suffice, so if it's on, then it would render the cursor using it instead of the game render.

made branch that attempts it, but the changes are not fully correct yet in code I think: https://github.com/ericoporto/ags/tree/feature-hardware-cursor

[wchristian]

Just chiming in that all of that sounds correct, except for: There would be two possible downsides i can think of:

• i think animated hardware cursors aren't possible
• special rendering effects for hardware cursor (shaders, etc.) aren't possible at all

[ivan-mogilko]

what is the wanted behavior what would have to be changed to achieve this behavior

it seems it would be to be able to play AGS games using hardware cursors, independently if they were built with this in mind or not.

That's a good overall goal, but please note that by these questions I also meant the cursor / program behavior. How is it initialized, how and when is it assigned the graphic, does it have same graphic as a normal game's cursor would have, and so forth. Basically, a feature description.

[ericoporto]

How is it initialized, how and when is it assigned the graphic, does it have same graphic as a normal game's cursor would have, and so forth

I don't know yet the answer to these, but I noticed in some systems I can assign the cursor graphic on every frame and it's fine, but in some systems, if I do this, the cursor does a minor flicker sometimes.

If it does have a special case to not do this on every frame, then it's possibly better if it's developer controlled - say turning this on disables assigning to Mouse.Graphic at runtime.

[ivan-mogilko]

How is it initialized, how and when is it assigned the graphic, does it have same graphic as a normal game's cursor would have, and so forth

I don't know yet the answer to these, but I noticed in some systems I can assign the cursor graphic on every frame and it's fine, but in some systems, if I do this, the cursor does a minor flicker sometimes.

In this I like to separate the planned behavior, and specific solutions. In other words: what would we want, and what we may achieve. For example, if the graphic cannot be reassigned every frame, then it may be intentionally postponed by the engine, assuming this behavior is also clearly documented.

From the above I can see that there are already two major cases: when the game was not configured with intent to use hardware cursors, and when the game was built with intent to use them from author's side. Possibly there may be a default behavior where engine decides what to do, optionally based on config, and a custom behavior with certain commands added in the script. The second case will override the first one. They may even not be added in the same version.

[ericoporto]

For example, if the graphic cannot be reassigned every frame, then it may be intentionally postponed by the engine, assuming this behavior is also clearly documented.

There are two other possibilities too, caching cursor sprites (as SDL cursors) or caching all initial cursors and only do at runtime for inventories and image changes (may be impractical because a view may have too many frames). Or having a separate API to prepare the sprite - will require more care from the game authors.