Add a gamepad API in the Engine

[ericoporto]

Describe the problem AGS is a Game Engine, and most engines have it's own Joystick API. In AGS, using Joysticks requires using a plugin. There's at least 3 different plugins floating around for this, and things aren't great for crossplatform with them.

If AGS had it's own gamepad API, then people would be able to make script modules with gamepad in mind.

Suggested change I would like to propose adding a gamepad API directly in AGS.

Now, there are thousands of different joysticks and SDL2 joystick API has a lot of surface, so there's obvious different ways to do this. I would like to proppose supporting SDL2 GameController API first and later figure out joystick

• I have an example implementation here (code has leftovers, still POC): github.com/ericoporto/ags/tree/feature-gamepad
• Editor and Engine Build: AGS-3.6.0.5-Gamepad.zip
• Example Project: GamepadExample.zip

Here's my current header!

```AGS Script enum eGamepad_Axis { eGamepad_AxisInvalid=0, eGamepad_AxisLeftX, eGamepad_AxisLeftY, eGamepad_AxisRightX, eGamepad_AxisRightY, eGamepad_AxisTriggerLeft, eGamepad_AxisTriggerRight, eGamepad_AxisCount }; enum eGamepad_Button { eGamepad_ButtonInvalid=0, eGamepad_ButtonA, eGamepad_ButtonB, eGamepad_ButtonX, eGamepad_ButtonY, eGamepad_ButtonBack, eGamepad_ButtonGuide, eGamepad_ButtonStart, eGamepad_ButtonLeftStick, eGamepad_ButtonRightStick, eGamepad_ButtonLeftShoulder, eGamepad_ButtonRightShoulder, eGamepad_ButtonDpadUp, eGamepad_ButtonDpadDown, eGamepad_ButtonDpadLeft, eGamepad_ButtonDpadRight, eGamepad_ButtonCount }; builtin managed struct Gamepad{ /// get number of connected joysticks import static int GetCount(); /// attempt to open a gamepad, returns null if it's not a valid gamepad import static Gamepad* Open(int index); /// if joystick is a valid gamepad, get it's name import static String GetName(int index); /// gamepad name import readonly attribute String Name; /// checks if gamepad is really connected import bool IsConnected(); /// checks if a gamepad button is pressed, including dpad. import bool IsButtonDown(eGamepad_Button button); /// get gamepad axis or trigger, trigger only has positive values. import int GetAxis(eGamepad_Axis axis); }; ```

You can just copy this header in an empty script header in a regular Editor if you just wish to rebuild the Engine.

References

• https://love2d.org/wiki/love.joystick
• https://github.com/love2d/love/tree/master/src/modules/joystick
• https://docs.unity3d.com/Packages/com.unity.inputsystem@1.0/api/UnityEngine.InputSystem.Gamepad.html
• https://github.com/gabomdq/SDL_GameControllerDB
• https://docs.godotengine.org/en/stable/tutorials/inputs/controllers_gamepads_joysticks.html
• https://manual.yoyogames.com/GameMaker_Language/GML_Reference/Game_Input/GamePad_Input/Gamepad_Input.htm

Notes

• Which branch? Well on one hand ags4 is the branch for new features, oth I can imagine it on ags3 one day replacing agsjoy...
• if you don't want to rebuild the editor to test script APIs, you can just place this header in an empty script header, and just build the engine
• one probably never want to save anything about the joystick itself since the game may be loaded with no device later, or the device may have changed index, there's still a mystery in how to communicate this to developer using this API
• mapping to physical device, say button 3 is Triangle in PS4 gamepad, but it's Y in Xbox One gamepad, and then Square in PS5, just hypothetically, if the developer wants to present an icon with the button name, do we want to have this physical mapping solved in the Engine or rely on it being scripted?
• mapping buttons to actions is probably a separate problem that could be addressed later, perhaps with some input struct which could then be linked to functions, but this is a different problem that is not going to be solved here.
• which aspects of the engine need to be adapted? Do I need to provide a way for characters to walk by pressing arrows instead of pathfinding?

Also, thanks for the wiki entry, it helped immensely figuring out what was needed to do in my code.

Post in the forums https://www.adventuregamestudio.co.uk/forums/index.php?topic=59945.0

[ivan-mogilko]

I would like to proppose supporting SDL2 GameController API first and later figure out joystick

Could you elaborate, what is the difference between gamepad api and joystick api? Do we need to have both? I do not have much experience with these kinds of devices.

[ericoporto]

Could you elaborate, what is the difference between gamepad api and joystick api?

Joystick API is designed to support any joystick, but joystick comes in different hardware flavors, different number of buttons, sticks and all, so it's responsibility of the game developer to figure how the user joystick maps to his game (or giving tools to the user to do so).

The gamepad controller API in SDL2 assumes for most of the needs for everything, you only need a Xbox 360 gamepad, and you can try to map other joysticks to this gamepad. There are Database of joysticks mapping to this type of controller and it's possible to use an external software to provide the mapping (like Steam), integration is provided through environment variables. This API simplifies the gamedev job.

Do we need to have both?

Unity3D, Löve and Game Maker supports both.

Porting to consoles should be easier using the gamepad controller API.

Technically, I think if you have only the full joystick API the gamepad can be implemented in script.

There's (or at least was in SDL 2.0.9 and before, I haven't checked again recently) a small catch, the joystick API may return some small differences not only depending on the device but on the system you are (win/Linux/Mobile/...) and the controller API also resolves these small differences under the hood - the only difference I ever pickup in this case was the trigger axis were reversed depending on win or Linux.

These differences can be smooth out by asking the player to do the mapping when using strictly the joystick API.

[ericoporto]

Hey, I am looking into skipping speech and display when a button is pressed. Looking into sys_events and also adding a new possible Wait function, not sure is needed, but it looks like something that makes sense ? The problem is basically there can be an any number of Gamepads and also there can be none, so my idea would be to respond to any button press... Any other ideas design wise to interact with Speech and Display? I also noticed mouse.Click () doesn't skip speech when clicking with the mouse could skip - but using SimulateMouseClick from plugin API does skip.

[ivan-mogilko]

adding a new possible Wait function, not sure is needed, but it looks like something that makes sense

I replied to something like this a while ago, and imho the Wait* functions are done wrong at the moment, as there's practically a Wait function for every possible device and combination of these, and adding more devices will increase number of functions. I would perhaps propose replacing these with a single WaitForInput, but in case there's a need to differentiate, then provide a flag set to be passed into this function, e.g. WaitForInput(eInputKeyboard | eInputMouse | eInputGamepad) etc.

I also noticed mouse.Click () doesn't skip speech when clicking with the mouse could skip - but using SimulateMouseClick from plugin API does skip.

I've replied to a that on forums a while ago. Mouse.Click uses exactly same method to trigger a click as SimulateMouseClick, they are equivalent in that. The question is, what are you trying to skip with it and where do you call it (which script function). Speech can only be skipped from rep-exec-always, but Display cannot be skipped like that, because rep-exec-always does not run during Display at all. This makes it impossible to skip Display messages from script.

EDIT:

Any other ideas design wise to interact with Speech and Display

I guess you'd have to add a new constant to the Speech.SkipStyle. Don't remember if these are flags atm: they should be in the engine, but may not be in the script. If they are not then it's best to make them, otherwise the number of possible values will increase with number of combinations.

EDIT 2: I have not looked into how this works atm, but how are you planning to handle gamepad button presses? Will there be a new callback for that, similar to on_key_press?

Alternatively, there could be new eKey codes for gamepad/joystick buttons. In such case - 1) you may use on_key_press to receive gamepad presses; 2) any property that stores a eKey code may also store a gamepad button code (for skipping speech too, and so on).

[ericoporto]

Just a minor curiosity: https://github.com/Unity-Technologies/UnityCsReference/blob/master/Runtime/Export/Input/KeyCode.cs

The unity keycodes includes buttons from multiple joysticks (joystick 1 to 8!), besides the keyboard and mouse buttons. I thought this was quite interesting.

[ivan-mogilko]

For the record, as I commented this in the PR thread some time earlier: a proposal for the engine to automatically handle connected and disconnected joysticks, and provide a list of opened joysticks in the script API. This is contrary to the classic agsjoy plugin API, where user had to call Joystick.Open themselves.

I believe this change is already implemented in the last version of PR #1759.

---

Something I wanted to add is - how to deal with joysticks internally in the engine, how to ensure that invalid joystick objects, saved in user vars, don't break the game script.

In AGS there's already a bunch of things that act in a following way: there are 2 objects per such thing, an internal engine's object, and a "script object", working purely as a "index" of the internal object. This script object may contain simply the ID of the internal object. This ID may be anything really, whatever necessary to map to the real object, such as integer index, or else. As examples, you may check out Overlays, Cameras and Viewports: all of these may be created at runtime, and so are Joystick objects (which may be connected at runtime).

The design is following (consider this a draft):

• Engine has a collection of JoystickImpl objects (e.g. in a vector, or any other container, whichever is convenient). These objects has real Joystick data as necessary, and a managed script handle to store a handle to the script object.
• ScriptJoystick struct is a descendant of a AGSCCDynamicObject class. It contains only joystick's ID (an integer, or else).
• Whenever game script asks for a Joystick* of certain index, you do this:
  • if the JoystickImpl does not have a handle stored (==0), then create and register a ScriptJoystick object using ccRegisterManagedObject and adding one internal ref to it to avoid its disposal with ccAddObjectReference. ALTERNATIVELY: you may create and register a script object right after creating a new JoystickImpl; the difference will be minimal.
  • if the JoystickImplobject already has a handle stored (>0), then use ccGetObjectAddressFromHandle to get the ScriptJoystick object's address, and return it.
• Whenever game script does anything with Joystick*, calls any of its member functions, the real internal JoystickImpl will be addressed using an ID found inside the ScriptJoystick.
• Why do we need a handle inside a JoystickImpl though? First - this way we don't duplicate ScriptJoysticks when having to return same one. Second - because we may use it to invalidate the ScriptJoystick whenever real joystick is deleted (disconnected).
• Upon disconnection and erasure of a JoystickImpl object, you do following:
  • get a respective ScriptJoystick using a handle (ccGetObjectAddressFromHandle);
  • invalidate this ScriptJoystick by resetting the stored ID to some invalid value (e.g. -1 for an integer index);
  • subref it (remove the internal reference) with ccReleaseObjectReference.
  • What happens now is that although the internal object is removed, the ScriptJoystick will remain so long as there's at least 1 script reference to it (if it's stored anywhere in user var). The scripts won't crash, and won't produce unexpected results, as you may always skip doing requested action for invalidated ScriptJoysticks.
  • This allows user script to test if joystick is valid (still open); ScriptJoystick will simply check if ID is valid (e.g. >0).

---

I think the remaining problem is a restoration of game saves...

Upon a quick thought, maybe we should not try to serialize joysticks at all, similar to how we don't try to restore File* script objects. Instead we just invalidate all ScriptJoysticks when saving, and/or when restoring. This will force the user to reassign any global Joystick* variable in "after restore" event.

[ivan-mogilko]

1759 is merged, is it enough to close this ticket, @ericoporto ?

[ericoporto]

Yes, I need to write the TO-DOs, I think I will write them in a single new ticket as task list - the mapping stuff, the IDs stuff, the events, ... I think they may be related.