Terminal keyboard - with shortcuts and terminal text-selection

[Nordskog]

What is this?

This PR adds an in-game keyboard to the terminals, replacing the SteamVR overlay keyboard where applicable. In addition to basic keyboard functionality is also includes:

• Command shortcuts ( left side )
• Item ( some individual, some by category ) shortcuts ( right side )
• The ability to point at the terminal screen and copy-paste chunks of text to the input with a click.

See this video for a quick rundown of what it does.

We did shortly discuss what shortcuts should be included. The items and categories included should cover everything you might need, with some extras for symmetry.

The commands on the left are the ones often needed in a hurry. If a terminal has other commands, you can just hit commands and then select it on the terminal to copy-paste it.

I worked with BepInEx 6.0.0-be.553 because later versions of UnityExplorer don't support 363 that ships with the mod. 553 also doesn't require the ClassName(IntPtr value) : base(value) { } constructor anymore, which makes it a lot easier to share code with a Unity project for testing. It also provides a lot of warnings for things that would otherwise just give you an NPE with no information.

The code in this PR includes the old constructors and both compiles and works with 363.

Additions

Most of the additions are contained within Core/UI/Terminal.

Layout The keyboard itself is created using a canvas, utilizing the Vertical/HorizontalLayoutGroup and LayoutElement components. KeyboardDefinition contains a number of classes used to define and inflate a layout consisting of the above components and their content ( background, buttons, text ).

Core/UI/Terminal/KeyboardDefinition

• KeyboardLayout - Base interface for a inflatable layout definition ( Used by LinearLayout and KeyDefinition )
• KeyboardStyle - Shared style information ( key size, colors ). Also manages their shared materials.
• KeyDefinition- Inflatable definition of a single key: size, label, function, appearance
• LayoutParameters - Describes how a layout item should be sized and positioned.
• LinearLayout - Inflatable wrapper for a LayoutGroup.

The above classes are used to define the keyboard layout in TerminalKeyboardInterface.GetBottomKeyboardLayout(), GetLeftKeyboardLayout(), and GetRightKeyboardLayout()

Interaction The keyboard and terminal screen are interacted with using TerminalPointer attached to the MainController. Rather than fight with Unity's EventSystem, this is implemented using BoxColliders, with PhysicalButton and TerminalReader extending from MonoPointerEvent, containing the usual enter/exit/up/down/move events and a few others. Through trial and error the buttons have come to live on layer 2.

Core/UI/Terminal/Pointer

• MonoPointerEvent - Pointer enter/exit/up/down/move interface. Injected classes can't implement interfaces so here we are.
• PhysicalButton - A button extending from MonoPointerEvent. Handles its collider, background, and calling an OnClick()
• PointerEvent - Information provided to MonoPointerEvent ( button ) when interacted with. Only contains world hit position.
• PointerHistory - Stores the pointer position for a few frames to smooth it. Used by TerminalReader
• TerminalPointer - The pointer itself. Fires a ray and calls the relevant methods on anything inheriting from MonoPointerEvent

Core/UI/Terminal

• RoundedCubeBackground - Rounded square mesh generated at runtime, because I didn't want to add any assets.
• TerminalCanvas - Wrapper for a Canvas ready to have a KeyboardLayout inflated into it.
• TerminalKeyboardInterface - Manages everything. Attaches to a terminal, inflates keyboard layouts, receives events from keyboard buttons and sends text to terminal.
• TerminalReader - Interfaces with the Terminal to highlight and copy-paste text from it. Extends MonoPointerEvent. Is a child of TerminalKeyboardInterface.

Other additions

Core/VR_Input/Dummy_InputHandler - Works the same way WeaponRadialMenu does to trigger InputActions via InjectInput, just spun off into its own thing instead of making a mess of existing code. The InputActions triggered by WeaponRadialMenu should eventually be moved over here.

Modifications

Core/VR_Input/VRKeyboard

• Awake() modified to instantiate KeyboardTerminalInterface,
• FocusStateChanged() modified to attach the keyboard to a terminal. Only does this if we manage to get a reference to the terminal, and motion controllers are being used, otherwise it displays the Steam overlay keyboard instead.

Core/VR_Input/Controllers

• SetMainController() - Modified to parent pointer to MainController

  • SetupTerminalCanvasPointer() - Instantiates the pointer and immediately deactivates it. Called by Awake()
  • ToggleTerminalCanvasPointer() - Activates or deactivates the pointer gameobject.

  Core/PlayerBehaviours/VRPlayer

  • GetInteractingTerminal() - Gets the terminal instance the player is currently interacting with.
  • SetWieldedItemVisibility() - Used to hide whatever weapon/tool the player is wielding when using the keyboard. Seems a bit sketchy but has been working fine.

Detours/TerminalInputDetours - modified to grab text from TerminalKeyboardInterface

Injections/Input/InjectInput - Modified to also submit events to our new Dummy_InputHandler

Core/GTFO_VR_Plugin - Modified to inject all the new classes

Util/ExtensionMethods - Reimplemented TMP_TextUtilities.FindNearestCharacter() as FindNearestCharacterInWorldSpace() because it spits out nonsense with some terminals.

Notes

TerminalKeyboardInterface is created in and referenced in VRKeyboard. It is inflated then deactivated on launch. It is reactivated and attached to terminals as needed. It is added as a child of VR_Globals. It is not used for chat, or if motion controllers are disabled. If it can't find the terminal instance it will fall back to the steamVR overlay.

The TerminalPointer is attached to MainController, and swaps left/right accordingly. It roughly lines up with gun forward.

I mostly abuse Default/UI with unity_GUIZTestMode set to UnityEngine.Rendering.CompareFunction.Always to make everything render ontop of everything else. We don't know what kind of geometry might surround the terminal, so we must assume it may end up buried in geometry at some point.

A setting has been added under "inputs", "Use Terminal keyboard", defaulting to on.

Notable problems

TMP_TextUtilities.FindNearestCharacter() with a null camera worked fine the 20-something terminals I tested, before giving me horizontally offset values for TERMINAL_437 in R6.5DX. Luckily the individual TMP_CharacterInfo objects have the correct positional values, allowing us to implement our own alternate as FindNearestCharacterInWorldSpace()

When interacting with a terminal, you can get a reference to it via either PlayerAgent.Interaction.m_currentCamInteractor or PlayerAgent.Interaction.m_proximityInteracts, but only one of them will be populated the first time the terminal is interacted with. After entering, exiting, and then re-entering the terminal, both will be populated. 99% of terminals will use m_proximityInteracts, but the very first terminal in R6.5A1 will instead only have m_currentCamInteractor populated at first.

When you want to want to trigger an InputAction, it is generally enough to just return true for DoGetButtonDown() when it is queried in InjectInput, like how WeaponRadialMenu currently operates. This also works fine for the InputAction.TerminalExit ... until you go to the outside in DX. For the rest of the mission the game will query this action twice every frame, and you must return true for both or the action will not trigger. Other actions ( TerminalDel, TerminalUp/Down/Left/Right ) are also queried twice, but continue to function.

Final words

We have cleared DX using this, with the only problem popping up then being the TerminalExit issue above, which has been fixed. The number of only-applies-to-this-specific-terminal problems that popped up does warrant more testing, but I have done partial runs of A1, B1, and DX without further issues popping up. Mission and checkpoint restarts also seem fine.

Being new to C# and Unity I have likely committed many sins.

Have mercy.

[DSprtn]

There are no sins in game mods, only working code. If I were judged for GTFO VR's codebase I'd get the death penalty many times over. I went through the code briefly and it does look pretty nice, lots of room for painless extensibility.

Amazing work on the terminal (and an amazing PR writeup!)