There is a vast number of control devices (controllers) available which can be connected to the PC and we want them to be available to control LSL scripted devices within SL. To do so we need to provide the control input to scripts (and the output, when a particular controller supports sending the feedback signal to the user.)

Some controllers have common control signals, for example most joysticks have common buttons. Some controllers have additional signals, for example direction signals for VR joysticks. Also the same signals may differ in range for different controllers.

On top of that the PC may have more than one controller connected at same time, e.g. one can have a keyboard a mouse and two joysticks connected to one's PC. This feature request suggests a system which allows scripts to handle most controllers in a way that is easy, at least at the script side.

To make it easier for servers, we could simply rely the signals from controllers directly to the scripts by using the viewer, but then the scripts will have to deal with each signal accordingly and the LSL scripted device would need separate driver scripts for each different controller. Instead of this we use an other approach.

How would you like the feature to work?

Instead of connecting the controllers and scripts directly, an intermediate layer is created, a so-called "virtual controller". LSL scripts see and interact with the VC. This communication is the same for all kinds of real controllers. You can think of the VC as of a driver for the connected real controllers.

I'd leave open where the VC layer is implemented. It seems reasonable to make the viewer do this part, then the viewer will translate between the connected real controllers and the signals of the virtual controller, while the messages between the viewer and the server are already unified control messages. But it is one of possible realizations.

Modules

The variety of different controllers and control signals is addressed by modules. A module bundles different control signals to one entity. For example a console joystic has WASD, jump and selection buttons, a VR joystick has a pointer and a selection signal etc.

When a real controller is connected to the PC, one or multiple module is loaded to the virtual controller, which provide the control signals of the connected controller. The LSL scripts can list the modules loaded to the VC and access the modules, i.e. the provided control signals.

Similar controllers will load the same modules (when they have exactly the same control signals) and the LSL scripts will see them the same way. Controllers of different kind or with different control signals will load different (additional) modules so that the scripts can handle the (additional) control signals accordignly.

Keyboard and mouse control signals can be bundled to a "Legacy" control module, so that also the keyboard and mouse control can be covered by the same system. When possible the control signals should detect also the Shift and Alt keys and differ between their left and right variant, which the regular control event not does.

Scripts should be able to filter (activate and deactivate) control modules if needed; then the control signals of the deactivated modules are not transmitted to the script. Alternatively the filtering can be done on the signal level like it is implemented for the control event.

Signal event

When a control signal arrives from the virtual controller, it is provided to the script within a signal event. I'd prefer that all signal data is given within the event parameters (which seems to be faster) but I see no way to provide all different kinds of signal data this way without using lists and lists are slow. For this reason, the signal events must be probably bundled and the scripts will use different access function by using the event index.

For example:


integer llSignalController(integer index);

Provides the number of the connected real controller the signal with given index is received from, e.g. 0 for the legacy controller (keyboard and mouse), 1 and higher for connected joysticks. This way we can have a number of controller devices connected to the same PC and differ between them.


integer llSignalModule(integer index);

Provides the module number which sent the signal with given index. This number could be predefined for every known module (scripts not need to know it exactly however) or it can be a running number assigned to the module as soon the controller is connected. Then the function


list llSignalModuleInfo(integer module);

would provide necessary data for the given module, e.g. which signal data this module can send.

integer llSignalButtonControl(integer index);

Provides the state of buttons sent via signal with given index, for example which buttons are pressed, held or released, as a bitflag. Up to 32 buttons can be covered.


vector llSignalGearControl(integer index);

Provides the position of the hand gears. Up to 3 hand gears can be provided at once, each as x, y, z coordinate of the vector.


vector llSignalPositionControl(integer index);

Provides the pointer data.

etc.

Feedback signal

When a script wants to send a feedback signal (so the controller would generate the feedback to the user), it uses a function like this


llSignalFeedback(integer controller, integer module, list feedback);

This sends feedback data back to the user via given controller number and the module that supports sending the feedback data. If this module supports the operation, the script has tested via the llSignalModuleInfo function as shown above.

The data is sent via a list, which is sub-optimal but allows to send any data within one function. Again, this is just a suggestion.

Control profiles

When a controller device is connected to the PC, the real control signals must be mapped to the virtual control signals. This is done via control profiles. They define which module the controller device defines, which inputs and outputs it has and what is the actual signal range (for translation into unified signal range).

I was told (thanks Animats) that when a HID device is connected to the PC, it tells which inputs it has but not tells what data is provided. However the HID tells the device ID and the brand ID, and from which a predefined profile for this controller can be selected. Only when an unknown controller device is connected, a new control profile has to be created. The control profile can be saved within a special XML file, maybe even one for every supported controller.

When the control mapping is realized by the viewer, the viewer could have an addition floaters where the user can edit an existing control profile or add a new one for the connected control device. When mapping is made by the SL servers, then there must be a way created to add customized control profiles for any uncommon controller, the user may want to connect to their PC.

Why is this feature important to you? How would it benefit the community?

SL has a control event, and it is good for simple applications. But it is quite limited when the application becomes advanced. For example vehicle systems may require the user to press Fwd and Bwd key at same time, or Left and Right key at same time to issue special actions, because the control event can detect only a few control keys. Adding for example a combat system to the vehicle requires the driver to use gestures to fire while driving. This is inconvenient and requires using an extra listen.

There are numbers of control devices on the market with a wide range of control signals. If it was possible to receive this whole range by SL scripts, the usability of the inworld devices would move a big step forwards.

Original Jira Fields

| Field | Value | | ------------- | ------------- | | Issue | BUG-234354 | | Summary | Virtual control device as intermediate layer between the scripts and the controller | | Type | New Feature Request | | Priority | Unset | | Status | Accepted | | Resolution | Accepted | | Reporter | Jenna Felton (jenna.felton) | | Created at | 2023-08-30T02:17:19Z | | Updated at | 2023-08-30T18:54:22Z | ``` { 'Build Id': 'unset', 'Business Unit': ['Platform'], 'Date of First Response': '2023-08-29T22:42:33.232-0500', 'How would you like the feature to work?': 'a', 'ReOpened Count': 0.0, 'Severity': 'Unset', 'Target Viewer Version': 'viewer-development', 'Why is this feature important to you? How would it benefit the community?': 'b', } ```

Jenna Felton commented at 2023-08-30T02:34:14Z, updated at 2023-08-30T02:35:10Z

I seem to not find how to attach related issues, hence in a comment. I think this request is related to this one

https://jira.secondlife.com/browse/BUG-232926 (Two layers of Puppetry implementation)

animats commented at 2023-08-30T03:42:33Z

Instead of connecting the controllers and scripts directly, an intermediate layer is created, a so-called "virtual controller"

That makes sense. At the LSL level, this needs to be reasonably simple.

The control profile can be saved within a special XML file, maybe even one for every supported controller.

Sounds good. The USB Human Interface Device protocol. is well defined, fortunately. No need for custom drivers for every game pad and joystick. Just some parameters.

What should the viewer send to the server? That's tough. It's going to be a a frequent data item, as frequent as mouse position, probably 30Hz. So it has to be compact. The info probably can be packed into 32 or 64 bits. A new event type? Perhaps control_extended, with 2 or 3 32-bit values, one being the old control events? LSL programs which use this info shouldn't have to know exactly what device is out there, although they need some minimal info about which inputs are connected to some control device. Sending an LSL list is probably undesirable due to the overhead of constructing and accessing lists.

This pretty much has to go over UDP to get the needed low latency, and AgentUpdate has no space left. So this has the implications of a new UDP message. It's viewer to server, so it won't break old viewers.

No idea how this should interact with llTakeControls and with puppetry. Or who gets which controls. An avatar attachment might want some control so it can aim a weapon, while at the same time the avatar might be in a vehicle which wants the same control. The pew-pew crowd will want that, so they can shoot while driving. Some outreach to people into that sort of thing is indicated.

If this works well, it should increase SL's appeal to gamers. There are millions of them out there with enough hardware to run SL.

Jenna Felton commented at 2023-08-30T12:59:52Z, updated at 2023-08-30T13:12:13Z

Thank you Animats :)

Originally I thought the new control data would arrive within the control event. But then realized while writing that it is not sufficient to differ the connected controllers and to provide all kinds of data the controller may send. So that I end up with a new event and bundling of the sent data like for touch or sensor events. I hope it can be efficiently implemented.

Maybe it can't, than probably keyboard and mouse would remain in the regular control event while all other controllers receive their own event. In that case I hope the control event would understand more control keys, than just Ctrl and sometimes Shift. Defining up to 32 keys would be actually required for the new control event to recognize joysticks with a large number of buttons. Than these buttons can be mapped to the new keys while these keys can be also used to recognize left and right Shift, Alt and Ctrl keys for the regular control event.

Spidey Linden commented at 2023-08-30T18:25:18Z

Issue accepted. We have no estimate when it may be implemented. Please see future release notes for this fix.

Jenna Felton commented at 2023-08-30T18:49:41Z

To make it more complete and because the text is partially misleading, i list more functions


integer llSignalNumControllers();

returns the count of connected virtual controllers, where controller 0 is the legacy controller (keyboard + mouse), unless these are not covered as suggested in the comment above.


list llSignalControllerInfo(integer controller);

returns the information about the virtual controller with given number. Any data, the scripts have to know about the controller. When it is decided to not open any information apart of the number of modules, the result can be an integer number of modules loaded into this virtual controller. I tend to think it is better to give each module a running number when the controller is loaded and address the module by this number. When a module has any fixed ID, it can be returned via list as result of the function


list llSignalModuleInfo(integer module)

provides information about the specified control module, including the number of signals the module bundles.


list llSignalInfo(integer module, integer signal);

provides information about a special signal.

This way a script can detect how many and which (virtual) controllers are connected, which control modules are available and which control signals will be detected. Now a script can request controls and filter the control modules or signals to get only control signals the script is interested in.

One more note. I like the Animat' suggestion to map all control signals to a 32 bit integer. When it becomes possible, maybe this allows then to avoid any list and any bundling of control signals and the script will run faster, we'd have then an event like this


control_signal(integer controller, integer module, integer signal, integer data) {}

where data is the mapped control signal.

JIRAUSER339840 commented at 2023-08-30T18:51:36Z

I'm currently looking into exposing gamepad controllers directly to LSL and am willing to entertain some of the ideas in this Jira issue, but no guarantee to all of it. In particular, I think we would need to support multiple simultaneous controllers, and we'll need a way for the viewer to remap the indices of the the various buttons and axes that actually get sent to the server. The idea of using one virtual controller to collect all the inputs of the various connected devices is probably a good way to go (rather than sending multiple sets of buttons+axes data indexed by controller device id).

The format by which the LSL script receives the data hasn't been figured out yet. Most game controllers supply a list of buttons (on/off) and a list of "axes" (floating point value in the range [-1, 1] or sometimes [0, 1]), and the state of these inputs is what needs to be streamed from client to server. I will re-read the relevant suggestions in this issue when I get closer to that part of the project.

A way for scripts to send feedback to controllers is outside the scope of the minimal deliverable feature. Perhaps we will consider adding it after the client–>server data stream is working and delivered.

I'm skeptical about making the LSL script get module info and parsing the controller's feature set. Right now it seems more complicated than necessary but I promise to reconsider the idea later.

secondlife / jira-archive

[BUG-234354] Virtual control device as intermediate layer between the scripts and the controller #11295