Refactor game controller system

[Ezward]

Problem Statement

Our current system hard-codes behavior in the joystick parts. This has unwanted side-effects;

• changes in linux drivers break our button/axis -> behavior mappings.
• users are stuck with the way we have mapped buttons/axis to behaviors; they cannot change which buttons do what (see Note 1 at bottom) and they cannot assign new behaviors.
• The current mappings assume the deep learning template. However we also have the path follow and computer vision templates and they have very different needs for the controller buttons/axes. In the near future we will make it easy for users to construct their own 'template'/vehicle and so they will need to decide how buttons should be mapped for their purposes.
• The code is complex, redundant and brittle. Game controllers are difficult to add or modify because the code does too much.

We know that XBox joystick has different mapping under different SBCs and even within different releases of the OS on a given SBC. We have split the joystick parts into two pieces;

• Joystick: Each kind of controller has a part subclassed from the Joystick() abstract class. This maps axis (analog) values and button presses (boolean 0 or 1 values) to names.
• JoystickController: Each kind of controller also has a part subclassed from the JoystickController() class. This calls functions and/or sets output values based on the changes in the axes and button states.

Together these two parts effectively map a button code to a function. The problem is that different drivers for the same controller (different across drive versions or OS versions of SBC versions) output different codes. So the 'X' button on an Xbox controller may be 0x05 on one driver and 0x23 on another. The breaks everything because the mapping of code to name in the Joystick() may be completely missing or may map to the wrong actually axis or button. The canonical example is the difference between the Xbox controller on RaspberryPi and Jetson Nano.

The other issue is that every JoystickController subclass must implement the behaviors itself.

1. this is super redundant.
2. this makes it very hard to modify what the axes and buttons do
3. this breaks if the driver changes the output codes and see (2).
4. We now have not just the deep learning template; we have the path follow and computer vision templates and they have different needs for how the controller should behave.

Goals:

• Make it easy to handle the fact that the same game controller may publish buttons and axes using different codes on different OS or version of an OS or SBC.
• Make it very easy to remap the behaviors that run when a given button is pressed or joystick is moved. So make it easy, using configuration, to change which button toggles recording for instance.
• make it easy to create function mappings for different templates.
• make it easy for a user to remap the controller for their own purposes, including mapping to their own function.
• In the process greatly simplify how we support game controllers.

Proposed Direction:

Replace the hard-coded behavior we have in our controllers with an event system where the controller emits a event with a name and value when an axis/button changes, then an event handler (a part) can handle the event to implement the behavior.

• We could implement this as a single-stage mapping where low-level codes are mapped directly to high level event names. So we may map a button with code 0x05 to the 'toggle-recording' event.
• OR we could implement a two-level mapping where we map low-level controller codes to low-level names, then map low-level names to high level events. So we have a table that maps low-level button code 0x05 to 'button-x' and then we have a separate part that uses a lookup table to map 'button-x' to the 'toggle-recording'.

Using a two level mapping creates more understandable low-level button names, like 'button-x', We would have configuration that maps from the button code (like 0x0f) to this more understandable name. We would have predefined configuration for supported controllers, which we would use by default based on the controller type selected in configuration. We would also allow the user to create their own controller mapping (perhaps we make a version of joystick creator that just outputs such a map) and point to it in configuration.

So then we would map these low level names to high-level events in the second mapping. It may be possible here to have a single mapping from low-level to high level provided that controllers have orthogonal names for their buttons (so 'button-x' maps to 'toggle-recording' to handle Xbox and a second mapping of 'button-triangle' also maps to 'toggle-recording' to handle Sony). We will want a set of lookup tables that handle all of our current functions and controllers. We want separate tables for each template, since they have different functionality.

We would still be left with the issue that the button/axis names might differ between drivers. We can handle differences between drivers by including extra tables. So we might have 'xbox' lookup table to handle the xbox controller on Raspberry Pi and an 'xbox-jetson' lookup table to handle the same controller on the Jetson because the xbox driver on Jetson outputs a different set of button-#/axis-# names than it does on Raspberry Pi.

Beyond the mapping of low-level codes to high-level events (be that in a single-stage to two-stage system) we need a system that delivers the event to a part that implements the necessary behavior. So we would have a part that handles toggling the recording state and it would run when we get a 'toggle-recording' event.

• The way I do this in path_follow is that I treat the high-level event name as a latched run_condition; when the event is emitted it is simply added to the vehicle memory using it's name. Then there is a part that uses that name as it's run_condition; so it will run when the event name is added to the vehicle state. At the end of the vehicle loop the event will be removed from the vehicle memory; this makes it so it only fires for one iteration the vehicle loop. This system makes it easy to fix the ordering of event handling relative to other parts; it's just a normal vehicle part with a run_condition; it will run in the order it is added to the vehicle and only when the run condition exists.
• Alternatively we might add a true event pub/sub system. We have to think about the order that things run; a true pub-sub system might run things in different order and at different times.
• Perhaps we compromise and rather than create a full pub-sub system we enhance what we already have to use a 'subscribe=event-name' argument when adding a part; the part will run if an event is present with the name; all events are automatically cleared at the end of a vehicle loop. This is basically like what I did using the run_condition, but it would not overload the run_condition system and it might have clearer semantics. It would have the benefit of working like parts work now, including ordering guarantees. (This is the winner I think!)

Note 1: The user has some customization available to the them now; we have the JoystickWizard that creates a custom joystick part that maps low-level code to a low level name. So technically users can remap buttons/axes to existing functions; but they cannot assign new functions. This is also complex; the Joystick Wizard ends up generating a Python class that it writes the the root of the my_car folder. If the user changes their joystick preference to "custom", then add_user_controller() looks for my_joystick.py in the my_car folder and imports it dynamically. It would be much better if the wizard just wrote out a lookup table (a standard csv file) and the custom joystick always used that as it's lookup table. Then we would not have to had the weird code that dynamically load the 'my_joystick.py' part. The user could easily remap things by editing the file. We could also have the system read configuration files based on the controller names in the configuration. So we might have a 'controllers' folder and in that folder we have CSV files to map low level codes to button names

Tasks

• [ ] Create an event system Add a new type of state to handle one-shot events. This might simply be a dictionary we insert into memory, or it might be separate dictionary. Parts can add named events with values to the event dictionary. The event dictionary is cleared the next time the part that inserted it runs. Note that behavior; this is to guarantee that an event goes through the event loop one full time. For instance, we may have parts that run before the joystick controller part that want to handle joystick events; if we clear the events at the end of a loop, they will never see the events. Alternatively we might want to clear the event when it is handled; when a part pulls it from the event dictionary. It would then be incumbent on the part to set some other state, perhaps a run_condition, so that other parts could respond. In this case we probably would still want to clear the event after it has been all the way through the loop. Perhaps we mark each event at the loop start and those events that are marked will get automatically cleared on the next loop start. More generally, we could have the event have a counter that is supplied when the event is emitted (written to the dictionary); that counter would get tested at the start of the loop; when it is zero the event is removed, otherwise it is decremented. That would allow the event designer to specify how long the event lives. The default value would be 0, so only parts that run after the part that emitted the event will see it. See the Feb 18 comment for more details.

• [ ] Load mappings at startup. Modify the templates to load the correct CSV files for the mappings from the controllers/ folder based on the joystick configuration and to pass them to the Joystick() to handle button name mappings and an instance of the event mapping part to handle mapping button names to high level event names.

• [ ] modify the joystick system to produce button events We modify the Joystick() part to take a lookup table in it's constructor. That table is loaded from a csv file in the controllers/ folder based on joystick configuration. The joystick part uses that table to map controller codes to button/axis names. When a button is pushed or an axis is changed then the Joystick controller inserts an event into the event system that corresponds to the button name. See the Feb 18 comment for more details.

• [ ] Create a part that maps one event to another event. This part would take a mapping in it's constructor. For this discussion, this would be the map from button name to high level behavior name, like 'button-x' to 'toggle-recording'. See the Feb 18 comment for more details.

• [ ] modify templates so joystick-controllable behaviors are event driven parts We modify some functionality in the templates that is called directly be the JoystickController() part so that is triggered by events instead. So the part will have an run_event filter, much like the run_condition filter. This works in conjunction with the run_condition filter, so events are only processed if the run_condition is set. If the run_event filter passes, then the part runs, otherwise it does not run.

Related Issues

• Incorrect PS4 gamepad keycodes in "controller.py"
• No throttle control when using XBox One gamepad in local control
• Make RC controller 'pigpio_rc' button(s) programmable
• Logitech G27 steering wheel use different axe (0 in stead of 2) for steering -> should be configerable (Gamepad API)
• Since moving from v 3.1.5 to v 4.1.0 the PS4 gamepad no longer functions per the docs
• Record behavior dataset without joystick
• xbox controller and instructions don't work for Jetson Nano

[DocGarbanzo]

@Ezward - this is a very thorough and comprehensive description of the problem and the possible solutions. I had to prototype a solution already in order to get the generic vehicle template approach to work, where we need to rely on all parts to communicate through the run() or run_threaded().

Without spending too much time on a strategic solution, I directly implemented a solution where the controller would simply spit out a map of the buttons that are pressed or released. This map is returned from the run() or run_threaded method and then one only needs a new generic part, say ButtonInterpreter which translates the button map into boolean variables. These variables simply get added to the vehicle memory and can then be used as inputs or run_condition in other parts. Obviously this requires the updating of some Part's run() methods. For example, the TubWriter would need an additional bool argument to check if it should remove the last n records and similar. If we think we might create too many bool arguments, then we would need to pass all of them in another dictionary and share that dictionary as input parameter for all parts that allow for some 'special' controller interaction.

Note, I don't think these variables should get cleared at the end of the vehicle loop. The reason is that sometimes the parts topology in the car can become complicated, e.g. for some reason you would want the AI pilot to run before your controller and for some other reason you want to run it afterwards, usually because you have some cyclic dependencies, so part one's (p1) input depends on part two's (p2) output and p2's input depends on p1's output. You simply put these part in the order in which you want to reduce latency, say p1 before p2 and then p1 will pick up p2's output in the next vehicle loop. For buttons to delete records or switch some behaviour on-the-fly, a one-loop latency is absolutely no problem, but if we were to clear them at the end of the loop we would miss out on using the output like mentioned above.

As you say, the mapping of physical buttons to button names might be platform dependent, so it should be taken care of in the controller itself, I would think. The mapping of buttons, say X to toggle recording should be happening in the vehicle design, because it depends on which vehicle or template you run and which controller you own and what your preferences are. I can also imagine to build an enhanced ButtonInterpreter which would allow some debouncing, or latching or distinguish between short and long presses. Something I have started using on my RC control, because it has only a single button (it's 3 channel).

[Ezward]

@DocGarbanzo can you give me a link to the branch where you are developing generic vehicle template approach?

[Ezward]

Looking at current Joystick and desired kinds of events, I think the layers could be

• Joystick tracks low level device events (basically the current Joystick class). These are retrieved, one at a time, by calling the poll() method. Poll() blocks until an event is available, so it should be called in a thread. Note that this is NOT a part; it is more like a device driver

  • button_change: (timestamp, button_id, button_value)
    • timestamp is milliseconds
    • button_id is string like, "button_1"
    • button_value is 1 for down and 0 for up
  • axis_change: (timestamp, axis_id, axis_value)
    • timestamp is milliseconds
    • axis_id is string like, "button_1"
    • axis_value is -32767 to 32767

• ControllerEvents - A threaded part that turns low level button_change events into clicks. The update() continually calls the Joystick.poll() in a thread. It maintains a two lists; a list of button events and a list of axis events.

  • button_change: these values are simply a pass through of that Joystick.poll() returns.
  • button_click: represents a button_down followed by button_up
  • timestamp is milliseconds of the button_up
  • button_id is string, like "button_1"
  • fast_click is a count of sequential fast_clicks, the number of clicks in sequence where the time between clicks is at or below the quick_threshold. Basically this indicates double-click, triple-click, etc. without having to have explicit separate events.
  • long_click is a count of sequential long clicks; the number of clicks in sequence whose duration is at or above the long_threshold AND whose time between long clicks in at or below the quick threshold.
  • axis_change: these values are simply a pass through of what Joystick.poll() returns.
  • The run_threaded() method returns the button and axis event lists and then clears the internal queue to start new lists.

• AxisButtonEvents - A standard (non-threaded) part that takes as inputs the button and axis lists emitted by the ControllerEvents part and then expands the events into individual events for each button/axis. This simply takes the most recent button click for a button and the most recent axis change for an axis and publishes them using the name of the button/axis. For instance, "event/click/button_1" the the most recent button_1 click event. The data include the timestamp, button_id, fast_click count and long_click count. For an axis it might be "event/axis/axis_3" and the data would be the timestamp, axis_id and axis_value.

It is possible for parts that implement behavior to use the outputs of the ControllerEvents part or the AxisButtonEvents part directly. However, it may be better to have another layer that turns these device input events into high level events. For instance, rather than a part that looks in the list of button events for a click event on "button_1" to toggle recording, it would be better to have a part that maps a "event/click/button_1" event to a "toggle-recording" event (button->behavior-event) and have a separate part that handles toggling the recording; that separates those concerns and makes everything more re-usable. In that way we can have other parts toggle recording, not just those that listen for button events.

I think we don't want to have to have one part per button->behavior-event translation. Ideally we can have one part that uses data to translate any number of button/axis events into high level behavior events. We could use the order of the list of inputs and outputs as the association; so the first input would map to the first output, etc. We could get the list of inputs and outputs from configuration. For example;

V.add(ControllerEvents(Joystick("/dev/input/js0")),
      outputs=["buttons", "axes"],
      threaded=True)

# AxisButtonEvents outputs events directly to vehicle memory
# so it can support a variable number of events
V.add(AxisButtonEvents(V.mem),
      inputs=["buttons", "axes"])

V.add(BehaviorEvents(), 
      inputs=["event/click/button_1", "event/axis/axis_3", "event/axis/axis_5"], 
      outputs=["toggle-recording", "user/throttle", "user/steering"])

V.add(ToggleRecording(), 
      inputs=["recording"], 
      outputs=["recording"], 
      subscribe="toggle-recording")

Note in the above example that AxisButtonEvents take V.mem as a constructor argument with the notion that it would have a variable number of outputs, so it can't declare them ahead of time.

Note that BehaviorEvents' inputs are the only thing that needs to change to create a custom mapping (and to handle driver and OS differences). For that purpose we can have a lookup that takes the controller and os as arguments, looks up the button-event->behavior-event as a Python dictionary, then turns that dictionary into a list of inputs and outputs.

We include a version of the dictionary called "custom". If the user wants a custom mapping then they edit the custom dictionary and choose "custom" as their controller type.

• [ ] TODO: how do we handle double-click events or double-long click events in this flow? More importantly, if we want to handle both a click and double-click from the same button then they need to have different event names and more importantly we need to wait for the fast-click threshold to expire before we issue the click, so we don't emit both a click and a double-click.
• [ ] Same with double-long click; also do we want to distinguish short-long, long-short and long-long? Or any double-click that ends in a long click is a double-long click?

[DocGarbanzo]

@DocGarbanzo can you give me a link to the branch where you are developing generic vehicle template approach?

@Ezward - this is here: https://github.com/DocGarbanzo/donkeycar/tree/part_factory

[DocGarbanzo]

I have one more comment to the Event System. I think if we add one-shot events to the vehicle memory, these events must be cleared by the part that generates those. Say a one-click event would be registered, only if the button is pressed through at least 2 vehicle loops and or a double click requires at at least 2 vehicle loops between the single clicks. Such an event would be issued when registered from the controller and would be cleared in the next loop. This will allow the parts that react to that event to be placed either before or after the controller in the vehicle loop. I think it is important to have that flexibility, and we already have parts which are issuing data into the memory that will be picked up by other parts only in the next loop, for example the current tub index is read by the webcontroller only in the next iteration. The causal topology structure in the car is not a DAG.

For your questions above, how to correctly recognise short or long or double clicks, I am already using some of this for my channel 3 controller, so I have some code for it. The controller part will issue a different message when pressed long (for me this issues a Ctrl-C like event to stop the vehicle) vs pressing short (which will clear the last 1s of data from the tub).

[Ezward]

@DocGarbanzo can you share your code with me?

[Ezward]

Proof of Concept

A proof of concept is available on branch 1097-refactor-game-controller-system. I've also opened a draft pull request to make it easier to see the changes and to comment on the code; that PR is not intended to ever be merged. Please address comments regarding the approach to replacing the legacy joystick code in this issue rather than the PR. Please make comments that are specific to the code in the PR.

This lengthy comment is a discussion of the approach taken to refactoring the joystick code as embodied in the branch.

Running the POC

• These instructions assume the developer installation has been done.
• Change the working directory to the root of the donkeycar project folder and check out the branch 1097-refactor-game-controller-system
• Activate the donkeycar environment, source env/bin/activate
• to run the vehicle loop python3 donkeycar/parts/controller_events.py
• to run the simulated loop with event logging: python3 donkeycar/parts/controller_events.py --vehicle=false

One-shot Events

The current Donkeycar game controller code tightly binds many behaviors to specific button/axes, like toggling the pilot mode with a button press or setting throttle with an joystick (axis) movement. That legacy design is a problem because we can't remap buttons/axes to different behaviors. That is a drag for two reasons; first it makes supporting new templates or new template behavior more difficult and second it makes supporting Dirk's declarative vehicle assembly basically impossible. This POC shows how we can eliminate the JoystickController-based classes completely and instead generate input events that can be handled by standard Donkeycar parts. Once we can handle button presses and axis movements with standard parts, then we are open to remapping template behaviors to any input control that we want. Further, because these behaviors are standard Donkeycar parts, they can be assembled using Dirk's declarative syntax.

The first thing we need to do is surface input control changes as one-shot events. The POC implements a one-shot event as values in memory that exist for a single complete pass through the vehicle loop and are then removed. Contrast this to 'normal' memory state, which is persistent for the entire life of the vehicle loop. We want events to be one-shot so that they can trigger code once in the loop when they are emitted and not over and over again on subsequent passes through the loop.

In addition to their one-shot nature, button and axis events must be named in a regular, predictable way so that using them as inputs or run_conditions is easy.

Refactoring Joystick

We already have a set of classes that can read controller inputs and rename them to something predictable (to the name of the buttons/axes on the game controller that is configured). That is the set of classes in controller.py based on the Joystick class. The Joystick class is specifically to read input events from the linux /dev/input/js0 device. There are other classes in controller.py that implement other kinds of controllers, for instance the RCReceiver class reads buttons and steering/throttle from an RC controller. These classes don't all conform to a uniform specification because there was no such specification. The first part of the POC is to create that specification and refactor the code, where necessary, to use it. This begins by creating a standard input controller api, AbstractInputController, that input controllers must implement if they want to participate in the input event system.

AbstractInputController: an abstract class that input controllers must be derived from. The most important method is poll(), which returns a tuple which can contain one button state change and one axis state change. poll() must be called continuously to retrieve input changes otherwise device driver buffering can cause problems. This is the method called by InputControllerEvents() in its threaded update() loop so that input changes are retrieved in the background.

LinuxGameController: an implementation of AbstractInputController for the linux /dev/input/js0 devices. The constructor can take a dictionary that maps from device-driver button/axis numbers to human readable button and axis names. This is how we map to native game controller names for each make of game controller, like button 0x133 -> 'X' or 0x13b -> 'start' and axis 0x03 -> 'right_stick_horz' for the Logitech F710. So the LinuxGameController.poll() method will emit state changes with the mapped button and axis names. Any button or axis that is not mapped will get a default name based on the device driver control number; for instance the prior button/axis examples would be 'button(0x133)', 'button(0x13b)', 'axis(0x03)' if no mapping is provided for them.

LogitechJoystick: an implementation of LinuxGameController that provides a button/axis mapping for the Logitech F710 game controller. This provides an example of how we would implement that mapping for the set of game controllers that we want to support directly.

If you look at LogitechJoystick you will see that it is verbatum the legacy joystick controller at parts/controller.py#L694. So the refactor at this point is pretty much much a very minor update to what we already have in order to read input control changes from the linux device driver; it really just formalizes the AbstractInputController interface so we can implement other kinds of input controllers that can work with the new input event system.

Button and Axis events

The much more important change reflected in the POC is in how we use input control changes to trigger code; for instance how we toggle through the pilot modes when a controller button is pressed. The legacy code includes a set of classes based on JoystickController that tightly bind an input control to a specific behavior for each make of game controller. The POC has examples of how to refactor the hard-coded behaviors in JoystickController into individual donkeycar parts so they can be triggered by the new event system.

InputControllerEvents: a class that reads input control changes from an AbstractInputController by continuously calling its poll() method in a thread and turns those control changes into one-shot axis and button events. The axis and button events are written to the vehicle memory as outputs so they can subsequently be used as inputs or run_conditions by any Donkeycar part. Importantly, the axis/button events that were output to vehicle memory exist for one pass through the vehicle loop are then removed from vehicle memory on the next pass through the vehicle loop. This is what makes them 'events'; they do not persist indefinitely in memory, but they do exist long enough for all parts in the loop to have a chance to use them. In addition to these one-shot events, InputControllerEvents also outputs the control's state as a persistent memory value when a control's state changes. This allows the state of one or more controls to be used as inputs to a part; for instance, a part can take the state of a button (pressed or released) as an input and do something differently while it is pressed.

It is worth noting that parts that are added to the loop after the InputControllerEvents part is added will see the input events in the same pass through the vehicle loop as when the events were generated. Parts that are added before the InputControllerEvents part is added will not see the event in memory until the next pass through the loop (so 50ms later assuming the default loop rate of 20hz). For this reason we should move the addition of the InputControllerEvents to the very top of the vehicle loop so all parts see the event is the same loop iteration that in which the event was emitted.

The nice thing about adding events to memory is that they can be used by other parts as inputs and run_conditions. So the remaining part of the POC is to show how we can refactor the hard-coded behaviors from the JoystickController based parts into individual Donkeycar parts that can be triggered to run using input events. The first example is TogglePilotMode, which refactors the code from JoystickController.toggle_mode() into a donkeycar part.

TogglePilotMode: a Donkeycar part that increments the pilot mode once each time its run() method is called. The pilot mode is incremented through this sequence; user -> local_angle -> local -> user -> etc. Since the run() method will increment the mode each time it is called, it should only be called when the mode should be changed; so the part should be added with a run_condition that will trigger it to run when the mode should be incremented. The most common time we would want that to happen is when an input controller button is pressed. For instance, if pressing 'button1' toggles the pilot mode, then the run condition should be run_condition="/event/button/button1/press" or if a double-click of the 'Y' toggles the pilot mode, then the run condition should be run_condition="/event/button/Y/click/2". Since the TogglePilotMode part is supposed to change the pilot mode, it must take in the current value as an input and output the new incremented value. So adding the part would look something like:

    V.add(TogglePilotMode(), inputs=['user/mode'], outputs=['user/mode'], run_condition='/event/button/button1/press')

The run_condition combined with the one-shot event semantics enforced by InputControllerEvents, where the event exists for exactly one full pass through the vehicle loop, guarantees that TogglePilotMode.run() will be execute only once when the associated button event happens. A similar pattern would be used to refactor the logic that toggles recording on and off.

NOTE: there is already a ToggleRecording part, but it is NOT what we want; it should be eliminated and replaced with individual parts that are triggered with events.

Button and Axis state

Sometimes we want to something happen while a button is pressed or not pressed. Or we might like to use one button as a modifier to another button's event, like a shift key. For a button we can do this using the persistent state of the control as an input. For buttons this is a key, like /button/X, which has the value 1 when that button (in this case the 'X' button) is pressed and 0 when it is not pressed. The StopVehicle part in the POC uses this technique to create a 'gesture' for quitting the vehicle using a game controller. The part is added with run_condition="/event/button/Y/click/2" so it only runs on a double-click of the Y button. It also takes the state of the X button as an input, inputs=["\button\X"] so it will only quit while the X button is pressed. The run_condition and inputs combine to make it so that the part will only quit when a 'gesture' is made with the controller; double-click the Y button while holding down the X button. That is very nice because it is easy to do, but hard to do by accident.

StopVehicle: a donkeycar part that quits the vehicle loop when it runs and a specific key is held down. To avoid accidentally quitting, a key event can be used as the run_condition and a key state can be used as an input to indicate if the part should quit. For instance, if the run_condition is a double-click event from the 'B' key and the input state is from the 'X' key, then the part will only quick when 'B' is double-clicked while holding down the 'X' button.

In the case of an axis control, the state is just the current value of the axis. We often want to use the value of the axis as a continuous input so our part does not need maintain axis state itself by listening to axis events. For axes the state's memory key is named like /axis/right_stick_horz; it has the value of the axis as a float in the range -1 to 1. The UserThrottle and UserSteering parts in the POC take the value of an axis and throttle and steering values respectively as inputs and outputs.

UserThrottle and UserSteering parts really only need to run when the underlying axis actually changes. To do this, they use the axis event as both a run condition and as an input so that the part will only run when the axis changes and the new value will be provided as an input. This technique to minimizes how often the part is executed

UserThrottle: a donkeycar part that takes in an axis value and outputs a throttle value. The part only needs to run when the associated axis emits an axis event; this can be done by using the axis event as both the input and the run_condition; the part will only run when the axis changes and the new value will be the input value.

UserSteering: a donkeycar part that takes in an axis value and outputs a steering value. The part only needs to run when the associated axis emits an axis event; this can be done by using the axis event as both the input and the run_condition; the part will only run when the axis changes and the new value will be the input value.

So the input state and event semantics give quite a lot of flexibility while still leaving most cases simple.

Event naming and ordering

The __main__ in the POC implements two modes. The default mode simulates a template by constructing and starting a Vehicle loop with the POC parts. In that mode it outputs changes to the vehicle state, like changes in throttle and steering. The POC can also be run in a mode that that just runs the joystick part in a loop and outputs detailed information on what events are emitted. In this second mode, the format of the button and axis events and the order they are emitted can be seen

For example, here are two separate clicks of the 'Y' button. The clicks are separated by enough time that each is considered a single click indicated by the suffix '/click/1'. Note how each click represents a completed press->release sequence; the 'press' and 'release' events are emitted just before the 'click' event.

'/event/button/Y/press' = '1716844802.4654794'
'/event/button/Y/release' = '1716844802.6975334'
'/event/button/Y/click/1' = '1716844802.6975334'
'/event/button/Y/press' = '1716844803.0975556'
'/event/button/Y/release' = '1716844803.4575663'
'/event/button/Y/click/1' = '1716844803.4575663'

The value of each button event is the time at which is was received. Generally this is not a value that is needed; the InputControllerEvents part does the work of matching presses and releases and detecting and counting multiple clicks. However, if your part is interested a specific order that buttons were clicked, then this could be helpful.

Here are 3 fast-clicks of the 'Y' button. Because the clicks are close together they are emitted as a multiple-click sequence; the first click has the suffix '/click/1', the second '/click/2' and the third '/click/3'. The suffix makes each of these events distinct. So, by using the double-click event as the run_condition, a part can listen for a double-click and ignore the first click in the sequence.

'/event/button/Y/press' = '1716844882.8968825'
'/event/button/Y/release' = '1716844882.9808855'
'/event/button/Y/click/1' = '1716844882.9808855'
'/event/button/Y/press' = '1716844883.076876'
'/event/button/Y/release' = '1716844883.1569147'
'/event/button/Y/click/2' = '1716844883.1569147'
'/event/button/Y/press' = '1716844883.2648852'
'/event/button/Y/release' = '1716844883.3569455'
'/event/button/Y/click/3' = '1716844883.3569455'

It is worth noting that it is theoretically possible for events to be emitted so quickly that multiple press and release events might be processed in the threaded part before the event loop gets around to calling run_threaded() to get the events. In this case all of the associated clicks events will be correctly emitted, but only one press and one release event will be emitted. That is why the click count is included in the click event; so no clicks are actually lost.

Here is an example where the right-vertical joystick axis was pushed forward. The value of each axis event is a float in the range -1 to 1 when zero is center-neutral. Some axes only emit half of the range, like the shoulder triggers. Some axes only emit discrete values; for instance the dpad axes only emit -1, 0 and 1.

'/event/axis/right_stick_horz' = '-0.023773918881801814'
'/event/axis/right_stick_vert' = '-0.814264351329081'
'/event/axis/right_stick_horz' = '-0.047486800744651635'
'/event/axis/right_stick_vert' = '-1.0'
'/event/axis/right_stick_horz' = '-0.06329538865321818'
'/event/axis/right_stick_horz' = '-0.023773918881801814'
'/event/axis/right_stick_vert' = '-0.814264351329081'
'/event/axis/right_stick_vert' = '-6.103701895199438e-05'
'/event/axis/right_stick_horz' = '0.0'

Throttle = 6.103701895199438e-05

Look closely at the axes values above; because of jitter we also we changes in the horizontal axis while we change the vertical axis. If we pull out just the vertical axis the sequence is this;

'/event/axis/right_stick_vert' = '-0.814264351329081'
'/event/axis/right_stick_vert' = '-1.0'
'/event/axis/right_stick_vert' = '-0.814264351329081'
'/event/axis/right_stick_vert' = '-6.103701895199438e-05'

Throttle = 6.103701895199438e-05

The horizontal jitter looks like:

'/event/axis/right_stick_horz' = '-0.023773918881801814'
'/event/axis/right_stick_horz' = '-0.047486800744651635'
'/event/axis/right_stick_horz' = '-0.06329538865321818'
'/event/axis/right_stick_horz' = '-0.023773918881801814'
'/event/axis/right_stick_horz' = '0.0'

We may want to add a parameter to filter small changes like this from the event stream.

Behaviors

There is another layer that we probably want to add; the layer that translates button/axis events into higher order 'behavior' events. This is described in previous comments. Basically we want a way to provide configuration to easily map button/axis events to higher order behaviors. For instance, in the POC we use run_condition='/event/button/button1/press' to run the TogglePilotMode part. But what if the chosen game controller does not have a button named 'button1'? The user would need to go to the template and edit the line where the part is added to change it to the button they want to press on their particular brand of game controller.

We don't want user to have to directly edit templates for this kind of behavior change. So we will create a part that will translate from button/axis events to 'behavior' events. The part will take a dictionary there the key is a button/axis event key and the value is the translated key. When it runs, the part will iterate it's dictionary and if it finds a key in vehicle memory that matches, it will output the translated key to memory with the original key's value. This has to be sensitive to the one-shot nature of events; so it will remember the any translation that it did when it was run and on the next run, if the original key no longer exists in memory (so it was a one-shot event), then the translated key will be removed from memory, so it also acts as a one-shot event.

The event translation dictionary will be in configuration, so the user can just edit it in their myconfig.py to change the translation. We will provide a set of standard 'behavior' event keys as constants. The templates will be refactored so the behavior parts use this 'behavior' names rather than button/axis events or states.

We will probably want to have one dictionary for each of the directly supported game controllers that provides the default behavior mapping. The CONTROLLER_TYPE configuration would be used to choose the behavior mapping for the chosen controller. If the user wants to customize how their controller runs the template, they would just edit that translation dictionary.

Non-standard game controllers

To handle non-standard game controllers (a controller that maps to /dev/inputs/js0 but for which we don't have a directly supported driver based on the Joystick class), we will have a translation table that is selected when CONTROLLER_TYPE = 'custom'; the user will need to edit this table with the specific button/axis names that it's controller outputs. Those may just be the default names, like

For non-standard game controllers, the Joystick class will output default control names like 'button(0x13b)' and 'axis(0x03)'. We can allow the user to output more human readable names that correspond to the labels on their controller by providing a dictionary in myconfig.py. That dictionary, the 'custom control namedictionary, would be passed to theJoystickclass when it is constructed. By default the custom control name table is empty, but the user can edit it so that button/axis changes emitted by theJoystickclass have names that correspond to their game controller's labels, which in turn makes theInputControllerEvents` output states and events with those names. To support this, we may refactor the Joystick Wizard to output the dictionary that is used for custom control naming. We can remove the user interaction and output of the python class that maps to behavior; they can edit the behavior translation dictionary in myconfig to do that.

Non-standard input controllers

We have a few classes the implement a Joystick-like part for input devices that are not game controllers. For instance, the RcReceiver part that reads a wireless RC controller's buttons, throttle and steering controls. Those need to be refactored to derive from and implement the AbstractInputController interface. Once they do that then their control changes can work with InputControllerEvents and the rest of the event system.

Conclusion

The POC has enough in it to make me feel that this is a good direction, but we should discuss it before I put more work in. I've put a lot of detail into the explanation of the POC and the the event stream in order to elicit some discussion. I think what is implemented works well for the use cases of which I am aware. However there may be better alternatives or there may be use cases I don't know about that don't fit well into this scheme.

Constructive input is desired.