No keypresses detected (seemingly random)

[Timtam]

Version

0.15.0

Operating system & version

Windows 11 (latest)

What you did

I'm currently starting to write a game in Bevy and am planning to use leafwing-input-manager for keyboard/gamepad/etc input. So far everything has worked fine, but sometimes when starting the game, no keystrokes get picked up. When closing the program down and starting it up again, it sometimes works, sometimes doesn't.

I do set up the ActionState conditionally though and don't leave it within the ECS all the time. I don't know if that is part of the issue or not. My code follows.

use bevy::prelude::{
    default, in_state, App, AppExtStates, Commands, DefaultPlugins, Entity, IntoSystemConfigs,
    IntoSystemSetConfigs, KeyCode, NextState, OnEnter, OnExit, Plugin, PluginGroup, Query, Reflect,
    ResMut, Startup, States, SystemSet, Update, Window, WindowPlugin, With,
};
use leafwing_input_manager::prelude::{
    ActionState, Actionlike, InputControlKind, InputManagerBundle, InputManagerPlugin, InputMap,
};

#[derive(SystemSet, Debug, Hash, PartialEq, Eq, Clone)]
pub enum MenuSet {
    PreInput,
    Input,
    PostInput,
    PreAction,
    Action,
    PostAction,
}

#[derive(Clone, Debug, Default, Eq, Hash, PartialEq, States)]
pub enum GameState {
    #[default]
    Uninitialized,
    Menu,
}

pub struct MenuPlugin;

impl Plugin for MenuPlugin {
    fn build(&self, app: &mut App) {
        app.add_plugins(InputManagerPlugin::<MenuAction>::default())
            .configure_sets(
                Update,
                (
                    MenuSet::PreInput,
                    MenuSet::Input,
                    MenuSet::PostInput,
                    MenuSet::PreAction,
                    MenuSet::Action,
                    MenuSet::PostAction,
                )
                    .chain(),
            )
            // only listen for input while a menu is active
            .add_systems(
                OnEnter(GameState::Menu),
                install_menu_keyhook.in_set(MenuSet::PreInput),
            )
            // handle keyboard input
            .add_systems(
                Update,
                process_menu_keyhook
                    .run_if(in_state(GameState::Menu))
                    .in_set(MenuSet::Input),
            )
            .add_systems(OnExit(GameState::Menu), uninstall_menu_keyhook);
    }
}

// possible actions in menus
#[derive(Copy, Clone, Debug, PartialEq, Eq, Reflect, Hash)]
enum MenuAction {
    Previous,
    Next,
    Click,
}

impl Actionlike for MenuAction {
    fn input_control_kind(&self) -> InputControlKind {
        InputControlKind::Button
    }
}

impl MenuAction {
    fn default_input_map() -> InputMap<Self> {
        let mut im = InputMap::default();
        im.insert(Self::Previous, KeyCode::ArrowUp);
        im.insert(Self::Next, KeyCode::ArrowDown);
        im.insert(Self::Click, KeyCode::Enter);
        im
    }
}

fn install_menu_keyhook(
    mut commands: Commands,
    query: Query<Entity, With<ActionState<MenuAction>>>,
) {
    if query.get_single().is_err() {
        commands.spawn(InputManagerBundle::<MenuAction>::with_map(
            MenuAction::default_input_map(),
        ));
    }
}

fn uninstall_menu_keyhook(
    mut commands: Commands,
    query: Query<Entity, With<ActionState<MenuAction>>>,
) {
    if let Ok(e) = query.get_single() {
        commands.entity(e).despawn();
    }
}

fn process_menu_keyhook(q_actions: Query<&ActionState<MenuAction>>) {
    if let Some(action) = q_actions.get_single().ok() {
        if action.just_pressed(&MenuAction::Previous) {
            println!("previous");
        } else if action.pressed(&MenuAction::Next) {
            println!("next");
        } else if action.just_pressed(&MenuAction::Click) {
            println!("clicked");
        }
    }
}

fn main() {
    let mut app = App::new();

    app.add_plugins((
        DefaultPlugins.set(WindowPlugin {
            primary_window: Some(Window {
                title: "Test app".into(),
                ..default()
            }),
            ..default()
        }),
        MenuPlugin,
    ))
    .init_state::<GameState>()
    .add_systems(Startup, setup);

    app.run();
}

fn setup(mut ns: ResMut<NextState<GameState>>) {
    ns.set(GameState::Menu);
}

Cargo.toml looks like this:

[package]
name = "keyboard-input"
version = "0.1.0"
edition = "2021"

[dependencies]
leafwing-input-manager = { version = "0.15.0", default-features = false, features = ["keyboard"] }

[dependencies.bevy]
version = "0.14.1"
default-features = false
features = [
    "bevy_asset",
    "bevy_gilrs",
    "bevy_state",
    "bevy_winit",
    "flac",
    "multi_threaded",
    "serialize",
    "x11",
]

What you expected to happen

Launching the app will switch into GameState::Menu and keystrokes get properly picked up by Bevy/leafwing-input-manager.

What actually happened

Sometimes it works and the println statements get printed properly, some other times it doesn't do anything. Further debugging shows that my actions get properly registered (all_actions_data() returns them properly), but the actual events don't show up in keys() or get_pressed().

Additional information

I don't get any log messages or anything when this happens. One thing that might be important is that i'm using a screen reader, namely NVDA. This screen reader doesn't intercept keys, it isn't responsible for the keys never arriving at the app, but I can see that sometimes when doing a cargo run on the cmd, the bevy window will instantly get focused, some other times it will open in the background and focus will stay within the console log window. To me it feels like this issue occurs more often when the console window stays focused other than the moments where the window receives focus automatically, but that is just a feeling, I don't have any proof.

Let me know if I can help further debug this issue.

Thanks!

[alice-i-cecile]

Alright, thanks for the report! Do you have any system-order ambiguities? The random nature of this makes me suspicious about it. The example or our test in CI should be helpful to explain how to do this. The output isn't great, especially for screen readers, so if you put up a reproduction repo I can take a look myself.

I would also be curious to see if this reproduces without the screen-reader and/or without LWIM: please let me know if you need a hand.

[Timtam]

Thanks for getting back to me. I added the edited schedule build settings, but don't get any log messages that are different from what I get anyway. I however also don't know what to expect. Yeah, output isn't great, but it works for debugging, at least here. I'm unfortunately unable to test without screen-reader as I rely on it, but I put up the minimal example as a repo on Github (https://github.com/Timtam/leafwing-input-manager-issue-demo). I usually run the demo (cargo run), try pressing up / down arrow a few times within the window, then alt + f4 out of it to see what happened. Sometimes I have to focus it first before doing so, as I've described above. Then repeat it five to ten times to get the inconsistent behaviour. Let me know if I can do anything else, thank you.

[alice-i-cecile]

Alright, thanks! I'll ask if anyone has the time to reproduce and investigate.

[ndarilek]

Hey, blind screen reader using Bevy developer here, I'll try to help you out.

I think there may be 2 issues here--the first being the UI/accessibility focus, which I've experienced but haven't dug much into and don't think is specific to LWIM. One thing you almost certainly have to do and can't automate is putting NVDA into sleep mode so it ignores all keystrokes sent to the app. In laptop layout this is done via NVDA-Shift-Z--not sure what the desktop layout command is.

Does the issue go away if you focus the window and use sleep mode? My production game struggles a bit and doesn't launch deterministically, but if I focus the window and ensure sleep mode is on, I haven't had input issues (well, not this blatant at least.) But sometimes the window gets accessibility focus, others it doesn't, and when that happens I have to do the focus/sleep-mode dance to have input reliably intercepted.

Hoping to find cycles to look into the weird accessibility focus issues during this release, and hopefully fixing that will make these problems with input go away too.

[Timtam]

Hi there, thanks for taking a look. Nope, sleep mode unfortunately doesn't help here. Running latest NVDA alpha, toggling sleep mode on while in the window, nor toggling sleep mode on while in the console window and alt tabbing to it helps, it just doesn't receive my input. I don't know if its actually key-related, at least arrow keys don't work. I haven't tested anything else yet.