Absorbing/suppressing keyboard/mouse input when hovering over egui window

[rezural]

Hey

I'm having issues (and generally a nice time!) with using bevy-egui, and mouse/keyboard input still being consumed by the main bevy system code. This would be something nice to suppress somehow, probably opt-in, however I do feel that generally the happy path is for egui related input to be handled by egui, then suppressed (if possible).

Currently I am suppressing if egui.has_pointer_focus() (or somesuch), however, all my input processing code on the bevy side has to be made aware of egui, and it is getting unweildy.

I don't have enough of an understanding of bevy's input system, and where bevy-egui actually get's it's input from, to know where this is solvable from, so I'm just asking here as it seems the most obvious choice.

Is this solvable within bevy-egui? Is it currently do-able with bevy's input system?

Thanks

[mvlabat]

Hi @rezural!

Thanks for the question. You're right, currently, the only way to do that is to pass Egui context to each system interested in the inputs and call has_pointer_focus (or the better choice might be wants_keyboard_input or wants_pointer_input). This is how I was doing it myself in my own project, but I agree - adding this to every system processing input can be cumbersome.

I think the only way bevy_egui can help is to introduce its own resources for input events, which would be emitted only if they are not consumed by Egui. This is something I'd probably explore closer to the 0.6 release, but I'm also happy to accept PRs if anyone wants to tackle this earlier.

[rezural]

Ok, good to hear.

I will ask around in bevy if there are any ways to accomplish this using the event/input system. If I manage to get something working, I will send up a PR.

[AlanRace]

It sounds like this issue is related to one I am currently having, so thought I would just add this here. Zooming in/out of a plot (holding Ctrl and using the mouse wheel) doesn't seem to work, however panning works fine (click and drag). Simple example below.

use bevy::prelude::*;
use bevy_egui::{
    egui::{
        self,
        plot::{Line, Plot, Value, Values},
    },
    EguiContext, EguiPlugin,
};

fn main() {
    App::build()
        .add_plugins(DefaultPlugins)
        .add_plugin(EguiPlugin)
        .add_system(ui_example.system())
        .run();
}

// Note the usage of `ResMut`. Even though `ctx` method doesn't require
// mutability, accessing the context from different threads will result
// into panic if you don't enable `egui/multi_threaded` feature.
fn ui_example(egui_context: ResMut<EguiContext>) {
    egui::Window::new("Hello").show(egui_context.ctx(), |ui| {
        ui.label("world");

        let sin = (0..1000).map(|i| {
            let x = i as f64 * 0.01;
            Value::new(x, x.sin())
        });

        let line = Line::new(Values::from_values_iter(sin));
        Plot::new("my_plot")
            .view_aspect(2.0)
            .show(ui, |plot_ui| plot_ui.line(line));
    });
}

[mvlabat]

@AlanRace zoom events support is an unrelated request to this one. I implemented it in 0.10.2. If you'll get any problems with it, feel free to submit a separate issue.

[jabuwu]

Isn't this already possible? Something like this

use bevy::{input::InputSystem, prelude::*};
use bevy_egui::{egui, EguiContext, EguiPlugin, EguiSystem};

#[derive(Default)]
struct EguiBlockInputState {
    wants_keyboard_input: bool,
    wants_pointer_input: bool,
}

fn main() {
    App::new()
        .add_plugins(DefaultPlugins)
        .add_plugin(EguiPlugin)
        .init_resource::<EguiBlockInputState>()
        .add_system_to_stage(CoreStage::PreUpdate, egui_block_input.after(InputSystem))
        .add_system_to_stage(
            CoreStage::PostUpdate,
            egui_wants_input.after(EguiSystem::ProcessOutput),
        )
        .add_system(test)
        .run();
}

fn egui_wants_input(mut state: ResMut<EguiBlockInputState>, mut egui_context: ResMut<EguiContext>) {
    state.wants_keyboard_input = egui_context.ctx_mut().wants_keyboard_input();
    state.wants_pointer_input = egui_context.ctx_mut().wants_pointer_input();
}

fn egui_block_input(
    state: Res<EguiBlockInputState>,
    mut keys: ResMut<Input<KeyCode>>,
    mut mouse_buttons: ResMut<Input<MouseButton>>,
) {
    if state.wants_keyboard_input {
        keys.reset_all();
    }
    if state.wants_pointer_input {
        mouse_buttons.reset_all();
    }
}

fn test(
    mut egui_context: ResMut<EguiContext>,
    keys: Res<Input<KeyCode>>,
    mouse: Res<Input<MouseButton>>,
    mut string: Local<String>,
) {
    egui::Window::new("Hello").show(egui_context.ctx_mut(), |ui| {
        ui.text_edit_singleline(&mut *string);
    });
    if keys.just_pressed(KeyCode::A) {
        println!("pressed A!");
    }
    if mouse.just_pressed(MouseButton::Left) {
        println!("pressed left mouse button!");
    }
}

[JAD3N]

A solution I ended up using was adding all my systems to a system set and adding run criteria to skip if egui is being used.

fn run_if_available(
    mut egui_context: ResMut<EguiContext>,
) -> ShouldRun {
    let ctx = egui_context.ctx_mut();
    if ctx.is_using_pointer() || ctx.is_pointer_over_area() {
        ShouldRun::No
    } else {
        ShouldRun::Yes
    }
}

[philpax]

@jabuwu's solution seems to work for me! If there aren't any issues with it that I'm missing (I'm using it for a tool, not a game) then I'd suggest adding it as an example 🙂

[TotalKrill]

This is another ugly hack I made that disables the RaycastSource, so that external plugins are "disabled", by not trying to do raycasting ( as I am using )

the bevy_transform_gizmo needs patching to handle lack of RayCastSource

fn raycast_enable_disable(
    mut commands: Commands,
    mut egui_context: ResMut<EguiContext>,
    camera: Query<
        (
            Entity,
            Option<&GizmoPickSource>,
            Option<&DrawShapeRaycastSource>,
            Option<&bevy_mod_picking::PickingCamera>,
        ),
        With<SwitchableCamera>,
    >,
    mut previous: Local<bool>,
) {
    let (camera, gizmoray, shaperay, pickray) = camera.get_single().unwrap();
    let mut entity = commands.get_entity(camera).unwrap();
    let ctx = egui_context.ctx_mut();

    macro_rules! remove {
        ($val:ident, $t:ty) => {
            if let Some(_) = $val {
                entity.remove::<$t>();
            }
        };
    }
    macro_rules! insert {
        ($val:ident, $in:ty) => {
            if let None = $val {
                entity.insert(<$in>::default());
            }
        };
    }

    let now = ctx.is_pointer_over_area();
    if now == *previous {
        // do nothing
    } else {
        if now {
            remove!(gizmoray, GizmoPickSource);
            remove!(shaperay, DrawShapeRaycastSource);
            remove!(pickray, bevy_mod_picking::PickingCamera);
        } else {
            insert!(gizmoray, GizmoPickSource);
            insert!(shaperay, DrawShapeRaycastSource);
            insert!(pickray, bevy_mod_picking::PickingCamera);
        }
    }
    *previous = now;
}

[lwiklendt]

Since Bevy 0.10 egui contexts have moved from resources to components. To filter out a system when egui wants input I created a run condition:

fn egui_wants_input(q: Query<EguiContextQueryReadOnly>) -> bool {
    q.iter().any(|ctx| {
        let egui_ctx = ctx.ctx.get();
        egui_ctx.wants_pointer_input()
            || egui_ctx.wants_keyboard_input()
            || egui_ctx.is_pointer_over_area()
    })
}

However, this requires the immutable_ctx feature to be enabled which is discouraged. Using the mutable equivalent

fn egui_wants_input(mut q: Query<EguiContextQuery>) -> bool {
    q.iter_mut().any(|mut ctx| {
        let egui_ctx = ctx.ctx.get_mut();
        egui_ctx.wants_pointer_input()
            || egui_ctx.wants_keyboard_input()
            || egui_ctx.is_pointer_over_area()
    })
}

does not work with run_if resulting in the error

.run_if(not(egui_wants_input))
        --- ^^^^^^^^^^^^^^^^ the trait `ReadOnlyWorldQuery` is not implemented for `bevy_egui::EguiContextQuery`
        |
        required by a bound introduced by this call

Previously the mutable version (ResMut equivalent) worked with iyes_loopless. Is it correct to use the immutable_ctx feature in this case despite the discouragment?

[TotalKrill]

The other way is to use a resource to store the "egui_wants_input" boolean, and then use that one to check the runcondition.

[mvlabat]

@lwiklendt yes, I think immutable_ctx can be used in this case, but caution is advised anyway, as these calls will lock the context for reading (if there's a concurrent write operation, a call will be blocked).

I like @TotalKrill's suggestion about writing the return values of wants_* calls to a resource. I think bevy_egui can even provide such a resource together with the run conditions that @lwiklendt describes. If anyone wants to create a PR adding these, I'll happily merge it (or I'll do it myself later, when it comes closer to another minor release).

[Youdiie]

@TotalKrill hi, I'm using bevy_transform_gizmo and bevy_egui. I also wanted to disable RaycastSource when moving mouse on egui, and I tried to use your code. However, code panicked with this.

Missing gizmo raycast source: NoEntities("bevy_ecs::query::state::QueryState<(bevy_ecs::entity::Entity, &bevy_mod_raycast::RaycastSource)>")', src\crates\bevy_transform_gizmo\src\lib.rs:399:14

I think when removing GizmoPickSource from camera, bevy_transfrom_gizmo calls panic. How did you solve this problem?

[TotalKrill]

Just patched the gizmo code so that it doesn't crash and used my own vendored version instead

[paul-hansen]

Is there a reason we don't just clear the bevy Input struct for the related inputs when they should be consumed by egui? Or was this option just overlooked?

Probably have a way to disable it in case it causes trouble, but it seems to be the documented way of handling this in Bevy: https://docs.rs/bevy/latest/bevy/input/struct.Input.html#usage

Multiple systems

In case multiple systems are checking for Input::just_pressed or Input::just_released but only one should react, for example in the case of triggering State change, you should consider clearing the input state, either by:

• Using Input::clear_just_pressed or Input::clear_just_released instead.
  • Calling Input::clear or Input::reset immediately after the state change.

[paul-hansen]

Adding this to my app seems to be working well in my initial testing.

app.add_systems(
    PreUpdate,
    (absorb_egui_inputs,)
            .after(bevy_egui::systems::process_input_system)
            .before(bevy_egui::EguiSet::BeginFrame),
);

fn absorb_egui_inputs(mut mouse: ResMut<Input<MouseButton>>, mut contexts: EguiContexts) {
    if contexts.ctx_mut().is_pointer_over_area() {
        mouse.reset_all();
    }
}

Edit: Added .before(bevy_egui::EguiSet::BeginFrame), to prevent race condition where it wouldn't work sometimes.

[TotalKrill]

I think this was just overlooked, or when the discussion started, this was before it was this easy with the sets. Anyway I will try your suggestion, because it does indeed look easier!

[tbillington]

@paul-hansen your solution worked great.

I had to alter it for my use case to also disable keyboard input since the user is typing in a text box. In addition to reset_all on the keyboard, I had to manually preserve the pressed states of meta keys to allow things like copy/paste (ctrl+c/v) and holding alt + arrow to jump words at a time.

fn absorb_egui_inputs(
    mut mouse: ResMut<Input<MouseButton>>,
    mut kbd: ResMut<Input<KeyCode>>,
    mut contexts: bevy_egui::EguiContexts,
) {
    if contexts.ctx_mut().is_pointer_over_area() {
        mouse.reset_all();
        let pressed_keys = [
            KeyCode::SuperLeft,
            KeyCode::ControlLeft,
            KeyCode::AltLeft,
            KeyCode::ShiftLeft,
        ]
        .map(|k| (k, kbd.pressed(k)));
        kbd.reset_all();
        for (key, _) in pressed_keys.iter().filter(|(_, p)| *p) {
            kbd.press(*key);
        }
    }
}

[lain-dono]

A more complete and more elegant solution for copy-paste. wants_pointer_input for drag-n-drop. I suppose it should eventually be in the documentation and/or examples.

fn absorb_egui_inputs(
    mut contexts: bevy_egui::EguiContexts,
    mut mouse: ResMut<Input<MouseButton>>,
    mut keyboard: ResMut<Input<KeyCode>>,
) {
    let ctx = contexts.ctx_mut();
    if ctx.wants_pointer_input() || ctx.is_pointer_over_area() {
        let modifiers = [
            KeyCode::SuperLeft,
            KeyCode::SuperRight,
            KeyCode::ControlLeft,
            KeyCode::ControlRight,
            KeyCode::AltLeft,
            KeyCode::AltRight,
            KeyCode::ShiftLeft,
            KeyCode::ShiftRight,
        ];

        let pressed = modifiers.map(|key| keyboard.pressed(key).then_some(key));

        mouse.reset_all();
        keyboard.reset_all();

        for key in pressed.into_iter().flatten() {
            keyboard.press(key);
        }
    }
}

[tbillington]

Adding mouse wheel support to lains snippet:

fn absorb_egui_inputs(
    mut contexts: bevy_egui::EguiContexts,
    mut mouse: ResMut<Input<MouseButton>>,
    mut mouse_wheel: ResMut<Events<MouseWheel>>,
    mut keyboard: ResMut<Input<KeyCode>>,
) {
    let ctx = contexts.ctx_mut();
    if ctx.wants_pointer_input() || ctx.is_pointer_over_area() {
        let modifiers = [
            KeyCode::SuperLeft,
            KeyCode::SuperRight,
            KeyCode::ControlLeft,
            KeyCode::ControlRight,
            KeyCode::AltLeft,
            KeyCode::AltRight,
            KeyCode::ShiftLeft,
            KeyCode::ShiftRight,
        ];

        let pressed = modifiers.map(|key| keyboard.pressed(key).then_some(key));

        mouse.reset_all();
        mouse_wheel.clear();
        keyboard.reset_all();

        for key in pressed.into_iter().flatten() {
            keyboard.press(key);
        }
    }
}

[philpax]

Updating tbillington's update to lain's update to tbillington's update to paul-hansen's implementation for Bevy 0.14:

// in your plugin or `main`:
app.add_systems(
    PreUpdate,
    absorb_egui_inputs
        .after(bevy_egui::systems::process_input_system)
        .before(bevy_egui::EguiSet::BeginFrame)
);

fn absorb_egui_inputs(
    mut contexts: bevy_egui::EguiContexts,
    mut mouse: ResMut<ButtonInput<MouseButton>>,
    mut mouse_wheel: ResMut<Events<MouseWheel>>,
    mut keyboard: ResMut<ButtonInput<KeyCode>>,
) {
    let ctx = contexts.ctx_mut();
    if !(ctx.wants_pointer_input() || ctx.is_pointer_over_area()) {
        return;
    }
    let modifiers = [
        KeyCode::SuperLeft,
        KeyCode::SuperRight,
        KeyCode::ControlLeft,
        KeyCode::ControlRight,
        KeyCode::AltLeft,
        KeyCode::AltRight,
        KeyCode::ShiftLeft,
        KeyCode::ShiftRight,
    ];

    let pressed = modifiers.map(|key| keyboard.pressed(key).then_some(key));

    mouse.reset_all();
    mouse_wheel.clear();
    keyboard.reset_all();

    for key in pressed.into_iter().flatten() {
        keyboard.press(key);
    }
}