Implementing interactivity for Canvas

[arctic-alpaca]

Hello,

Context

Implementing mouse and/or keyboard behavior for the canvas widget could allow for interactive charts, maps with drag navigation and similar interactive elements.

Problem description & Solution

Currently, Canvas doesn't implement MouseHandler. As far as I can tell, this should be pretty straight forward to add, but I don't know if there are some factors I'm missing. Similar for the KeyDownHandler.

The ActivateHandler could also be useful to allow focus of a canvas to use shortcut keys specific to what the canvas shows (eg, arrowkeys for a map, modifiers like ctrl-click).

In addition to the KeyDownHandler, I think a KeyUpHandler could also be helpful for things like movement with arrowkeys (e.g. for traversing a 2D map).

Examples and MockUps

The following already seems to work.

Modified crates/widgets/canvas.rs:

use super::behaviors::MouseBehavior;

use crate::{api::prelude::*, prelude::*, proc_macros::*, theme::prelude::*};

widget!(
    /// Canvas is used to render 3D graphics.
    Canvas: MouseHandler {
        /// Sets or shares the render pipeline.
        render_pipeline: DefaultRenderPipeline,

        pressed: bool
    }
);

impl Template for Canvas {
    fn template(self, id: Entity, ctx: &mut BuildContext) -> Self {
        self.name("Canvas").style("canvas-three").pressed(false)
            .child(
                MouseBehavior::new()
                    .pressed(id)
                    .enabled(id)
                    .target(id.0)
                    .build(ctx),
            )
    }

    fn render_object(&self) -> Box<dyn RenderObject> {
        PipelineRenderObject.into()
    }
}

This can be used similar to a button:

.child(
    Canvas::new()
        .attach(Grid::row(2))
        .render_pipeline(DefaultRenderPipeline(Box::new(
            Graphic2DPipeline::default(),
        )))
        .on_click(move |states, point| {
            println!("mouse up: {:#?}", point);
            true
        })
        .on_mouse_move(move |states, point| {
            println!("mouse move: {:#?}", point);
            true
        })
        .on_mouse_down(move |states, point| {
            println!("mouse down: {:#?}", point);
            true
        })
        .on_mouse_up(move |states, point| {
            println!("mouse up: {:#?}", point);
        })
        .on_scroll(move |states, point| {
            println!("scroll: {:#?}", point);
            true
        })
        .build(ctx),
)

Steps I think would be needed for this feature

• [ ] Implement MouseHandler for Canvas
• [ ] Implement KeyDownHandler for Canvas
• [ ] Implement ActivateHandler for Canvas
• [ ] Implement KeyUpHandler
• [ ] Implement KeyUpHandler for Canvas

I'd like to try to work on this, if this is something that's acceptable. Any feedback would be appreciated.

[FloVanGH]

Sure that’s acceptable thank you 🙂

[arctic-alpaca]

This is going quite alright, I struggled a bit with the problem of out of canvas mouse up events registering as described in #356.

This is where I'm currently at. My plan going forward is to implement a canvas_behavior similar to text_behavior so a canvas can be properly focused to allow keyboard input only if focused.

[arctic-alpaca]

Is there a reason for not allowing on_key_down_key to access StatesContext?

My plan going forward is to implement a canvas_behavior similar to text_behavior so a canvas can be properly focused to allow keyboard input only if focused.

As far as I can tell, it isn't possible to have a .on_key_down method that's focus aware. So at the moment, the .on_key_down and on_key_up methods fire whenever a key is pressed. This means interactivity would either need a different API than the .on_... methods or a way to discard input if the canvas isn't focused. Is it possible to make the .on_... methods focus aware? This would be my preferred option.

[FloVanGH]

Nice work thank you.

This is where I'm currently at. My plan going forward is to implement a canvas_behavior similar to text_behavior so a canvas can be properly focused to allow keyboard input only if focused.

There is a FocusBehavior maybe it is what you need.

Is there a reason for not allowing on_key_down_key to access StatesContext?

I work on a new message concept for the states and now it looks like it will replace the StatesContext in callbacks with a MessageSender. This should be available on all callbacks.

[arctic-alpaca]

There is a FocusBehavior maybe it is what you need.

I couldn't figure out how to add this to canvas so I created a canvas_behavior which is basicly a stipped down version of text_behavior only keeping the focus management. This seems to work as I can switch focus between a TextBox and the Canvas.

Here is the current code (branch canvas_interactivity).

This can be tested with the following main.

```rust use orbtk::prelude::*; use orbtk::shell::prelude::Key; use orbtk::widgets::behaviors::{FocusBehaviorState, CanvasBehaviorState}; // OrbTk 2D drawing #[derive(Clone, Default, PartialEq, Pipeline)] struct Graphic2DPipeline{ color: Color, } impl RenderPipeline for Graphic2DPipeline { fn draw(&self, render_target: &mut RenderTarget) { let mut render_context = RenderContext2D::new(render_target.width(), render_target.height()); let width = 120.0; let height = 120.0; let x = (render_target.width() - width) / 2.0; let y = (render_target.height() - height) / 2.0; // render_context.set_fill_style(utils::Brush::SolidColor(Color::from("#000000"))); render_context.set_fill_style(utils::Brush::SolidColor(self.color)); render_context.fill_rect(x, y, width, height); render_target.draw(render_context.data()); } } #[derive(Default, AsAny)] pub struct MainViewState { my_color: Color, } impl MainViewState { fn print_something(&mut self) { println!("test"); } } impl State for MainViewState { fn init(&mut self, _registry: &mut Registry, _ctx: &mut Context) { self.my_color = Color::from_name("black").unwrap(); println!("MyState initialized."); } fn update(&mut self, _registry: &mut Registry, ctx: &mut Context) { println!("MyState updated."); } fn update_post_layout(&mut self, _registry: &mut Registry, ctx: &mut Context) { println!("MyState updated after layout is calculated."); } } widget!( MainView { render_pipeline: DefaultRenderPipeline, text: String } ); impl Template for MainView { fn template(self, id: Entity, ctx: &mut BuildContext) -> Self { self.name("MainView") .render_pipeline(DefaultRenderPipeline(Box::new(Graphic2DPipeline::default()))) .child( Grid::new() .rows(Rows::create().push("*").push("auto").push("auto").push("*")) .child( Button::new() .text("spin cube") .v_align("end") .attach(Grid::row(0)) .margin(4.0) .on_click(move |states, _| { states.get_mut::(id).print_something(); true }) .build(ctx), ) .child( TextBox::new() .water_mark("TextBox...") .text(("text", id)) .margin((0, 8, 0, 0)) .attach(Grid::row(1)) .on_key_down(move |states, key_event| { println!("on_key_down_text: {:#?}", key_event); false }) .build(ctx), ) .child( TextBlock::new() .attach(Grid::row(2)) .text("Canvas (render with OrbTk)") .style("text-block") .style("text_block_header") .margin(4.0) .build(ctx), ) .child( Canvas::new() .attach(Grid::row(3)) .render_pipeline(id) .on_click(move |states, point| { println!("on_click: {:#?}", point); true }) .on_mouse_move(move |states, point| { println!("on_mouse_move: {:#?}", point); true }) .on_mouse_down(move |states, point| { println!("on_mouse_down: {:#?}", point); true }) .on_mouse_up(move |states, point| { println!("on_mouse_up: {:#?}", point); }) .on_scroll(move |states, point| { println!("on_scroll: {:#?}", point); true }) .on_key_down_key(Key::Escape, move || { println!("escape down"); true }) .on_key_up_key(Key::Escape, move || { println!("escape up"); true }) .on_key_down(move |states, key_event| { println!("on_key_down: {:#?}", key_event); false }) .on_key_up(move |states, key_event| { println!("on_key_up: {:#?}", key_event); true }) .on_changed("focused", move |states, event| { println!("on_changed_focus"); }) .build(ctx), ) .build(ctx), ) } } fn main() { // use this only if you want to run it as web application. orbtk::initialize(); Application::new() .window(|ctx| { orbtk::prelude::Window::new() .title("OrbTk - canvas example") .position((100.0, 100.0)) .size(420.0, 730.0) .resizeable(true) .child(MainView::new().build(ctx)) .build(ctx) }) .run(); } ```

As far as I can tell, it isn't possible to have a .on_key_down method that's focus aware. So at the moment, the .on_key_down and on_keyup methods fire whenever a key is pressed. This means interactivity would either need a different API than the .on... methods or a way to discard input if the canvas isn't focused. Is it possible to make the .on_... methods focus aware? This would be my preferred option.

I'm still unsure how to solve this problem. Is there a way to check in the .on_... methods whether the canvas currently is focused? I couldn't figure out if there is way to access the focused field of CanvasBehavior from MainView.

[FloVanGH]

I'm still unsure how to solve this problem. Is there a way to check in the .on_... methods whether the canvas currently is focused? I couldn't figure out if there is way to access the focused field of CanvasBehavior from MainView

Unfortunately not you have to check the focus inside of the state.

[arctic-alpaca]

Do you mean inside the update methods in impl State for MainViewState {...? Could you please give me an example how I could access the focused field from there is that's what you mean? Nevermind, I figured it out.

[arctic-alpaca]

I'm quite happy with how this works at the moment, but I encountered some weird behavior. I added two examples to the code, the first canvas_working.rs seems to work as expected. If the Canvas is focused, mouse movement, key_up, key_down, mouse_up and mouse_down get printed to the console. The second one, canvas_not_working is the same code, just with the TextBlock and the TextBox switched around. This doesn't print mouse_down or mouse movement when over the Canvas but does print it when over the part of the Canvas that's overlapping onto the TextBox while the Canvas is focused. I'm not sure what's happening here, any input would be appreciated. This might be related to #170.

[FloVanGH]

Yeah could be a issue with the Grid. Have you tried to use fix row sizes instead? If you start your example with cargo run --features debug you can see the raster of the widgets.

[arctic-alpaca]

I tested fixed row sizes and apparently the position of the TextBox, TextBlock and the Canvas in code is the relevant factor. In my testing, the problem only occured when the TextBox was created right before the Canvas. The debug feature shows no overlap with this row sizes and I have no idea what's happening.

This works regardless whether the `Row` position is switched between `TextBox` and `TextBlock`.

```rust ... .child( .child( Grid::new() .rows(Rows::create().push("30").push("30").push("150")) .child( TextBox::new() // Grid::Row(0) or Grid::Row(1) works .attach(Grid::row(1)) .water_mark("TextBox...") .text(("text", id)) .margin((0, 8, 0, 0)) .build(ctx), ) .child( TextBlock::new() // Grid::Row(0) or Grid::Row(1) works .attach(Grid::row(0)) .text("Canvas (render with OrbTk)") .style("text-block") .style("text_block_header") .margin(4.0) .build(ctx), ) .child( Canvas::new() .id(CANVAS_ID) .attach(Grid::row(2)) ... ```

This doesn't work, regardless whether the `Row` position is switched between `TextBox` and `TextBlock`.

```rust ... .child( Grid::new() .rows(Rows::create().push("30").push("30").push("150")) .child( TextBlock::new() // Grid::Row(0) or Grid::Row(1) doesn't work .attach(Grid::row(1)) .text("Canvas (render with OrbTk)") .style("text-block") .style("text_block_header") .margin(4.0) .build(ctx), ) .child( TextBox::new() // Grid::Row(0) or Grid::Row(1) doesn't work .attach(Grid::row(0)) .water_mark("TextBox...") .text(("text", id)) .margin((0, 8, 0, 0)) .build(ctx), ) .child( Canvas::new() .id(CANVAS_ID) .attach(Grid::row(2)) ... ```

[FloVanGH]

Ok as soon as I can spent some time I will check this.

[FloVanGH]

I found the source of the problem https://github.com/redox-os/orbtk/blob/e19445cc3b9eee8e89138d91c5f447959cfb2de9/crates/api/src/systems/event_state_system.rs#L266. This part of the code is intended that children that are clipped by parent does not recognized mouse move on the clipped parts. I think this part of the code works not correct. Its not critical therefore I will remove this code and create an issue to fix block mouse move outside of clipped paths.

[FloVanGH]

is removed

[arctic-alpaca]

Thank you very much! I'll work on an example and test the interactivity implementation a bit more and then create a PR.

[arctic-alpaca]

The mouse down event still doesn't work. I'm not sure why, especially since click events work and replacing the https://github.com/redox-os/orbtk/blob/5bf1e2b6497b59469bdf604ff1ff826f066b75fb/crates/api/src/systems/event_state_system.rs#L221 part with the same part from the click handling doesn't change anything.

[FloVanGH]

Sorry but I'm little bit busy at the moment. I will check it next week.

[arctic-alpaca]

Thank you for the head up. I most likely won't be able to work on this for this month as I have to focus on University.

[arctic-alpaca]

Sorry for the long delay. I rebased the code to be up to date with the develop branch.

I looked a bit more into this, but I couldn't find the source of the problem.

[arctic-alpaca]

When commenting out .child(text_behavior) in text_box.rs, the mouse_down event works as expected. Seems like the problem is related to that. https://github.com/redox-os/orbtk/blob/085f04a87b7dd793a13fc22e2a0dffd2c010737d/crates/widgets/src/text_box.rs#L113