`Control` class: Common response function names

kmatch98 commented 3 years ago

With the advent of new widgets such as icon_widget, a clear need has arisen for common function naming schemes for the Control class. The objective is so that the main event handler can use a common approach for calling multiple widgets to respond to various input events.

Of course, widgets aren't required to conform to any pre-defined response functions, but it will likely make it easier to write a Main loop event handler if widgets have the same function naming and a common expected effect.

Here's one example. Looking at the PyGUI, it defines responses to four mouse events:

mouse down
mouse up
mouse drag (button is pressed)
mouse move (button is not pressed)

This issue is raised to gather suggestions for the basic response functions to use in the Control class.

Let's discuss a few basic functions that make sense in a touch environment on small microcontroller screens. The objective is not to collect 100% of cases, but to capture the few things that would be most used, and then grow from there.

jposada202020 commented 3 years ago

I would add the events from the kivy project.


    on_touch_down()

    on_touch_move()

    on_touch_up()

https://kivy.org/doc/stable/guide/inputs.html. I thinks could be closer to our architecture and purposes

kmatch98 commented 3 years ago

I like your set of functions. I agree that we just need a few basic functions.

I recommend we include a placeholder function for “no touch” or “update()”. I found a function like that is necessary if a widget is doing some kind of animation even when it’s not being interacted with. Like an animation of an icon or maybe a graph that needs to update its graphics in the background.

And if a widget doesn’t need one of these methods, it can just default to the superclass’s empty pass function.

jposada202020 commented 3 years ago

@kmatch98 are you planning in doing a slider, meaning, a non-discrete switch button? to control PWM, voltage output, and other things like that? I am asking because:

it will be a good playground to try some of the above
If you are not planning to do it I can take a look.

kmatch98 commented 3 years ago

I agree a slider is a great example that will need all those touch options. Go for it!

jposada202020 commented 3 years ago

Kmatch, hardware will come at the end of this week, (meaning Touchscreen) So I could work on the actions, howeevr rough prototype

jposada202020 commented 3 years ago

By the way I re-use your switch_round :)

FoamyGuy commented 3 years ago

I think we shouldn't use names with the on_ prefix for now. I feel like that prefix implies that the function will get called for you when an event happens. Since we don't support that type of behavior yet I think folks might be confused by these functions if they have experience with languages that use callback functions with similar names.

jposada202020 commented 3 years ago

Good point, I am not familiar at all with other languages callbacks so the prefix on_ does not ring any bell, maybe it could be called when_moved, when_touched? There are other alternatives, in matplotlib, a callback method is added to the class, this way when needed they add a .callback() to the object to trigger a behavior or in tkinter (and this is weird and I do not like it) they add the callbacks as a parameter or a lambda function inside class instance definitions.

kmatch98 commented 3 years ago

How about the following:

Functions that send the touch position:

contains - check if touch is within touch boundary
touch_down - first time pressed down
touch_move - still pressed down but moves somewhere else, location could be anywhere (within boundary or outside boundary)
touch_up - touch was removed

Function with no touch position:

update - if a widget needs to do some background tasks (maybe continuous animations, or other low priority housekeeping)

jposada202020 commented 3 years ago

I like them

jposada202020 commented 3 years ago

Moving here some discussion regarding this. Context here is a widget called Slider.

Comments from @kmatch98 regarding is first review, and the way to handle their behavior:

First Review

@jposada202020 This is a really cool. It really responds to the finger pretty fast! Here are my comments. Sorry if it looks like a lot of things, most are minor tweaks, I think the main question is about I expected the slider should respond to touch, and that's open for discussion.

graphics: The slider can go outside the background box’s range, if slid all the way to the right side.

touch behavior: The slider does not “center” on the touch point. I have some touch-calibration error on my screen, but I think there is some offset in the code.

touch action: For sliders, I think the touch-response should be as follows:

Must first receive a “touch_down” within the touch_boundary.
If touch is kept down, any “touch_move” actions will cause the slider to change values, even if the touchpoint is outside of the touch_boundary.
- (Optional bonus behavior) The farther away the finger is from the slider, the “slower” the reaction. That means, that you can be more accurate if you touch the slider, then move your finger vertically down, and then the slider can be placed more accurately when your finger is farther away. I think this behavior is used on sliders for videos, and I appreciate it if you are trying to adjust the slider precisely. Maybe this could have a “gain” option to adjust the sensitivity based on vertical distance from the slider.
When the touch is released from the screen, the slider must be directly touched again to go back to #1.

Implementation: If you implement the behavior above, I think you will need a state variable to track whether the slider has received a “touch_down” event. To clear the state variable, I think you will need to create a “touch_up” Control function to let the slider know when it is released.

function names: I recommend changing the when_inside command and renaming it to the contains boolean function. Then your while loop can use:

if p:
    if my.slider.contains(p): 
        my_slider.touch_down(p) # let the slider know it has been touched down 
    my_slider.touch_move(p) # handle any touch movements if the slider is still touched down
else: 
    my_slider.touch_up(p) # let the slider know it has been released

Another option is to add a public state variable so you can check the slider's "touch" state. That way you could selectively call touch_move only when the "touch" state is "down". Not sure if that's better or worse than the above code suggestion.

touch boundary: I think that more padding should be available at the “bottom” of the slider than the top. When you touch the slider, you want to still be able to see the slider above your finger. So your finger’s touchpoint will likely be “below” the slider. I think it will be useful to have an option for non-uniform touch_padding. I suggest including some default bottom_touch_padding since that would be expected by the user.

simpletest example: The slider is really small on my pyPortal. Maybe make it a bit larger.

value: I think the value should default to be a float between 0 and 1.0. Or the min/max value could be set during the initialization, similar to what I think is done in the progress bar.

jposada202020 commented 3 years ago

Same topic from @kmatch98 regarding the handle of the touch

I'm thinking about the touch-response some more and maybe what I said above needs to be updated. I don't think the simple code that I put above will work right in the case if you have multiple sliders.

We need to handle a situation where one slider receives a "touch_down" and then any other sliders (or other widgets for that matter) should not be triggered with touch events until the slider gets the "touch_up".

I suspect @FoamyGuy has some experience with this. Do you think that the event loop should handle which widget should have the "focus"? For example, once we first touch a slider, we should only send "touch_move" commands to slider, until the next "touch_up" (when finger is released from the screen).

Should this be handled by the event loop, or should each widget need to have the right state variables to understand whether it has focus.

Here's a suggestion:

The main event loop should keep track of the full "touch_down"->"touch_move"->"touch_up" cycle. (This means that the event loop must first send a "touch_down" signal before ever sending a "touch_move" or "touch_up".) The event loop needs to keep track of which "touch_*" state it is in. But it doesn't need to keep track of which widget should have "focus". It will keep sending these signals to all widgets and widgets have to decide if they have "focus".
The widget should keep track if it received a "touch_down" response.
The widget should only respond to "touch_move" or "touch_up" if it already received a "touch_down" response.

This is just a general concept (of course you could develop weird widgets that don't follow this exactly).

Your thoughts are welcome.

jposada202020 commented 3 years ago

Comments from @jposada202020

@kmatch98 Interesting.... I did this in a different way in the past... as we are merging MP, I would like to see if we could use some of the new features introduced. Meaning, For me as it is, you would have an event loop, meaning, what you said, the main code will verify, but we add all the widgets to an event loop and all will get updated at the same time, instead have them async.

I agree with you, this discussion is very important, not only for this, but for all the widget development, we need to establish some kind of design rules. Our purpose here will be maintainability, easy of support, easy of maintainability and the most important that for the user will be as easy as to say

if widget.touch:
   ...do something

Proposal lets copy paste this comments to the graphics team, you know I am down for whatever is easier for the user, and it would be easier in the future to add more features.

I am a superfan of the text parameter keywords, because, if matplotlib taught us something is you always want to add more feature, and the enum thing restrict this, at least make it more difficult. But I understand why people like their ENUMS :)

After all this discussion, could you re-post this comments in the graphics team, I could do it, but do not want to do this without your approval

jposada202020 commented 3 years ago

Comments from @FoamyGuy

Most of the systems I have experience with use some sort of callback type system for UI related events. I'm wondering if something like that might be possible with the new features from Micropython, but I am not sure.

In the absence of true callbacks. I do think having the logic in some sort of main event loop is probably the best way to go. I think What @TG-Techie has done with TG-Gui is great. Perhaps we should be working toward making our widgets compatible with this system. I'm not 100% how it handles "touch move" type actions, but I'm pretty sure it can, because I think I recall TG-Techie showing a sliding selector widget example on a stream one day.

jposada202020 commented 3 years ago

Comments from @jposada202020

That is a good idea. I recall seeing somewhere too, maybe twitter... but yes you are right. I remember he mentions something about the event loop. Regarding the MP depending on what they decide to merge. Anyway as you said, we know now, that what TG does. works so we could build from that.

TG-Techie commented 3 years ago

Questions and Thoughts

@jposada202020 "In the absence of true callbacks", could you elaborate please?

Please excuse that I may have missed some things in reading this thread; what are the new features from micropython?

I agree on_ is very reminiscent of other frameworks, it might carry connotations with it.

General Lessons from the TG-Gui

I'm not 100% how it handles "touch move" type actions

So the circuitpython implementation of a tg-gui event loop differentiates between something that is selected as a whole, Takes action on behalf of the input, and widgets that have continuous inputs of coords like sliders and scrollables.

From trial and error, this felt closer to the touch experience you'd find on a modern device. (I studied an Ipad and an Iphone).

There are other reasons for separating updating from selecting but, for example, picture a vertically scrolling list of buttons where touch zones fill all of the list. If the select and update functionalities are the same, the user would never be able to scroll. By separating the two it differentiates between continuous gestures and directive actions.

TG-Gui Protocols/Interfaces

These are just the method names I choose, they aren't necessarily the right ones for the CircuitPython framework.

The naming convention of the methods:

_select_() and _deselect_() for selectables
_press_ (or _action_) as "actionables", can call callbacks or manipulate internal state
_start_coord_ (Tuple[x, y]), _update_coord_(Tuple[x, y]), and _last_coord_(Tuple[x, y]) for updateables (inspired decsritpions, I know)

those are single leading and trailing underscores ("sunder" if you will)

Framework Construction Thoughts

(callbacks not included in this section) This framework may have different requirements/design goals from what is described below.

Another difference from the proposed seems that TG-Gui widgets gain control abilities through structural subtyping, like rust's traits or swift's protocols. Again this is just how I choose to do it; I judged it saved on overhead from previous versions of TG-Gui (versions 1-4 had inheritance-based control) and added more flexibility to the framework.

Widgets can use any combination of the above-listed protocols just by adding the methods to the class body of the Widget subclass.

Buttons have the selectable and actionable methods
Toggle Switches just have the actionable methods (with _action_ changing the toggle's state)
Sliders and scrollable widgets just have the updatable methods

TG-Techie commented 3 years ago

I hope that helps clarify?

jposada202020 commented 3 years ago

Thank you, for he clear explanation.

When I am referring with callback, I other python implementation, while define the widget, you define at the same time what to do when is interacted with. And you do not need to control that, this is control in a higher level by the API. So things are static, until interact with. Not sure how, as I was always the user and not the constructor. But parsing to the link you provided I guess that event loop is the brains of the event operation, The only difference in this case will be that we need a set of rules for the user to add to the control event. It will be difficult to predict all the possible cases and applications and design that event loop beforehand.

Thanks again

TG-Techie commented 3 years ago

You bring up some good points, thank you.

Yes, what I linked to is the internal brains of an event loop. By linking to it I'm (implicitly?) agreeing with KMatch's note on how having a pre-defined interface for response functions "will likely make it easier to write a Main loop event handler". Those response functions don’t necessarily need to be exposed to the user in the same way they are exposed to an event loop.

A distinction I failed to point out, is what I described above is not "user-facing". A person wiring an interface shouldn't need to understand or manipulate how an event loop ties into the implementations of control. I would think that should be the job of Widget and library maintainers.

EX: Widgets' __init__ functions could take action={some callback} for button or when_toggled={some other callback} for a toggle switch. However, both could use the same underlying _action_* method to hook into the event loop.

@kmatch98 were you proposing that all user-facing code share a common term for "on press" and the like in the constructor?

It will be difficult to predict all the possible cases and applications and design that event loop beforehand.

😂, no plan survives first contact with ~~implementation~~ user?

(* insert other method name)

kmatch98 commented 3 years ago

@kmatch98 were you proposing that all user-facing code share a common term for "on press" and the like in the constructor?

It will be difficult to predict all the possible cases and applications and design that event loop beforehand.

😂, no plan survives first contact with ~implementation~ user?

(* insert other method name)

@jposada202020 Thanks for moving the discussion over here!

I’m proposing that we come up with a few common responses that widgets will have and stick with those names. If we are inconsistent then it will take a lot of time for a user to read through all the docs to discover whether a button uses “press” and a slider uses “touched”. That will be chaos to write an event loop.

Also I would like for there to be re-usable event loops so folks don’t have to always write a new event loop.

For example I want to have an event loop that handles my button presses and my slider inputs. If each widget has similar “Control” functions, we can send the same commands to all the widgets, not having to know if the widget will actually do anything with that command. The widget can choose to ignore an event loop’s command if it doesn’t make sense to it.

A concrete example:

For a slider, once it is touched, any touch movement should change the slider value (even if the touch is moved outside the slider touch_boundary, like the slider in YouTube videos). Once the slider is touched, no other widgets should respond to any touch movements. When touch is removed the slider stays at its last value.

Here’s what I propose: Event loop always follows this event sequence:

Touch down
Touch moved
Touch released
Call update on all widgets periodically

Note: The event handler must keep track of the difference between the first touch (touch_down) and a touch movement. (@TG-Techie, I think this is how your TG-GUI event loop is constructed.)

The event handler can check if the touch_down is within the touch_boundary of a widget (“contains”), and if so it calls the touch_down Control function for that widget. (Alternately the event handler could send “touch_down” to all widgets and they can check whether they should respond to that location of touch point. Either way is fine to me, and this second way is more consistent with the other two I propose, but contains gives the event loop more options, see *** below.)

Now, the Event handler spews “touch_moved” points to all widgets. Note: Widgets have to keep track of their own state variables to know if they were “touched_down”, and whether they should respond to touch_move events.

When the touch is released, the event handler spews “touch_up” commands to all widgets.

Some widgets may need to do updates when nothing is being touched (like animations). I think an additional “update” function should be called in the event loop so the widget has a chance to update itself periodically. Widgets are responsible for not “hogging” the processor cycles.

So I propose a widget should define and respond to (or choose to ignore) these five Control class functions:

contains
touch_down (Note: A widget must keep track of its “touched_down” state, maybe we should use a consistent name for this state variable)
touch_moved
touch_up
update

*** Some things I haven’t specifically considered are “layers” or “views” in case some widgets should get priority over others for touch event “focus”. If we use the “contains” method, the event loop could choose to only send the “touch_down” command to the first (“top”) widget in its list that responds with “contains” is True.

jposada202020 commented 3 years ago

@TG-Techie One think to consider, regarding the design, we are in the middle of two worlds.. For example...You are a maker, you need the API flexible enough to make something with it. and create cool stuff like your watch, and we have a lot of people like that. If the API is not flexible enough, the maker could not do cool projects, no show and tell.. At the same time, we need to make it easy enough for people just to use, and easy enough that they just do the things, with not too much lines of code. If you go across the Sensors libraries, they need max 10 lines of codes to do get something back, we need an API like that also, I know is tricky :) In my own experience, I do not know how my email GUI works, I just want that it works.

And finally yes, the user will always find a way to do the only thing you did not think of :)

TG-Techie commented 3 years ago

Note: The event handler must keep track of the difference between the first touch (touch_down) and a touch movement.

@kmatch98 , I'm not quite sure what you mean by handler; I would guess it is the widget being interacted with is the "handler"?

Call update on all widgets periodically

Are you suggesting .update() would be used to update state or event stuff? (or both?) I've found it helpful to handle state separately from control and touch inputs.

Widgets are responsible for not “hogging” the processor cycles.

I think that's a great approach, especially for the embedded space!

At the same time, we need to make it easy enough for people just to use, and easy enough that they just do the things, with not too much lines of code.

👍, I very much agree. Have you heard of the design principle called progressive disclosure?

(@TG-Techie, I think this is how your TG-GUI event loop is constructed.)

TG-Gui (on circuitpython) doesn't currently have the widget being interacted check if the coordinates are contained within it, but it really should😅. However, TG-Gui is structured so the underlying implementation can change but the user-facing API doesn't. (This is what allows TG-Gui to run on circuitpython or the desktop and will later allow drag and drop to be added)

jposada202020 commented 3 years ago

+1, I very much agree. Have you heard of the design principle called progressive disclosure?

Not a clue, but I think with this post I will progressively learn 💻 about it. 😄

kmatch98 commented 3 years ago

Note: The event handler must keep track of the difference between the first touch (touch_down) and a touch movement.

@kmatch98 , I'm not quite sure what you mean by handler; I would guess it is the widget being interacted with is the "handler"?

To clarify, I mean that the main while loop has to detect and then send commands always in the touch_down -> touchmoved -> touch up sequence.

Call update on all widgets periodically

Are you suggesting .update() would be used to update state or event stuff? (or both?) I've found it helpful to handle state separately from control and touch inputs.

My proposed update command should be for doing non-touch background processing. For example if there is an animation playing inside of a button widget, the button should get some processor cycles to do its own not-so-time-critical “updating”. Touch response speed should take top priority but sometimes a widget may want to do other things in the background. I propose the main while loop should call update on each widget periodically when it doesn’t have any urgent touch events to send out. (I guess this “background update” work could alternately use the async/wait structure but I’m not sure if this may be confusing for new users. I haven’t tried it so I can’t speak from experience.)

Widgets are responsible for not “hogging” the processor cycles.

I think that's a great approach, especially for the embedded space!

At the same time, we need to make it easy enough for people just to use, and easy enough that they just do the things, with not too much lines of code.

👍, I very much agree. Have you heard of the design principle called progressive disclosure?

If this means: Let users know the basics they absolutely need to know up front but hide complexity until they want to do something totally specialized. If so, then I’m all for it.

adafruit / Adafruit_CircuitPython_DisplayIO_Layout