Open spkersten opened 5 years ago
This is something I looked at fairly extensively when we refreshed the material widget catalog to meet android accessibility guidelines. Specifically, Android recommends at least 48 by 48 minimum hit-testable area (and iOS 44 by 44).
One of the first solutions I looked at was something similar to what you've proposed above. The problem is - how do you adjust hit testing to work without requiring a single gesture to hit test every single render box? Once the layout bounds and the hit testable bounds don't line up you always need to walk the tree.
Ultimately, what we found is that making a visual element's hit testable bounds larger without increasing the size of the element is actually the hack, and not the other way around. While some wiggle room is okay, It's not intuitive if the two don't agree. The correct solution is to adjust the designs and UX.
If the widget needs small adjustments, use padding. This is the approach we've taken with many of the material widgets. This can be configured via https://docs.flutter.dev/flutter/material/MaterialTapTargetSize-class.html in the theme, (so that we can remove the padding on desktop platforms).
(Caution) you may increase the size of a child hit testable region to at least the size of a parent. This is done for the dismissible section of the material chips. The render object that wraps the chip redirects taps from the right third of the chip to make it easier to tap. https://github.com/flutter/flutter/blob/0bf0f5c1dad62ef40efd248ff9e442bed4764e74/packages/flutter/lib/src/material/chip.dart#L1719
Oh, and the other issue with increasing hit test area: it's not possible to guarantee that these hit test regions do not overlap, unless we did another layout pass
@jonahwilliams In the cases we would need this, the parent is limiting the size the child can be expanded to, so your enumerated solutions don't work there. Saying the problem is the UI design sounds a bit too easy for me. There are cases where element are far enough away in the design, but where the natural structure of widgets make it hard to expand tappable widgets. For example cards in a sliver grid with axis spacing: Making cards invisibly bigger and reducing the spacing is an awkward solution.
how do you adjust hit testing to work without requiring a single gesture to hit test every single render box? Once the layout bounds and the hit testable bounds don't line up you always need to walk the tree.
Widgets could hitTest their children when the event happens within their bounds extended by a small margin (44/2 or 48/2). Or widgets could report a sizeForHitTesting upstream besides the normal size.
Being able to modify the hitTestArea of a widget would also solve the issues with OverflowBox, where anything in the "overflow" area can't receive any hitTest events, since it's outside of the parent's boundaries.
Kind of hack, but works well. The main idea is: add a Member into GestureArena.
import 'package:flutter/gestures.dart';
import 'package:flutter/material.dart';
import 'package:flutter/rendering.dart';
import 'package:flutter/widgets.dart';
void main() => runApp(MyApp());
class MyApp extends StatelessWidget {
@override
Widget build(BuildContext context) => MaterialApp(home: HitTest());
}
class HitTest extends StatelessWidget {
@override
Widget build(BuildContext context) {
return Material(
child: Column(
crossAxisAlignment: CrossAxisAlignment.stretch,
mainAxisAlignment: MainAxisAlignment.spaceAround,
children: [
SizedBox(
width: 100,
height: 100,
child: GestureDetector(
onTap: () => print('Tap Yellow'),
child: Container(
alignment: Alignment.bottomCenter,
color: Colors.yellow,
height: 100,
width: 100,
child: ExpandedHitTestArea(
child: Container(width: 20, height: 10, color: Colors.red),
onTap: () => print("I'm hit! I'm hit!"),
),
),
),
),
Container(
height: 100,
color: Colors.black12,
child: Padding(
padding: const EdgeInsets.all(8.0),
child: Column(
crossAxisAlignment: CrossAxisAlignment.start,
children: <Widget>[
ExpandedHitTestArea(
child: Text("<"),
onTap: () => print("Tapped"),
),
SizedBox(height: 20),
Text("Title"),
],
),
),
),
],
),
);
}
}
class ExpandedHitTestArea extends SingleChildRenderObjectWidget {
final VoidCallback onTap;
ExpandedHitTestArea({
Key key,
Widget child,
this.onTap,
}) : super(key: key, child: child);
@override
RenderObject createRenderObject(BuildContext context) => RenderExpandedHitTestArea(onTap: onTap);
}
class TestGestureArenaMember extends GestureArenaMember {
TestGestureArenaMember({
this.onTap,
});
final VoidCallback onTap;
@override
void acceptGesture(int key) {
if (this.onTap != null) {
this.onTap();
}
}
@override
void rejectGesture(int key) {}
}
class RenderExpandedHitTestArea extends RenderBox with RenderObjectWithChildMixin<RenderBox> {
RenderExpandedHitTestArea({
@required this.onTap,
});
final VoidCallback onTap;
@override
void performLayout() {
child.layout(constraints, parentUsesSize: true);
size = child.size;
}
@override
void paint(PaintingContext context, Offset offset) {
if (child != null) {
final BoxParentData childParentData = child.parentData;
context.paintChild(child, childParentData.offset + offset);
}
}
@override
bool hitTestSelf(Offset position) {
return true;
}
@override
void handleEvent(PointerEvent event, BoxHitTestEntry entry) {
if (event is PointerDownEvent) {
TestGestureArenaMember member = TestGestureArenaMember(onTap: onTap);
GestureBinding.instance.gestureArena.add(event.pointer, member);
} else if (event is PointerUpEvent) {
GestureBinding.instance.gestureArena.sweep(event.pointer);
}
}
@override
bool hitTest(HitTestResult result, {Offset position}) {
const minimalSize = 44;
final deltaX = (minimalSize - size.width).clamp(0, double.infinity) / 2;
final deltaY = (minimalSize - size.height).clamp(0, double.infinity) / 2;
if (Rect.fromLTRB(-deltaX, -deltaY, size.width + deltaX, size.height + deltaY).contains(position)) {
result.add(BoxHitTestEntry(this, position));
return true;
}
return false;
}
}
Kind of hack, but works well. The main idea is: add a Member into GestureArena.
import 'package:flutter/gestures.dart'; import 'package:flutter/material.dart'; import 'package:flutter/rendering.dart'; import 'package:flutter/widgets.dart'; void main() => runApp(MyApp()); class MyApp extends StatelessWidget { @override Widget build(BuildContext context) => MaterialApp(home: HitTest()); } class HitTest extends StatelessWidget { @override Widget build(BuildContext context) { return Material( child: Column( crossAxisAlignment: CrossAxisAlignment.stretch, mainAxisAlignment: MainAxisAlignment.spaceAround, children: [ SizedBox( width: 100, height: 100, child: GestureDetector( onTap: () => print('Tap Yellow'), child: Container( alignment: Alignment.bottomCenter, color: Colors.yellow, height: 100, width: 100, child: ExpandedHitTestArea( child: Container(width: 20, height: 10, color: Colors.red), onTap: () => print("I'm hit! I'm hit!"), ), ), ), ), Container( height: 100, color: Colors.black12, child: Padding( padding: const EdgeInsets.all(8.0), child: Column( crossAxisAlignment: CrossAxisAlignment.start, children: <Widget>[ ExpandedHitTestArea( child: Text("<"), onTap: () => print("Tapped"), ), SizedBox(height: 20), Text("Title"), ], ), ), ), ], ), ); } } class ExpandedHitTestArea extends SingleChildRenderObjectWidget { final VoidCallback onTap; ExpandedHitTestArea({ Key key, Widget child, this.onTap, }) : super(key: key, child: child); @override RenderObject createRenderObject(BuildContext context) => RenderExpandedHitTestArea(onTap: onTap); } class TestGestureArenaMember extends GestureArenaMember { TestGestureArenaMember({ this.onTap, }); final VoidCallback onTap; @override void acceptGesture(int key) { if (this.onTap != null) { this.onTap(); } } @override void rejectGesture(int key) {} } class RenderExpandedHitTestArea extends RenderBox with RenderObjectWithChildMixin<RenderBox> { RenderExpandedHitTestArea({ @required this.onTap, }); final VoidCallback onTap; @override void performLayout() { child.layout(constraints, parentUsesSize: true); size = child.size; } @override void paint(PaintingContext context, Offset offset) { if (child != null) { final BoxParentData childParentData = child.parentData; context.paintChild(child, childParentData.offset + offset); } } @override bool hitTestSelf(Offset position) { return true; } @override void handleEvent(PointerEvent event, BoxHitTestEntry entry) { if (event is PointerDownEvent) { TestGestureArenaMember member = TestGestureArenaMember(onTap: onTap); GestureBinding.instance.gestureArena.add(event.pointer, member); } else if (event is PointerUpEvent) { GestureBinding.instance.gestureArena.sweep(event.pointer); } } @override bool hitTest(HitTestResult result, {Offset position}) { const minimalSize = 44; final deltaX = (minimalSize - size.width).clamp(0, double.infinity) / 2; final deltaY = (minimalSize - size.height).clamp(0, double.infinity) / 2; if (Rect.fromLTRB(-deltaX, -deltaY, size.width + deltaX, size.height + deltaY).contains(position)) { result.add(BoxHitTestEntry(this, position)); return true; } return false; } }
I build a new package: expand_tap_area
increasing the size of the element is actually the hack
If something is less code, easier to change, easier to understand and more resilient, I don't really understand how it can be called a "hack". For better or worse, Material is a boxy ui, with paradigms that lend themselves well to this approach, but this assumption is being pushed too deep into the framework itself.
It's biased against fun playful UI's that break rules, and overlap content, and have things peek from behind other things, etc... which is a shame, cause Flutter has all the tools to make these sorts of UI's easily, but is handcuffed by this "in the box" thinking wrt hit detection.
Adobe AIR did this performantly, at 60fps, on an iPad 2, 10 years ago. I know that might not be completely fair comparison but it's hard to justify this limitation for performance reasons, in 2021, after 10 cycles of moores law.
Adobe AIR did this performantly, at 60fps, on an iPad 2, 10 years ago.
I would be interested in an apples-to-apples comparison here. What exactly did Adobe AIR do? What exactly was the performance of hit-testing in that environment? What's the performance of the equivalent in Flutter today on the same hardware?
Well, simply put: AIR allowed pixel perfect UI hit detection in at a reasonable cost, without caring about parent bounds or negative margins. Hit areas were as big as they looked visually (though this was easily overridden), if it was on-screen and visible you could touch it, and it didn't matter where you where in your parent's coordinates.
This is super flexible and robust, and really lets the author express themselves in whatever way they see fit, without battling the framework.
Here's a quick demo I put together, close to 1000 buttons in each. https://user-images.githubusercontent.com/736973/124080755-f7af0200-da07-11eb-92e2-40fa704a738b.mp4
Both AIR and Flutter use about 3% of my CPU at peak when doing mouse-overs for 10 seconds. The AS3 buttons are offset in negative space in their parent, just to show it does not matter.
If I drop the button count to a more reasonable 40 buttons, we see about 0.5% CPU usage, and again both mechanisms seem to have similar performance.
AS3: https://github.com/esDotDev/air-flutter-hit-tests/blob/master/air_hit_tests/src/Main.as https://github.com/esDotDev/air-flutter-hit-tests/blob/master/air_hit_tests/src/Button.as
Flutter: https://github.com/esDotDev/air-flutter-hit-tests/blob/master/flutter_hit_tests/lib/main.dart
How deep is the tree in AS3? What was its hit testing algorithm? How did it determine sibling order (e.g. when two "cousin" buttons overlap, which one gets the events)? Were all objects able to get events or only those who registered as event targets? How efficient were tree updates (e.g. reordering buttons, or moving them about such that the hit order was different)?
Right now both trees are as flat as possible. I can't speak to the internal AIR runtime algorithms. The cousin with the parent higher in the display list would render on top.
Any object can listen for mouse events:
circle.addEventListener(MouseEvent.MOUSE_OVER, dimObject);
circle.addEventListener(MouseEvent.MOUSE_OUT, restoreObject);
Unity has something similar: https://docs.unity3d.com/ScriptReference/MonoBehaviour.OnMouseOver.html
I seem to remember re-ordering being cheap if you're just doing a few items, more expensive if you're doing a full re-order of 100's of items. So for most use cases it worked good, cause you usually just want to swap some items, or pop a single child to the top of the list which would both be very cheap: https://sodocumentation.net/actionscript-3/topic/1628/working-with-display-objects
Also of note, the AS3 is doing pixel perfect hit testing, which I can't seem to do for the MouseRegion
so it's just reacting to the entire bounding box. So here AS3 is actually doing much more work as it needs to test a vector shape, while flutter can just test bounding boxes.
For flat trees, this seems doable. The problem in Flutter with this kind of approach is our trees are usually very very deep (I've seen depths in the thousands). If you're able to keep your trees relatively flat in Flutter then I suspect that you would get the same performance for hit testing. To do that basically you'd have to create your own much more powerful render objects rather than use the "many tiny widgets" approach that Flutter uses, and in that render object just make hitTest always call hitTestChildren.
Another use case of such expanded hit testing area would be when we set the offset of RenderBox
children outside of its size. This way, the children cannot be hit test.
There is a new package that attempts to solve this problem, by deferring hit-tests to a widget further up the tree, more info here: https://pub.dev/packages/defer_pointer
This lets you easily break outside the parent bounds, and set new parent bounds at any place in the tree you choose.
@esDotDev But you still increase the size of the widget, when applying padding, right? An ideal solution increases only the tap area, not the size of the widget.
No it does not increase the size of the widget, it just picks a parent widget to do the hit test (wherever you place the handler in the tree).
See here, a 100x100 box, 2 buttons are offset outside of it, they work fine: https://github.com/gskinnerTeam/flutter-defer-pointer/blob/master/example/lib/examples/simple_offset.dart
The widget only knows that someone else will handle its hit testing, and maybe it's rendering as well.
@esDotDev Thanks for the clarification, I understood your package wrong. Your package makes the widgets outside their parent clickable. Do you have an idea how your package could help in the following use case?
The picture should illustrate, that I want to increase the HitTestArea without increasing the size of the widget itself.
Should be the same idea:
SizedBox(width: 100, height: 100, child: Stack(children: [
Positioned.all(-10,
child: DeferPointer(
child: GestureDetector(onTap: ...))
]))
This btn will appear to flutter as 100px by 100px, but the hittable region is actually 120x120
I'd like to add a use-case to this. I don't know if there's a canonical approach to this with Flutter, or if the defer_pointer
package handles it.
I'm working on a resizable panel system, like all the resizable areas in IDEs and numerous other app categories.
Resize boundaries show a resize cursor when you get close to the divider. Trying to use a MouseRegion
for the divider is a problem, because the divider is only a couple pixels wide. Adding invisible MouseRegion
s along the edges of the panels isn't a great solution because the panels may have no idea where they're placed in the layout, so they don't know where to place such an invisible MouseRegion
, nor is that part of each panel's UI concerns.
In addition to this use-case, I would predict that as Flutter extends its reach on desktop and web, the need for proximity based cursor selections will increase, which seems like a sub-category of this problem.
I believe defer_pointer
should handle this fine, but this is a great example of how you don't have to go very far to immediately run into this issue.
Expanding hit sizes beyond their layout box, for easier hit detection, is definitely one of the core use cases of this technique. The Material teams says you should never do this, but pragmatic reality says otherwise. Often the simplest solution to implement and to maintain, is to allow the button to overflow it's bounds, rather than changing it's bounds.
https://pub.dev/packages/extra_hittest_area
This works for me
Since flutter framework has more or less decided not to fix this, i've come up with my own solution which is a hacky way of doing things but for now this is the best approach for me
import 'package:flutter/material.dart';
//
// Adapted from
// https://medium.com/@diegoveloper/flutter-widget-size-and-position-b0a9ffed9407
// https://medium.com/saugo360/https-medium-com-saugo360-flutter-using-overlay-to-display-floating-widgets-2e6d0e8decb9
//
// used to increase toucharea of components
// example usage:
// CustomOverLayComponent(
// padSize: Size(20, 20),
// onTap: () {
// print('on tap received');
// },
// onPanUpdate: (details) {
// print('pan update');
// },
// child: Container(
// width: 1,
// color: Colors.blue,
// )),
//
// The container will be padded to width size of 21 instead of just 1
// allowing for a larger touch area
//
class CustomOverLayComponent extends StatefulWidget {
final Size padSize;
final Function? onTap;
final Function? onPanUpdate;
final bool debugMode;
final Widget child;
const CustomOverLayComponent(
{super.key, required this.child, required this.padSize, this.onTap, this.onPanUpdate, this.debugMode = false});
@override
State<CustomOverLayComponent> createState() => _CustomOverLayState();
}
class _CustomOverLayState extends State<CustomOverLayComponent> {
final GlobalKey _globalKey = GlobalKey();
final LayerLink _layerLink = LayerLink();
OverlayEntry? _overlayEntry;
bool get isOverlayShown => _overlayEntry != null;
double top = 0;
double left = 0;
double height = 0;
double width = 0;
_getSizes() {
final RenderBox renderBoxRed = _globalKey.currentContext!.findRenderObject() as RenderBox;
final sizeRed = renderBoxRed.size;
height = sizeRed.height + widget.padSize.height / 2;
width = sizeRed.width + widget.padSize.width / 2;
}
_getPositions() {
final RenderBox renderBoxRed = _globalKey.currentContext!.findRenderObject() as RenderBox;
final positionRed = renderBoxRed.localToGlobal(Offset.zero);
top = positionRed.dy - widget.padSize.height;
left = positionRed.dx - widget.padSize.width;
}
void removeOverlay() {
_overlayEntry?.remove();
_overlayEntry = null;
}
void toggleOverlay(Widget child) => isOverlayShown ? removeOverlay() : _insertOverlay(child);
void _insertOverlay(Widget child) {
_overlayEntry = OverlayEntry(
builder: (_) => _dismissibleOverlay(child),
);
Overlay.of(context)?.insert(_overlayEntry!);
}
Widget _dismissibleOverlay(Widget child) => Stack(
children: [
Positioned(
top: top,
left: left,
child: CompositedTransformFollower(
link: _layerLink,
showWhenUnlinked: false,
offset: Offset(-(widget.padSize.width / 4), -(widget.padSize.height / 4)),
child: Container(
height: height,
width: width,
color: widget.debugMode ? Colors.red.withOpacity(0.5) : Colors.transparent,
child: GestureDetector(
// onTap: removeOverlay,
onTap: (() {
if (widget.onTap != null) widget.onTap!();
}),
onPanUpdate: ((details) {
if (widget.onPanUpdate != null) widget.onPanUpdate!(details);
}),
),
),
),
),
// child,
],
);
@override
void initState() {
super.initState();
Future.delayed(const Duration(milliseconds: 0), (() {
_getSizes();
_getPositions();
toggleOverlay(widget.child);
}));
}
@override
void dispose() {
removeOverlay();
super.dispose();
}
@override
void didChangeDependencies() {
removeOverlay();
super.didChangeDependencies();
}
@override
Widget build(BuildContext context) {
return CompositedTransformTarget(
key: _globalKey,
link: _layerLink,
child: widget.child,
);
}
}
Flutter needs to support this io. Hit test will be failed after transformation when the Box goes outside. I am making a physic engine using flutter. Elements are transformed using translate and rotationto display correctly. I can't find a way to make it work with gesture detection.
Any news here?
@matthew-carroll did you find a solution for this?
MouseRegion
s onHover
and onEnter
happen only when cursor is exactly on the 1px divider which is not ideal. Making the divider larger than 1px is not a solution, it takes unwanted real estate.
Secondly, when we use the GestureDetector
s onPanUpdate
to slide/move the panels, onExit
fires and user loses the visual clue (I make the divider different color on onHover)
I'm working on a resizable panel system, like all the resizable areas in IDEs and numerous other app categories.
Resize boundaries show a resize cursor when you get close to the divider. Trying to use a MouseRegion for the divider is a problem, because the divider is only a couple pixels wide
@aytunch I never found any built-in solution. I think I paused my work on the resizable panel system. I'll probably come back to it.
I think my approach at this point would be to place an invisible hit area in the app Overlay
, which is aligned with the divider widget that changes color when you hover over it.
This alignment might be accomplished with CompositedTransformTarget
and CompositedTransformFollower
. The Flutter Bounty Hunters also has our own version of those widgets that add more control over positioning, called Leader
and Follower
which are available in follow_the_leader
: https://pub.dev/packages/follow_the_leader
Would you like the Flutter Bounty Hunters to build a package specifically for resizable panels for desktop apps? If so, would you like to collaborate on that?
@matthew-carroll thanks for the detailed explanation. follow_the_leader
has potential to mimic the behavior we are looking for. I will need to experiment to see if it covers all edge cases. Second thing I will try will be to create a render object and make the HitTest
larger than the RenderBox
. I will write my findings if I have any luck.
I am just playing around with Flutter desktop to create a panel system. This is harder than I think, but Flutter community could greatly benefit from it. I can think of so many desktop apps being built on top of such a design system (code editors, image/video/audio editors, browsers, tools, etc)
Well, simply put: AIR allowed pixel perfect UI hit detection in at a reasonable cost, without caring about parent bounds or negative margins. Hit areas were as big as they looked visually (though this was easily overridden), if it was on-screen and visible you could touch it, and it didn't matter where you where in your parent's coordinates.
This is super flexible and robust, and really lets the author express themselves in whatever way they see fit, without battling the framework.
Here's a quick demo I put together, close to 1000 buttons in each. https://user-images.githubusercontent.com/736973/124080755-f7af0200-da07-11eb-92e2-40fa704a738b.mp4
Both AIR and Flutter use about 3% of my CPU at peak when doing mouse-overs for 10 seconds. The AS3 buttons are offset in negative space in their parent, just to show it does not matter.
If I drop the button count to a more reasonable 40 buttons, we see about 0.5% CPU usage, and again both mechanisms seem to have similar performance.
AS3: https://github.com/esDotDev/air-flutter-hit-tests/blob/master/air_hit_tests/src/Main.as https://github.com/esDotDev/air-flutter-hit-tests/blob/master/air_hit_tests/src/Button.as
Flutter: https://github.com/esDotDev/air-flutter-hit-tests/blob/master/flutter_hit_tests/lib/main.dart
Adobe did it 15 years ago and Flutter still can't do it
It would be useful to be able to expand the hit test area of a widget. Often, the size of a widget can't be increased without ugly hacks or at all, while the current size is too small to make it easily tappable.
The code below contains two examples where this would be useful (and a failed attempt to achieve this). In the first, the red container might be the thumb of a slider at the bottom of an app bar or a handle at the top of a panel for resizing it. In the second case, the button with "<" is inside several widgets to align it with "Title" below it. (In our code, there are several more layout widgets between the padding and "<" to handle animations) In both cases, increasing the size of the widget inside the gesture detector, while keeping the layout visually the same, would lead to a awkward layout.
The
ExpandedHitTestArea
widget is an attempt to achieve an expansion without affecting the size. However, it doesn't work because all parent widgets don't hit test their children when the hit point is outside the parent.