[Enhancement][hotreload][P0] Add an overlay layer on Application, for In-App Selection Mode support

[chabiss]

Summary

VS would like to implement element selection behavior for Android/iOS. We need a way of both intercepting mouse events and drawing adorners over existing Xamarin applications.

API Changes

var grid = new Grid();
Application app = ...;
application.OverlayLayer = grid;

It might be desirable to use something less obvious or make it clear users shouldn't use the layer themselves. If users start using the OverlayLayer in their own apps we will conflict with them.

For example, this is the API VS uses for Uwp. For Wpf VS places a semitransparent HWND over the user's application. This solution isn't possible on Xamarin.

Intended Use Case

VS will use the overlay layer to intercept mouse events when in selection mode.

[Redth]

Tentatively @rmarinho has been spiking on an API that could look something like this:

interface IAdornerService {
    Rectangle GetRectForView(IView view);
    IView? GetViewAtPoint(Point point);
    ILayout AdornerLayer { get; }
}

[rmarinho]

Hi, I have something I worked a little bit for now , right now we don't have all the bits on MAUI to give you a absolute position using the maui controls and layout.

Right now we could provide a native "Overlay" for adding the adorners, for iOS it would be a CALayer for the window, for Android a ViewGroupOverlay.

We will provide a service that could be requested from our ServiceProvider, we would provide a interface like ...

To get the frame/rect for the adorner the IView provides a Frame

public interface IAdornerService
{
    void HandlePoint(Action<Point> onTouchDown);
    IView? GetViewAtPoint(IWindow window, Point point);
    void ClearAdorners();
#if __ANDROID__
    Android.Views.ViewOverlay? GetAdornerLayer();
#elif __IOS__
    CoreAnimation.CALayer? GetAdornerLayer();
#endif
}

Example of usage:

//Get the adorner service
var adornerService = MauiApplication.Current.Services.GetService<IAdornerService>();
//Get the overlay 
var parent = (ViewGroupOverlay)_adornerService.GetAdornerLayer();

// Handle touches on the UI
_adornerService.HandlePoint(point =>
{
    _adornerService.ClearAdorners();
    AddAdornerAtPoint(MainLayout, point);
});

//Add the adorner to the overlay
void AddAdornerAtPoint(Microsoft.Maui.ILayout layout, Point point)
{
    var view = _adornerService.GetViewAtPoint(layout, point);
    if (view == null)
        return;

    var rect = view.Frame;

    Android.Views.View nativeView = GetAdorner(
                (int)this.ToPixels(rect.Location.X),
                (int)this.ToPixels(rect.Location.Y),
                (int)this.ToPixels(rect.Size.Width),
                (int)this.ToPixels(rect.Size.Height));

    parent.Add(nativeView);
}

//Get a native adorner shape
Android.Views.View GetAdorner(int x, int y, int width, int height)
{
    var native = new ImageView(this);
    var shape = new Android.Graphics.Drawables.GradientDrawable();
    shape.SetStroke(2, Color.Red.ToNative());
    native.SetImageDrawable(shape);
    native.Layout(x, y, x + width, y + height);
    return native;
}

A small of video of a POC working https://user-images.githubusercontent.com/1235097/114227834-6df21a00-996d-11eb-8eff-eff9afe27907.mov

[maxbrister]

@rmarinho What coordinate space will the adorner layer be in? To draw adorners around a view we'll somehow need to map the views frame into the coordinate space of the adorner layer.

Which window is the adorner layer on? It looks like GetViewAtPoint takes in a window, but GetAdornerLayer doesn't.

[rmarinho]

Hi @maxbrister right now i played with making the anorder layber in native coordinate space , to map from a view frame to native coordinate space in android you just have to convert the View frame positions using the ToPIxels extension , on iOS the frame will match to the coordinate space.

Yes you are correct GetAdorner needs to take a IWindow..

public interface IAdornerService
{
    void HandlePoint(Action<Point> onTouchDown);
    IView? GetViewAtPoint(IWindow window, Point point);
    void ClearAdorners();
#if __ANDROID__
    Android.Views.ViewOverlay? GetAdornerLayer(IWindow window);
#elif __IOS__
    CoreAnimation.CALayer? GetAdornerLayer(IWindow window);
#endif
}

[rmarinho]

One thing missing is maybe some way to clear the adorners when for example you scroll , how does it work on Windows?

Do we need to expose a HandleGesgture and allow you to make decisions base on that?

[maxbrister]

On Windows we capture all input when adorners are enabled, so you can't really scroll.

It might make sense to just invalidate each added adorner if there is any input. That way the adorner can redo layout.

[drasticactions]

VisualDiagnosticOverlay and the needed APIs were implemented in Preview 11. We should be good to work on this.