[Experiment] Cross Frameworks Controls

[jsuarezruiz]

What if we can create drawn controls and be able to use them independently of the UI framework? In this way, a control for example DataGrid created for WinUI could be used in .NET MAUI or perhaps Avalonia without problems.

What would we need?

• Create libraries for WinUI, Avalonia etc where GraphicsView (view that exposes the Canvas where to draw) is implemented in each platform using the best option (WinUI, SkiaSharp etc).
• Each framework has different options for expressing bindable properties. We need to resolve this point. There would be interesting options such as generating code etc.

What is the benefit? Imagine that a drawn DataGrid control is created in the Windows Community Toolkit. Voila, having it effortlessly in the .NET MAUI Community Toolkit (even sharing the code) and other frameworks would be just amazing!.

[RChrisCoble]

The icing on that cake is Microsoft.Maui.Graphics support in Blazor WebAssembly in the June MAUI release. The controls in this repo will have serious reach on so many platforms.

[robloo]

Implementing a control once and then sharing it among all UI frameworks would be wonderful. We have all written and re-written the exact same control so many times over the years it's a lot of wasted time. That said, you can't do this level of abstraction with ONLY a graphics stack. Granted, the graphics stack is a lot of it (a composition layer like in WinUI would go a long way to write once).

The other missing pieces are pretty major: Binding, Event model (INotifyPropertyChanged, etc), Input, Styling, Localization, Popups/Windowing, Text formatting, etc. A control needs all of these pieces which have nothing to do with just drawing itself. Microsoft has gone a lot of different directions over the years and WPF/Silverlight/UWP/WinUI/Xamarin.Forms/MAUI now have differences in all layers.

As I've thought about this same problem over the years the best idea I had is to create a 'Control Description Language' and then compile that for each framework independently. The 'Control Description Language' would have abstraction layers for all of the pieces mentioned above.

The end goal would be to create a packaged control that could be dropped into a framework just like a component is dropped into a system modelling application. For example, all models are written in the heavily math/physics based Modelica language which is commonly shared among physics-based simulation software. Modelica is so powerful it can model all domains -- electrical/thermal/mechanical/etc. A 'Control Description Language' must be equally powerful.

Bottom line: I don't think you realize how complicated this would be and don't think it's possible with your ideas. You need all features of the UI framework to implement a control. It would have been better to standardize on UWP.

[jsuarezruiz]

Well, you are right that my proposal it is not as detailed as it should, I will update it.

In each platform, in addition to allowing drawing, it would be necessary to have an abstract API to manage interaction events (touch, etc this already is implemented). You are right that it would also take quite a few more options in the abstraction layer to cover other details. Although I think that, being able to define a control in an abstract way with its properties, styles, interaction, it would allow many cases and it would already be an interesting proposal, it would not cover everything, more options would be needed in the abstraction layer. Regarding differences such as DependencyProperty vs BindableProperty etc my idea was to use source generators to allow defining a property, marking it and generate the necessary code for each platform.

It is certainly a complex idea that requires a much more detailed Spec, I will update it. Thanks for the feedback :)

[robloo]

This area has always been an interest of mine. Rewriting the same control a few times will tend to start one thinking about what can be done differently :)

Regarding differences such as DependencyProperty vs BindableProperty etc my idea was to use source generators to allow defining a property, marking it and generate the necessary code for each platform.

I have the least concern about this :) WinUI is already generating property code automatically from IDL files.

It is certainly a complex idea that requires a much more detailed Spec, I will update it. Thanks for the feedback :)

I think my overall point is simply that all UI frameworks started with a similar concept of sharable controls. Whether or not it's cross-platform or cross-framework is actually irrelevant to the point I'm making. In the pursuit of enabling fully featured, shareable controls, UI frameworks inevitably require a lot of features that grow into... well... the UI framework itself. In other words you need the full complexity of a UI framework like UWP to write fully-featured/sharable controls. So my point is spec-wise you should be scoping to include almost all areas of UWP itself -- a task much bigger than one person can do as well. In the end though, perhaps the correct approach all along is simply to use the UI framework syntaxes and conventions that already exist -- i.e. UWP like the Uno Platform. This was really the point of UWP all along.

With massive amounts of code generation, abstraction layers and a control description language all of your goals should be possible. I'm just not convinced yet it's actually practical. Plus in my mind we want to avoid:

Let's also not forget Maui.Graphics.Controls is a direct result of the difficulty maintaining a UI framework that uses native implementations of each platform. It is better to do things only once and only very low-level (input, pixel rendering) is per-platform. So by proposing this you are ironically repeating some of the same mistakes Xamarin Forms made that you are working to fix by Maui.Graphics.Controls.

Bottom line: I don't mean to discourage this idea and it is possible. However, it requires a seriously committed team and almost perfect strategy to execute successfully. It would take an all of Microsoft effort between the various teams (WPF,WinUI,Maui). If successful though, it would push UI development forward a generation, just like WPF did back in the day.

[ToolmakerSteve]

I agree with @robloo. Actually, I have a stronger view: Not worth the effort to attempt. Much harder than you think.
"Opportunity Cost": If people attempt to do that, what useful project could they have been focused on instead?

I see no reason to tackle yet another can of worms, trying to deal with different varieties of XAML. That's going in the wrong direction, imho.

---

The most important goal is being able to use ONE AWESOME standard everywhere. Without all the issues XForms has, trying to use native controls.

.Net just needs MAUI to have the option of using this "render to bitmap" approach EVERYWHERE. No more native controls. (I say this, having spent years working with Xamarin Forms. See the 3000+ open issues in github.) That could be reliable TODAY, while MAUI team is still struggling to fix open issues using native controls. (Assuming the renderer refactoring has been finished.)

Demonstrate that. Show that MAUI could (if you want it to) use Graphics.Controls to render to a bitmap. 100% identical in all environments. Ignoring the UI widgets available in the local environment. Side-stepping the differences in user interaction mechanisms in the different environments.

Most importantly, show that working in a browser. That's where Microsoft's offerings are weakest vs. the competition (Flutter; ReactJS).

---

Najak3d and I started to make a demo showing MAUI as a GUI on top of a 3D game engine. (First Urho3D, with Unity planned next.)

But we are stopped by this issue.

If I can't get the basics to work, then there is no point spending time with these technologies, until the moving parts (.net 6, Windows Desktop Apps, MAUI, VS 2022) come together, sometime next year.

Its a shame. I would love to help show the full potential of "run identically everywhere".

[listepo]

@jsuarezruiz I think we need partial properties I tried to use private fields, but not used fields do not look very nice for example:

// source
public partial class TestView : ContentView
{
    [CrossProperty]
    private readonly string? title; // unused
}

// generated
public partial class TestView
{   
    public static readonly BindableProperty TitleProperty = BindableProperty.Create(nameof(Title), typeof(string), typeof(TestView), null);

    public string Title
    {
        get => (string)GetValue(TitleProperty);
        set => SetValue(TitleProperty, value);
    }
}

can you have any thoughts?

[hintsofttech]

create drawn controls and be able to use them independently of the UI framework?

will custom drawn controls support to web rendering directly or not ? without web rendering how can be this crossplatform ?

[Suriman]

@jsuarezruiz, @hintsofttech's question is very important. Any answer regarding it?

[jsuarezruiz]

@jsuarezruiz, @hintsofttech's question is very important. Any answer regarding it?

Maui.Graphics has a WebGL implementation https://github.com/dotnet/Microsoft.Maui.Graphics/tree/main/src/Microsoft.Maui.Graphics.Blazor and GraphicsControls makes use of this library for drawing. I have done small tests (related with other platforms, like Linux etc). It would be necessary to create and map what is necessary to launch .NET MAUI Apps on the web and complete some implementations beyond the Canvas where draw (touch events, etc). It could be a technically possible option. However, this library is experimental. The main goal right now is to validate the interest. The progress I make is mainly oriented to keeping everything up to date working, fixing bugs and/or solving the questions and topics related to its validation (performance, accessibility, etc). So the answer is, yes, it could reach the web, linux or other platforms but at the moment it is an experimental library and the efforts are now more oriented towards its validation.

[Suriman]

Thank you very much, that's good news.

[charlesroddie]

That said, you can't do this level of abstraction with ONLY a graphics stack.

I don't agree with this at all.

The other missing pieces are pretty major: Binding, Event model (INotifyPropertyChanged, etc), Input, Styling, Localization, Popups/Windowing, Text formatting, etc. A control needs all of these pieces which have nothing to do with just drawing itself. Microsoft has gone a lot of different directions over the years and WPF/Silverlight/UWP/WinUI/Xamarin.Forms/MAUI now have differences in all layers.

Binding/Event model: not the responsibility of a graphics layer. Dotnet has the ability to allow any binding approach to be specified, in a completely type safe way (i.e. much better than the platforms you mention which all use a pre-generics approach). The graphics layer doesn't need to know about it. It only needs to define settable properties (if mutation is the approach).

Input: An interesting one. We are working on keyboard input but the options are either to draw the keyboard or to allow a user to plug in a native keyboard.

Styling/localization/text formatting: the drawn control chooses what options to give the user; no limitations here.

Popups/Windowing: not the responsibility of drawn controls. Drawn controls only render things inside these.

[robloo]

You can disagree all you want but I know what a graphics stack should be. If you throw everything into it, it's not a graphics stack anymore and it is instead a framework where you are breaking separation of concerns.

Its good to be optimistic. But looking at what Maui is today, what Microsoft has done in the past, the resources you have on the team, etc I highly doubt you will come anywhere close to this proposal.

Prove me wrong and I'll engage more. Otherwise I'm stepping away so as not to waste time.

Finally, introducing generics in more places is neat and there are certainly some areas that can be modernized.