Hosting Avalonia in WebAssembly

[itsamelambda]

Would it be possible to host Avalonia with WebAssembly and .NET in a browser?

Then we could write the same C# code and run it native in all browser and on all desktops.

[kekekeks]

It would be possible and we are currently considering our options. The major blocker is that HTML5 doesn't have a proper 2D drawing API, so Skia needs to be compiled to wasm and somehow linked to resulting application. The problem with linking is that Mono doesn't currently provide P/Invoke support for wasm target. Another problem is that P/Invoke means indirect calls (i. e. calls via pointer), while WebAssembly doesn't support indirect calls between modules. That means that Skia needs to be linked statically, so we'll need a custom Mono build.

It would be easier with CoreRT, since it supports static linking out of the box, but they are still implementing MSIL instructions for wasm target, so it's not ready even for an experimental port.

[legistek]

Sorry if this is an obviously terrible idea, but why wouldn't you use WebGL?

[danwalmsley]

Maybe this helps? https://github.com/aspnet/Blazor/blob/dev/README.md

[kekekeks]

@pmoorelegistek We will be using WebGL at some point. The problem is that WebGL by itself doesn't support drawing 2D vector graphics. So we need Skia anyway.

[legistek]

Oh I see what you're saying. I didn't realize it lacked basic 2D drawing. I take it the plain HTML5 canvas doesn't have good enough performance either?

I've been watching this project for awhile and would love to help how I can. Getting this on the web would be the Holy Grail. (Befitting the name Avalon(ia))

Get Outlook for Androidhttps://aka.ms/ghei36

From: Nikita Tsukanov Sent: Friday, March 9, 12:30 AM Subject: Re: [AvaloniaUI/Avalonia] Hosting Avalonia in WebAssembly (#1387) To: AvaloniaUI/Avalonia Cc: Peter N. Moore, Mention

@pmoorelegistekhttps://github.com/pmoorelegistek We will be using WebGL at some point. The problem is that WebGL by itself doesn't support drawing 2D vector graphics. So we need Skia anyway. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/AvaloniaUI/Avalonia/issues/1387#issuecomment-371725952, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AMw7S1GOycAJX3aLJnlzy5xM6yq_KrTiks5tciGKgaJpZM4SR4pe.

[legistek]

Hmm. The problem with compiling Skia to WASM is you'll lose any benefits of hardware graphics acceleration on the device. Browser will be doing all the math, painting pixels, etc. I don't know that you'd ever get anything close to 60fps with a complex UI.

Have you tried just using the HTML5 Canvas or SVG?

[kekekeks]

Skia can use OpenGL ES profile for hardware-accelerated rendering. Which is basically what is provided by WebGL. HTML5 Canvas and SVG are not sufficient to render Avalonia.

[legistek]

You're saying Skia / OpenGL will be able to access hardware acceleration (i.e. the GPU) after compiled to WASM? Are you certain about that? Everything I understand about WASM would lead me to think not.

I would think you'd be better off using three.js - or better yet porting it to C# - wrapped around WebGL.

I'd be happy to look into it some more and experiment. Do you have a working prototype using Mono-to-WASM at the moment with any rendering at all?

[kekekeks]

You're saying Skia / OpenGL will be able to access hardware acceleration (i.e. the GPU) after compiled to WASM

Emscripten toolchain has complete support for WebGL. It might be a bit difficult to make it work with Mono since it has it's own js glue code, but other than that I don't see any issues.

[danwalmsley]

https://github.com/praeclarum/Ooui/wiki/Xamarin.Forms-with-Web-Assembly

No need for Skia now,

[kekekeks]

@danwalmsley it uses HTML. Avalonia can't be rendered with HTML.

[adoris]

Is there are any try to use Skia/WebGL? Any branch for this experiment?

[legistek]

if I understand right, what @kekekeks is saying is that it's not feasible at the moment because Mono can't use P/invoke to call the Skia functions. So while they can compile Skia to WASM, there would be no way for the .NET code to access it.

What I don't understand fully, though, is why it has to be Skia. There are Javascript libraries that provide 2D drawing function wrappers over WebGL (e.g., https://threejs.org/). I'd think it at least worth an experiment to make a DrawingContext that wraps three.js and go from there. If that works, porting threejs to C# and simply including it in the Avalonia build would not be too bad of a project and would probably get you performance on par with Skia.

[kekekeks]

@pmoorelegistek The main problem is the complete lack of a proper text measurement API. We can't get our TextBox to work if we don't have an accurate position of each individual symbol. Even for our TextBlock we need to be able to measure and lay out text lines properly. There is no way we can do that with the APIs available to JavaScript.

[yowl]

I added an issue for pinvoke support on mono so this can be tracked. https://github.com/mono/mono/issues/8007

[TonyHenrique]

See http://www.noesisengine.com/webgl/Samples.Buttons.html

[legistek]

It looks nice but what are we looking at exactly?

---

From: TONY HENRIQUE notifications@github.com Sent: Thursday, April 12, 2018 6:38:15 PM To: AvaloniaUI/Avalonia Cc: Peter N. Moore; Mention Subject: Re: [AvaloniaUI/Avalonia] Hosting Avalonia in WebAssembly (#1387)

See http://www.noesisengine.com/webgl/Samples.Buttons.html

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/AvaloniaUI/Avalonia/issues/1387#issuecomment-380976051, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AMw7S61hESqCLDWhVkG6jvegNBn7KHWtks5tn-VngaJpZM4SR4pe.

[kekekeks]

Noesis has it's own rendering engine, if I recall correctly. A proprietary one. We don't have the resources to create a whole new 2D graphics library, so we are using an existing one, which is Skia. Until we have a way of linking it in on wasm target (be it Mono or CoreRT), we won't be able to get Avalonia working in the browser.

[legistek]

Hey Nikita - since this thread keeps coming alive - I looked deep into three js and the code for text rendering could easily be adapted for text measuring. I'd be willing to make the modifications if you're willing to consider something besides skia.

And beyond that, I don't think a port of three js to C# - or at least enough of it for our purposes - would be at all that difficult.

This is my company, and as you can see from our product we have a vested interest in your project: www.legistek.com

[kekekeks]

Well, you see, the only text measurement API that I've found for HTML5 Canvas is measureText call which returns a TextMetrics object. The problem is that it doesn't support anything but text width:

As you can see, "advanced" properties are only supported on Google Chrome and need to be explicitly enabled in browser settings. Getting text width is unfortunately not enough for implementing our IFormattedText interface.

[kekekeks]

Does three.js have it's own rendering engine that can draw text using TrueType/Freetype fonts with proper subpixel rendering?

[kekekeks]

If it does, it might be worth to create some kind of a wrapper, but currently calling JS code from WASM is very inconvenient.

This is currently the only way of calling JS code from C#, which is basically an eval call.

There is some existing glue code here (JS part here) which uses the internal call from an alternative mono wasm target implementation, which is basically an eval too, so it could be adapted.

[legistek]

https://github.com/mrdoob/three.js/blob/f81506e172571ab106d0164530bbc1a4802fc2d4/src/extras/core/Font.js

They are going through the font glyphs and turning them into path geometries and then feeding them to WebGL. If they can do that then they could have measured them. It looks like nothing more than an oversight that they didn't include measuring functions. (I'm not yet clear on what font format they're expecting but my assumption would be WOFF).

re JS interop, again I would assume the endgame would be porting what we need from three.js to C# so that all the calculating code would be done in WASM and the only time you'd need to interop would be the final render into WebGL. But why not try a POC at least?

[kekekeks]

@pmoorelegistek It seems that Mono WebAssembly SDK is providing libmono and an example C source file with the entry point. That means that we can link additional libraries to the resulting runtime mostly without issues. mono-dl-wasm.c file still needs to be changed for supporting __Internal but at least we could use existing build scripts

[jmacato]

@kekekeks I'd like to bring this lib to our attention: SharpFont; could be useful for text rendering, alongside the PixelFarm renderer that i linked before.

I just noticed that PixelFarm is already using this particular library, so yeah

[birbilis]

Could be of some use: https://github.com/mattdesl/fontpath (convert Fonts to paths) http://paperjs.org/features/ (this seems to use Canvas, so not sure how fast it is)

I believe Three.js is an interesting option (maybe combined with FontPath above)

[birbilis]

also looked into PixelFarm mentioned above, and it does seem to use a Typography module and support various renderers (including OpenGL ES I think) https://github.com/PaintLab/PixelFarm https://github.com/LayoutFarm/Typography https://github.com/LayoutFarm/HtmlRenderer

[insinfo]

NanoVG, maybe this is an excellent OpenGL-accelerated 2D API option to be ported to WebGL, it's a small API with several ports of it for other BackEnds. https://www.youtube.com/watch?v=c8IsoDQ7ea8 https://github.com/memononen/nanovg https://github.com/cmaughan/nanovg https://github.com/ollix/MetalNanoVG https://github.com/bkaradzic/bgfx/tree/master/examples/20-nanovg https://github.com/ocornut/imgui https://github.com/wihu/NanoGUI-Mono https://github.com/wjakob/nanogui https://sites.google.com/site/bitiopia/ https://github.com/cambiata/NanoVGDotNet https://github.com/jpbruyere/Crow https://github.com/jpbruyere/vkvg

[kekekeks]

NanoVG is a C library. So we are back to the square one where we need to somehow enable native code interop. I'll try to get Skia# working on top of WebAssembly first before trying managed alternatives (which will be super-slow because Mono currently runs managed code using interpreter)

[kekekeks]

BTW, we can't use LGPL code (nanovgdotnet is LGPL-licensed) for WASM target since everything will be linked to a single file once Mono gets a proper compiler instead of interpreter.

[PerArneng]

This would be so cool. I think it could be the killer app since there are not alot of options today for doing that. Some demos from other UI toolikts running in webassembly in the browser:

• https://danilw.github.io/GLSL-howto/nanogui/nanogui.html
• https://pbrfrat.com/post/imgui_in_browser.html

[jpbruyere]

canva and SVG are the browser's 2d api's.

[galvesribeiro]

There are some people working Skia for wasm https://github.com/Zubnix/skia-wasm-port but like @kekekeks said, you need a custom built mono liked with skia... I have canvas APIs wrapped for Blazor https://github.com/BlazorExtensions/Canvas using the CanvasRenderingContext2D APIs. So, it wouldn't be hard to use it but, the subpixel measurement required by Avalonia text renderer would probably not work with it...

[kekekeks]

@galvesribeiro I had some success with building libSkiaSharp last month (I even have some code that actually creates some skia types from C#) and encountered two major problems:

1) The current Mono interpreter has a hardcoded list of function signatures that can be P/Invoked. If I try to call something else it fails with errors like CANNOT HANDLE COOKIE VIIFFI. We need make that call pluggable, analyze the list of imported function signatures used by SkiaSharp and generate needed trampolines

2) WASM implementations in Chrome/Firefox have an arbitrary limit on the number of functions declared in the binary fine. I was actually hitting that limit even with incomplete (no libwebp, pdf, svg, xml) libSkiaSharp and getting "functions count of 1589554 exceeds internal limit of 1000000" error. To make it load I had to remove GPU support which will hurt performance by a lot. The problem might be caused by the fact that I'm currently adding all libSkiaSharp exports to the mono.wasm binary. We might need to integrate a linker step that removes unused DllImports, reanalyze the SkiaSharp.dll to get the list of actually used imports and only then generate a separate C file. However I'm afraid that without stripping GPU support the function count will stay way to huge.

The first problem is solvable, but we might be hitting a somewhat "fundamental" limit with the second one. I'm not sure if it's possible to reduce the function count without removing the needed features.

[kekekeks]

We might also try to resort to WASM dynamic linking to make libSkiaSharp live in a separate wasm file. That should increase the total function count limit.

[galvesribeiro]

We might also try to resort to WASM dynamic linking to make libSkiaSharp live in a separate wasm file. That should increase the total function count limit.

I'm not sure with the current wasm implementations we can do that... Last time I checked, you can't dynamic load anything neither can you link another .wasm.

[kekekeks]

According to this doc dynamic linking should be possible in MVP. We can't make DllImport work with it out of the box and have to resort to C codegen, but it should be possible.

[galvesribeiro]

Yeah, what I meant was that I'm not sure the browsers support that yet. There are discussions in some other frameworks where they can't so some stuff just because of that same problem. I may be wrong, its being a while since I don't check it...

[birbilis]

Can't a wasm file talk to javascript? If it can, that one could load other wasm(s) and act as a bridge between them

[kekekeks]

We need to at least somehow setup a shared linear memory for multiple wasm files.

[kekekeks]

https://github.com/mono/SkiaSharp/wiki/Building-on-Linux https://github.com/mono/SkiaSharp/wiki/Building-on-Linux-(LEGACY)

[kekekeks]

FYI: I've spent a weekend tinkering with WASM target and SVG render target about a month ago and decided that the current performance of Mono interpreter is way too slow. It also might be a fault of our layout algorithms or me missing something since the debugging story on WASM is currently not existing

You can see the incomplete demo here: http://testapp.keks-n.net/ don't even try to open calendar or datagrid pages, they take ages to load.

[ghuntley]

Nice work @kekekeks!

[ghuntley]

@kekekeks you might enjoy this blog post re: skia and perf. Hopefully it gives you some ideas you can incorporate. https://platform.uno/skiasharp-support-for-webassembly-via-uno-platform/ :)

[moderndev]

wondering if there's anything you can share with the mono folk on improving mono for this type of workload ?

[Gillibald]

I think for now we should closely watch the UNO project what they come up with. They are already working with the Mono team.

[kekekeks]

@moderndev the thing is, it's currently kinda pointless to work on WASM backend because of performance issues both on Mono and Avalonia sides. We could get some nice demo that would be nowhere near production-ready state (i. e. seconds per frame instead of frames per second) until those issues are fixed. That's why I was focusing on XAML compiler this year instead: it help all existing platform backends and brings us closer to WASM.

Also, as the time goes by, it's getting easier to run code on WASM and it's not that hard to write a new backend, it usually takes a week or two to get things going.

[Perksey]

Have you considered using WebGL via WebAssembly?

[kekekeks]

Yes, we did, Skia can utilize WebGL for rendering.

Right now we are focusing on optimizing the core framework, since it runs way too slow on WASM target as it is. At the same time WASM tooling getting improvements, so it will be easier to get a port later.

[legistek]

I've spent a LOT of time working with Blazor, both in server-side and WASM mode, and have several observations that may be helpful:

First, just to get the obvious out of the way, Mono WASM (even interpreted) is way fast enough today to handle the backend of a lookless UI. By that I mean dependency properties, binding, styling, etc. etc. I know because I implemented them in a new Blazor framework we've created for a proprietary project. (I'd pre-compile the XAML if possible though; I don't know if Avalonia can do that, sorry).

Second, the bottleneck with making a UI with WASM is in the interop with Javascript. For reasons that escape me, the WASM designers in their wisdom chose not to give the WASM VM direct access to the DOM. Therefore anything that touches on what the user sees in the browser has to ultimately go through JS. And, of course, this requires serializing things to JS-friendly strings (which WASM has no inherent support for, and are thus super inefficient) that JS then de-serializes back into its own object model. Big stinking mess.

So how is that problem solved? Minimize interop, obviously.

Blazor does this by using a retained mode rendering system. You build a render tree in Razor or C# (Razor compiles to C#, so same thing), Blazor serializes it and sends it to the JS runtime, which then constructs the DOM. Then Blazor diffs the render tree every time anything changes, and only sends the changes back through JS. The initial render can be slow if the DOM is very complex, though this will certainly be better if not unnoticeable with AOT compilation. Subsequent renders feel like native apps. Again, even with interpreted Mono, and a completely "lookless" render tree. Meaning for every UIElement equivalent in a WPF-type visual tree there'd be a <div> with unique styling.

Of course this is conceptually very similar to what WPF does. WPF uses a retained mode rendering compositor that is deep under the hood. When you handle OnRender all you're doing is helping to build a tree of commands that get sent to a compositor in a native DLL. The compositor is what operates at 60fps and what actually does all the heavy lifting (i.e, what WebGL or Skia would be doing).

Point being, if you're going to successfully build a web platform for Avalonia or anything else, IMHO you're going to need to build an entirely Javascript-based compositing engine and use a retained mode rendering system like Blazor or WPF. A Skia WASM port, I don't believe will get you nearly acceptable performance because there will still be too much interaction between WASM and JS. I would forget about Skia entirely, and focus on designing data structures representing a render tree that is built with C# and can be efficiently diff'd, serialized and sent to the JS runtime only when it changes. I'd design that JS compositing engine probably with ThreeJS. Again, I'd stay away from Skia or anything else that would require a lot of interop between WASM and JS.

There's zero doubt this can be done, because we know Blazor works, and that's with using plain old HTML + CSS as the compositing engine. There's every reason to think it would be even faster and more impressive with a WebGL based compositing engine. But unfortunately I think that compositing engine has to be 100% Javascript to get anything close to 60fps.