[FEATURE] Support SkiaSharp as a Blazor Extension

[RChrisCoble]

My understanding is SkiaSharp running on WebAssembly is in progress with a target release of May. This would allow a "Blazor Client" application to use SkiaSharp and Skia fully running in a supported client browser.

One highly desired feature after that would be supporting the Blazor model of having the option to run the business logic calling SkiaSharp either client or server via a "Blazor Extension". Running from the server is important for scenarios of smaller download size, thin client support, taking advantage of server capabilities, etc. I understand the performance would be quite different in the server scenario, but it would still 'work'.

An example of this concept used on an HTML5 Canvas API is here: https://github.com/BlazorExtensions/Canvas

This isn't rocket science, as the HTML5 canvas is a known API existing in the client browser, and the Blazor Extension wraps the canvas API so when running on the client, the extension simply calls the canvas API directly. When running on the server, the Extension marshals the API call to the client over the Blazor SignalR connection. There's some extra magic in that linked Extension to batch calls for performance, but you get the idea.

While I'm not an expert on the SkiaSharp architecture, my assumption is the only way this would work is have Skia still run in the client browser, and SkiaSharp (or a thin wrapper) run on the server to marshal the calls over SignalR. Alternatively (but probably impossible for this repo) is have Skia marshall the calls over SignalR and have them execute using WebGL or Canvas.

So, ultimately the feature request is once SkiaSharp can run in Client Blazor, please support SkiaSharp as a Blazor Extension.

[mattleibow]

https://github.com/mono/SkiaSharp/issues/1219#issuecomment-611157056

From @RChrisCoble:

Regarding:

[FEATURE] Support SkiaSharp as a Blazor Extension (#1194) This is what we are going to be doing here.

When it's a Blazor Extension you can utilize the SkiaSharp API on the Client or Server Blazor deployment model. I believe what you are doing here is akin to the forked Uno repo, allowing Skia/SkiaSharp to work properly when running under WASM in a client browser. My specific feature request around making it a Blazor Extension was to also allow the API to run on the server with the rendering happening on the client.

This would most likely mean Skia (and maybe SkiaSharp) would need to be running in the Client Browser, with the SkiaSharp API calls being packaged up and marshaled over the Blazor SignalR connection when the Blazor Extension was running on the server.

An example of this was done with the HTML5 Canvas here: https://github.com/BlazorExtensions/Canvas

If you look at that GitHub example, they wrote a wrapper around the HTML5 canvas API's which ended up doing one of the following: (1) If the Blazor Extension was running in a client browser, issued the draw call directly or (2) If the Blazor Extension was running on the server, marshaled the call over the SignalR connection to the client to be executed.

The Blazor server deployment model is valuable in situations where you can't deploy multiple MB's of DLL's down to the client just to draw anything. (with that multiple MB's NOT being Skia/SkiaSharp, but the business logic using the SkiSharp API.)

[mattleibow]

Seems that we might need to wait for https://github.com/dotnet/aspnetcore/issues/5466 to be completed so we can link SkiaSharp with the WAM part. Not sure about dynamic linking...

[RChrisCoble]

[EDIT]
I see this was posted under the Blazor Extension feature and not under the WASM feature. So I changed my comment here.

@mattleibow Does this mean moving forward with running SkiaSharp in a client browser under WASM is moving forward Ok, but running it as a Blazor Extension (meaning I could run it on the server as well) needs to wait on #5466

I'm hoping it's the latter. If we could run SkiaSharp directly in the client browser still that would be fine.

[mattleibow]

@RChrisCoble, no prob!

This is a mix of issues/features. Let me break it down and then maybe you can see where exactly you are needing a feature or if we need to focus on some things.

First, the real basics, SkiaSharp obviously works just fine on the server as a function/API that you can use as an endpoint for the basic <img src="/path/to/generated/image" />. This is pure server code that uses basic HTML to get the image. I assume this is not what anyone is talking about, but just for general clarity.

For the WASM/client code, this is actually 2 separate cases: Uno or Blazor.

Uno is working just fine and this is currently what the #1333 PR is for. It adds a Uno view and runs SkiaSharp right in the browser today. When ready, this will just be a matter of adding the view to the XAML/code behind and then the browser will draw it for you. Currently, Uno takes advantage of WSL to effectively build your app on Linux under the hood and produce the working website.

Blazor is a bit different as it does not yet support the feature to statically link the app with SkiaSharp libraries. This is what https://github.com/dotnet/aspnetcore/issues/5466 is all about - basically add the features to compile the WASM bits with static libraries. It appears (at the time of writing) that we have to wait for .NET 6 next year or so before we get this feature.

The difference between Uno and Blazor is the app model. Uno is basically UWP apps, but in the browser. Blazor is an ASP.NET Core Razor website, but in the browser.

[mattleibow]

As I write the comment above, there is the fact that Blazor also has the server components. Now, that might just be possible today because the code actually runs on the server and new png/jpg files are sent to the client. I have not investigated this case at this time, but might be useful to look into if it actually helps in the case what you are looking for. I am assuming you are more wanting the code to actually run on the client browser.

[RChrisCoble]

@mattleibow Thanks for that detailed description, that helped me considerably. Now I understand how Uno is able to statically link to make this work, compared to Blazor which needs AOT support which was dropped from .Net 5.

We have a C#/.Net engine we wanted to run in the client browser directly, using SkiaSharp to render. That would have been the ideal solution for speed/performance. It sounds like the only avenue to that end right now is leveraging Uno/SkiaSharp which we'll have to evaluate once #1333 PR comes in.

That being said, this engine participates in a framework of other components that often have to run on the server right now because of how they were built. If we could use a SkiaSharp "Blazor Extension" that only supports server side execution (for now), with the forward looking statement that the extension would someday be supported using Client WASM enabled via .Net 6 AOT compilation, that would be valuable.

[EydenJones]

So does mean that being able to draw using skiasharp in a client side blazor app isn't going to happen until late 2021? (release date .net 6 which will include AOT compilation) :-(

[mattleibow]

Unfortunately, that looks to be the case, but I can assure you the team is working on this as fast as they can. I'm in the chats and I see all the good words being thrown around.

Not sure when exactly things will go out, but hopefully sooner than later - even if a preview. I think previews usually start early in the year, so it is quite soon.

[Flippsor]

@mattleibow great work so far! Sad to hear that we have to wait for AOT compilation from mono. I already tried the Canvas/WebGl Blazor extension, but it is not very performant due to JS Interop calls. I also tried the .NET binding for WEBL through Webassembly (https://github.com/WaveEngine/WebGL.NET) in Blazor and got it working (see screenshot). Not quite sure how they do the binding to Webassembly in detail, but maybe it is possible with Skiasharp, too?

Edit: Maybe Webassembly.Bindings can help?

[mattleibow]

I had a look and things still have a sad thing. As you saw, going from .NET to JS to C++ is not the best. Especially with the number of calls being made. The WebGL part is not too bad as it is just talking directly to JS. Unfortunately, the libSkiaSharp lives in C++, so we have to hop the bridge twice.

Thinking wildly here, it might be possible to create a Blazor app, and then use Uno as a simple container to hose SkiaSharp things. Like an iframe or maybe set up some communication between the two. The main app can be Blazor, and the drawing part can be an iframe. This obviously depends on the use case... Sort of treat the Uno part as a separate "canvas"...

[mattleibow]

I made some interesting thing: an Uno iframe inside a Blazor app. The communication is a bit dodgy right now, but might not ne too bad depending on the use case.

https://github.com/mattleibow/SkiaSharpUnoBlazorApp

[Flippsor]

This looks really great! Already checked it out and got it working. Investigating the code, i recognized that the communication is done through the iFrame via JS Interop. This will be a little bit tricky because we are planning to create an editor where the same objects are used in blazor lists and also drawn onto canvas. So as i understand, using the same object reference in blazor & SkiaSharp will only be possible when AOT compilation arrives?

[mattleibow]

Sadly yes. Right now, it is 2 entirely different runtimes and websites, so there is no way to reuse anything.

But, I want to try out a different communication model. Using the messaging. This should allow me to use json objects. But, this will still go out via js.

[mattleibow]

Wait a sec... I wonder if there is a way to pin memory in js and use that in both sides... I do this, sort of, for skia... I create the pixels and then pass the pointer to js, might be able to read it in uno/blazor

edit

That will only ever work for blittable structs. Not useful for general classes.

[mattleibow]

I did an improvement with the communication. No longer using slow fragments, but direct JS.

[mattleibow]

I have been hacking away on this, and I think I may be able to do something for Blazor server-side: https://github.com/mattleibow/SkiaSharpBlazorComponents

<div>
    <SkiaSharpImage OnPaintImage="@PaintImage" />
</div>

@{
    void PaintImage(PaintImageEventArgs e)
    {
        var canvas = e.Surface.Canvas;

        canvas.Clear(SKColors.Blue);
    }
}

[Flippsor]

Great work!

Afaik the API for Blazor Server and Blazor Webassembly for Razor components is identical. So people could start with Blazor Server + Skia and then switch to Blazor Webassembly + Skia when the AOT is ready.

[mattleibow]

That is the plan!

But the client wasm one also is a bit different with the fact that I can have a canvas view...

I need to investigate if we should

• use an image for the server and a canvas with a real time render
• use an image everywhere, and have an additional option for the client...

I am very far from the web space, I will need some help 😅

[RChrisCoble]

I would think the best you can get here is drawing an image on the server and having that sync across using the Blazor SignalR connection.

For the client, as long as the draw calls are not going over the JS/Interop boundary to slow it down, whatever solution should work.

[matdennoigi]

@mattleibow Your code only works on Blazor Server model, not work on WASM, because your SkiaSharp version that you used for code can't work on WASM. Actually, your Skia rendering was excuted on server then passed to client through HTTP.

[GeorgeS2019]

@mattleibow based on the SkiaSharpBlazorComponents example, how would you improve the code to make SkiaSharp animation possible.

[mattleibow]

Animation? That is hard. Because only the server can render, the animation might have to be pre-rendered. Not sure if that is going to be nice with latency.

We really need the ability to link native libraries with Blazor. Like reeeeeeaaaaly need it. The only thing that is possible to do now is render a few png and then fake it. But that is not good at all.

[mattleibow]

The only thing that can work for client rendering is an embedded Uno app.

[GeorgeS2019]

@mattleibow have you tried checking the animation performance with embedded Uno app? Perhaps the combination of client side embedded Uno and server side image rendering may be a good use case, during the transition to AOT in the coming .NET6

[mattleibow]

The embedded Uno app should run at native performance of AOT WASM in an iframe.

I have not done any actual tests, but I was able to get 60fps with a flappy bird game, and that should not be affected by an iframe. And, that game was also just the interpreter.

[GeorgeS2019]

@mattleibow it takes around 1 min or more to compile the SkiaSharpUnoApp alone through WSL2.

• Do you have suggestions what I have done wrong or what I could improve that.

The project works. I still try to understand what "Hack" you used to make embedding UnoApp works within Blazor. I compare that with the https://github.com/unoplatform/Uno.SkiaSharp/tree/uno/samples. It seems there is lot of copying the compile output files from the SkiaSharpUnoApp to that of the SkiaSharpUnoBlazorApp.

I do not have sufficient experience with Uno.SkiaSharp. It is unclear how much I could bring the Unit Tests in Uno.SkiaSharp into the SkiaSharpUnoApp and SkiaSharpUnoBlazorApp project setup you have implemented.

I hope you will continue your investigation and perhaps we will see your work in coming community asp.net standout or coming Microsoft events.

I also hope the discussion here will speed up the implementation of AOT that you see as key to bringing SkiaSharp to Blazor.

[mattleibow]

Another related issue for getting better WASM support: https://github.com/dotnet/runtime/issues/44636

[GeorgeS2019]

@mattleibow thanks for the tip. I will follow up

[RChrisCoble]

Hello @mattleibow, have you thought any further on how SkiaSharp might run as a Blazor Server module?

Option 1: Load Skia and SkiaSharp in the browser. Abstraction layer sits on top of SkiaSharp API’s on server and marshals draw calls to client. Option 2: Load Skia only in the browser. SkiaSharp maintains same surface API and runs on the server, but marshals calls to the remote Skia running in the client browser.

[RChrisCoble]

Hello @mattleibow.

My suggestion is a stepwise approach while we wait for Blazor AOT to come into place:

• Start a POC for a SkiaSharp Blazor Server extension leveraging Uno’s SkiaSharp client rendering support.
  • This might be as simple as just drawing a rectangle from the server to the client.
• Assuming this works, expand out the support with additional SkiaSharp API calls.
• As .Net 6 Beta AOT support starts to roll out, update the POC to support SkiaSharp rendering in the client and remove the Uno implementation.
• Harden the implementation for a PR back into the SkiaSharp repo.

[mattleibow]

Just adding my emails here as a way to keep the world informed:

---

That is pretty much what I think we might be able to do. Think of SKPicture as a binary serialization of the final drawing commands that are going to be sent to the client. Just like the canvas extensions do now with commands, skia does with pictures.

For example, if I want to send a very simple 1024x1024 “image” to the client, it is just 165 bytes (uncompressed) – which I am thinking might be less that the equivalent with canvas extensions.

I have been putting together a little demo of what I mean. I am not to clued up on what to do to make something native like Daniel suggested, but I decided to make a managed “library” that runs in the client browser and just draws.

https://github.com/mattleibow/BlazorSkiaSharp

Unfortunately, I think there might be a bug in the deserialization process in WASM in skia. Everything works on the server and the picture is inflated from the commands. The client can also serialize just fine. There seems to be some issue with deserialization on the client.

I’ll look into this tomorrow, but this might be worth looking at. Instead of sending over drawing commands as a batch, skia will send them over pre-optimized for drawing. This will include any pre-processing for clipping and calculations. This also allows you to leverage the server for any additional logic.

That link attached is what I have in the meantime consists of 2 apps: the Blazor app and the .NET 5 console app (which is the client library). When the server completes drawing some frame, it sends the commands over the wire by calling a JS function. The client then does the rest by basically calling into the console app running on the client. This client then draws the actual frame as pixels and blits it onto a canvas.

1. Draw the frame as a batch: https://github.com/mattleibow/BlazorSkiaSharp/blob/63bc2350dcf9d3cfb4b336ae553b1c9d63715c72/WebApplication1/Pages/Index.razor#L26
2. Send the commands to the client by calling a JS function: https://github.com/mattleibow/BlazorSkiaSharp/blob/63bc2350dcf9d3cfb4b336ae553b1c9d63715c72/WebApplication1/Pages/Index.razor#L47
3. Trigger the batch processing on the client: https://github.com/mattleibow/BlazorSkiaSharp/blob/63bc2350dcf9d3cfb4b336ae553b1c9d63715c72/WebApplication1/wwwroot/js/example.js#L15
4. Draw the frame for real on the client: https://github.com/mattleibow/BlazorSkiaSharp/blob/63bc2350dcf9d3cfb4b336ae553b1c9d63715c72/ConsoleApp1/Program.cs#L38
5. Update the actual canvas by calling another JS function: https://github.com/mattleibow/BlazorSkiaSharp/blob/63bc2350dcf9d3cfb4b336ae553b1c9d63715c72/ConsoleApp1/Program.cs#L48
6. Blit the frame to the screen: https://github.com/mattleibow/BlazorSkiaSharp/blob/63bc2350dcf9d3cfb4b336ae553b1c9d63715c72/WebApplication1/wwwroot/js/example.js#L18

I’ll have a look and see why this is not working at the moment, but maybe this is something you can try? I am using .NET for the client, Google has CanvasKit that is a JS lib and you can totally make your own as there is no real work being done.

[mattleibow]

I finally managed to get everything working, you can check out my repo again: https://github.com/mattleibow/BlazorSkiaSharp

Seems I had explicitly disabled deserialization in the compiler flags 😐

With regards to your questions:

• I am using base64 now because I wanted to have a “human readable” message in characters that can appear on the screen, but you can totally send it any way – even as the byte array form.
• You can apply those drawing to any canvas, in fact in my example, the page just has one canvas and I keep rendering to that – no DOM changes.
• We still need AOT, but we now no longer need to wait for Blazor to get it. In my example, I am using the Uno bootstrapper to take the console app and compile it to WASM. I believe the .NET team has something as well that they use to build their own bits. I was not to familiar with it, so went for the Uno bootstrapper as I wanted to get this to you as soon as I could. You can chat to them and see if they have a nice process that you can use.
• With CanvasKit, Google does the AOT and then they have a wrapper in JS. What I am doing is basically making a similar thing, but using .NET so I don’t have to write the JS-native interop code. It is a little weird. However, the benefit of that is there is no dotnet.wasm to download. But you now have to write some C++ interop code: https://github.com/google/skia/blob/master/modules/canvaskit/canvaskit_bindings.cpp This has been done here, but it may not be something that you would be comfortable with.

In my example, the client bit that runs SkiaSharp is not a Blazor app, but just a normal console app. Think of it as a parallel to CanvasKit. Instead of writing binding code in C++ and then wrapping it in JS, I got the binding in C# and the wrapper in C#. Ittotally can be used as a standalone library and can be placed on a npm repository. My example effectively has a blazor app that depends on a drawing library, that just so happens to be a .NET WASM library.

[mattleibow]

This is my repo: https://github.com/mattleibow/BlazorSkiaSharp

So basically folks, I have effectively created a separate JS library to do all this work, and it coincidentally just uses .NET as the main driver. You could replace the .NET with JS bindings.

[RChrisCoble]

Thanks @mattleibow for all your help and expertise in getting this prototype together.

In your opinion, where do you think this final supported solution should 'live'? I'm pretty sure I won't be the only person that wants to draw with SkiaSharp in Blazor. Once we get AOT and can finally compile the client bindings without Uno or CanvasKit, would this be a solution that ends up in the SkiaSharp repo? A new repo? Or nowhere and we just figure it out individually?

I realize you feel it doesn't makes sense in the SkiaSharp repo as the core tenant of the repo is a definitive wrapper around Skia, but I would argue this SKPitcure solution will only work with matching SkiaSharp builds on client and server (when rendering on the server), meaning the Skia and SkiaSharp versions need to be aligned. That would be much easier if the Blazor Client or Server implementation was built directly in the SkiaSharp repo. Of course one could argue the oppose by looking at the Blazor Canvas repo which obviously don't contain the rendering logic for the browser canvas itself.

At any rate, from a consumer perspective (which is selfishly mine) pulling this entire end to end support from a single repo is the most attractive option. I'll note that link you shared with me regarding System.Graphics had specific implementations for Blazor client and server baked into the framework, and that's a 'wrapper' around the concrete implementation being injected. Not all that dissimilar to SkiaSharp wrapping Skia.

[mattleibow]

It all depends on what the solution finally ends up as. It may fit in with the more official https://github.com/BlazorExtensions, or we could add it to the https://github.com/mono/SkiaSharp.Extended repo (sort of a community toolkit for SkiaSharp).

Because this is mostly independent to skia, we don't really want to tie up a release of this library with the slower SkiaSharp. I agree that this might be better shared, so I would definitely be on board to adding it to one or the other repos.

[EydenJones]

Any updates with the recent .net6 previews? Can we be hopeful running skiasharp wasm in the browser (client side) this year? Sorry for the impatience :-)

[RChrisCoble]

@EydenJones I believe we're waiting on preview support of the following before this feature can continue:

https://github.com/dotnet/runtime/issues/44636

[GeorgeS2019]

@RChrisCoble Thanks, now we know where to make loud impatient noise!

[GeorgeS2019]

Are we moving closer to SkiaSharp as a Blazor Extension => Update

[mattleibow]

Are we moving closer to SkiaSharp as a Blazor Extension

Yes.

The PR is almost ready: https://github.com/mono/SkiaSharp/pull/1811

A sample repo that you can see the code and try it out right now: https://github.com/mattleibow/SkiaSharpBlazorWebAssembly

[SalimiHabib]

Cannot find SkiaSharp.Views.Blazor namespace "SkiaSharp.NativeAssets.WebAssembly" Version="2.88.0-preview.145" is installed Do I need any other nuggets ?

[GeorgeS2019]

@SalimiHabib please read this first if your OS is Windows

[SalimiHabib]

my vs version is Enterprise 2022 Preview (64-bit) Version 17.0.0 Preview 4.1 is there any release for VS 2022 Preview 5 ?

[GeorgeS2019]

@SalimiHabib the change has been committed. I will wait for VS 2022 Preview 5

[mattleibow]

You need the preview feed: https://aka.ms/skiasharp-eap/index.json

Check the PR for more info https://github.com/mono/SkiaSharp/pull/1811

[mattleibow]

I have merged #1811 and will release a public preview of the new client-side, wasm SkiaSharp.Views.Blazor package on NuGet as soon as the build is green.

[Stamo-Gochev]

@mattleibow Is there a timeline when https://www.nuget.org/packages/SkiaSharp.Views.Blazor is going out of pre-release state?

[Stamo-Gochev]

@mattleibow I see that the SkiaSharp.Views.Blazor package still works in Blazor WebAssembly apps only: https://github.com/mono/SkiaSharp/blob/ce7778c0c48b5ea668d91420023b295d5551006f/source/SkiaSharp.Views.Blazor/SkiaSharp.Views.Blazor/Internal/JSModuleInterop.cs#L12-L15

Is there a timeframe for when the functionality will be supported on Server-side Blazor (if it all)? This will allow the full functionality to be available regardless of the Blazor model as otherwise this forces users to stick to wasm apps only, which is not always possible.

If porting this to work on Server-side Blazor is not possible, can you provide more details what are the limitations for it to happen?