Question: Custom backend (with skia)

[pum-purum-pum-pum]

I wrote(WIP) skia backend for druid(and hence piet) and I found that it seems like there is no other option for writing custom backends rather than to fork druid. Of course, I can maintain a fork, cause I'm not changing any stuff inside druid except Cargo.toml files, but it would be nice to have some mechanism for creating custom backends as a separate crate which just depends on druid and implements druid-shell backend via some API.

Currently, druid-shell chooses the backend via features, but in theory, it also could be done via traits and a "custom" feature with which you can provide your custom backend to it. Not sure about the relation with piet here, seems like it could be done separately. Or maybe it should be on another level of abstraction and one should implement some traits that druid-shell provides.

Am I missing something? Do you have any other solution to do this? Will you consider merging something that solves this? I can work on this, but I need a hint on how to better do this.

Thanks for the awesome library btw.

[cmyr]

Oops, saw your other issue first. In any case:

This is not something I had previously considered. Having truly custom backends feels a bit difficult, and would require a fairly substantial bit of work; I'm not necessarily opposed, but what would make more sense to me would be getting your backend merged into piet, and then allowing it to be selected via a feature flag.

This does of course have its own problems, particularly around maintenance; it would involve adding a new backend to piet-common, and that means we would need to ensure it continues to compile.

The other issue is that the piet backend is fairly tightly integrated into druid-shell in a way that does not abstract easily. That is: the mechanism for creating a RenderContext varies between platforms and backends, and I think it would be difficult to have an abstraction for that; at the very least i think you will need to have some conditional compilation at the point where you are setting up your backend.

In any case I'm happy to answer questions and talk through possible solutions, and I'm curious to hear more about your motivation. :)

[pum-purum-pum-pum]

Skia is something that is required for the project I'm working on but I'm not sure about the general case.

About my motivation: I'm developing GUI for embedded devices and also I have requirements to use Skia. I also want to have my own druid-shell platform (with DRM/KMS). Hence I only need part of the functionality of druid -- mb from my perspective perfect situation is not using piet and druid-shell at all because it's a place where I need to have more fine-grained control myself. Druid use case for me is having generic GUI widgets (layout stuff for example -- like scrolling, flexbox etc.) and handy GUI architecture.

I've implemented my own mock platform inside druid-shell which supports Skia and I wonder if that could be something that I can do from outside of the druid without forking it -- by providing some trait(s) implementation for example. It looks like if I add trait(s) which all platforms will implement it'll be possible to create some API that uses this trait instead of relying only on the features when selecting platform. (I'm guessing here). It will solve the issue of forking druid for me. I'll implement my custom backend in my crate and will have control of it in my repository, rather then specifying some fork from actual GUI crate. The only problem is that piet-common is still specified in druid-shell Cargo.toml and even exposes itself with put use. Looks like fixing that for me is only possible by implementing skia backend for piet.

Having a fork wich solves my problems is something that I'm trying to avoid here, but if so -- how much effort would it be to keep it upstream given that I'm only changing few lines of code and only add separate code? How often internal API change?

Here is my current code for druid-shell https://github.com/pum-purum-pum-pum/druid/tree/skia_druid/druid-shell/src/platform/skia (it's WIP and uses winit. I'll need to write DRM/KMS backend in the future)

Sorry for long message

[cmyr]

I think embedded devices are a really interesting use-case for druid, and I would be happy to try and support that. I would definitely be open to adding a new druid-shell backend that targets some set of embedded devices, and I'm also open to adding piet-skia.

I'm less sure about whether or not we can express the current set of relationships well as traits. It's possible, and you'r welcome to experiment, but there's a lot of tricky bits. One problem is that we would have to make all of the druid code generic over the various types of the particular piet backend, and there are a lot of those types.

The alternative that we have discussed is using dyn Trait for the various piet types. The main concern with this is that if you have two piet backends in use at the same time, you have the possibility of type mismatches; for instance if you're in a druid app that uses piet-skia but you also want to generate an svg with piet-svg, you end up with multiple incompatible dyn RenderContext types floating around.

I don't think this is a huge issue, but it's something to be aware of. In any case, if we want to go with traits, I think we will want a solution like this.

[pum-purum-pum-pum]

Thanks for providing useful context. I'll try to experiment with it and will reply here later.

[pum-purum-pum-pum]

I see now. I tried doing it and I have for example pub struct WindowHandle<T: WindowHandlePlatform>(T) which spoils ContextState in druid and hence every context we have (by requiring template parameter in type). Using dyn Trait can solve this indeed. That is what I'll try to do next unless there are some other options. The only problem I see is that it changes the druid's behavior -- it will dynamically dispatch druid-shell functions, but I doubt that it could cause any performance issues. (and also one you've mentioned with multiple implementors might be a problem, but I haven't touch piet yet and in case for other dyn Traits it looks ok)

[pum-purum-pum-pum]

Tried dyn trait and there are problems with this approach too. For example CustomCursor is not a trait and hence cannot be expressed without some template parameter. Unless we'll invent some trait for all Custom cursors.

[pum-purum-pum-pum]

However, if it's fine having Pain/Layout/Update.../Ctx\< T > depending on some generic parameter T: WindowHandlePlatrom I can continue my investigation. What do you think?

[raymanfx]

I'm also interested in something like this. To give a bit of context: I work in the automotive industry and would like to write some services in Rust (instead of C++). For the backend part, this is already starting to pay off. Now I'd like to rewrite some frontend parts (mostly camera data visualization and vehicle data plotting) in Rust as well. So far, I've found iced and druid are the only two frameworks I like. To be honest, I prefer the research based approach of druid.

However, software rendering is a no go - we often have data that has to be refreshed at 30 or even 60 Hz (multiple realtime camera image streams). Having a Skia piet backend should make druid run with GPU acceleration. Now my specific use-case may require a bit more work: I have multiple Image widgets that refresh at 30 or 60 Hz. Hardware accelerated drawing for those widgets alone could probably save most the CPU cycles I need to waste right now. But AFAIK surface integration (e.g. through wgpu) with druid is not possible right now. Otherwise I could probably come up with my own Image class which internally manages a Vulkan/OpenGL texture surface.

If dynamically adding a skia backend to druid by means of traits etc. is not possible, would you be willing to add it under a feature flag so it can be chosen at compile time, @pum-purum-pum-pum?

[cmyr]

@pum-purum-pum-pum I guess I missed a notification on this. I'm not sure how much help I can offer; you're dealing with problems we haven't really encountered yet, and I'm reluctant to introduce major architectural changes right now. At some point in the medium-term we're going to be doing a bunch of architectural work, exploring splitting druid itself into two parts; and perhaps something we can consider, during this work, is a way to more clearly isolate the backend.

@raymanfx video is definitely something we'd love to have, but doing it right is definitely a research project, and it isn't blocking any of our current work. I would be happy to see a skia backend in piet, and that should not be a crazy amount of work.

[raymanfx]

@cmyr Sorry for hijacking this issue once more - do you have some things on your mind regarding video, perhaps a roadmap or feature backlog? I am currently building a cross-platform camera app using Druid that I am almost ready to show off as a first MVP. Still spending much time on the backend crate, https://github.com/raymanfx/eye-rs. Right now I'm just using an async image widget and a delegate that sends captured frames to it.

Maybe we should open a tracking issue for video stuff? I'm willing to help out and tackle some of the pain points I encountered when working with Druid (not too many, I was pleasantly surprised!).

[cmyr]

@raymanfx the problem with video is that it is highly platform dependent. I don't have very much experience with this, but I know that generally you'll want to use some platform API to create a 'video view' and then you provide it with some media and control playback via some proxy object or similar. This is not a great fit for how druid works currently, where we just have a single view on the platform that we control; figuring how (for instance) to get our layout system to work with additional views would be a project.

On a long enough time-scale, we would like to move to a world where we are using a single gpu-backed graphics implementation for all of our drawing across all platforms; in that world we might prefer to take a more ground-up approach and come up with our video playback implementation, but I get serious "here be dragons" vibes about that whole idea; it's certainly not something I've ever gotten near in the past.

So: I would be happy to have a tracking issue for video, but realistically it's not going to be an immediate concern for quite a while. 😢