Rename embedder to something more suitable (running multiple instances of the embedder)

[mv0]

Presumably this change is a bit controversial, I'm merely suggesting a name change, not really sure the one that I picked is really good or not.

Naming this homescreen doesn't really make sense in platforms where there are multiple instances of the embedder running so I suggested (re)naming it flutter-ed (flutter-embedder).

Obviously the meta-flutter recipes also need changing, but I said to get the ball rolling and see how people feel about this.

[jwinarske]

@mv0 I'm fine with renaming, only flutter-ed is a really poor choice. Just search ed...

I vote using flutter. It's not present in Linux, and it would be completely valid to claim it.

[jwinarske]

@mv0 To be clear, running multiple instances of the homescreen app is not the proper approach, it's a workaround use case, and should be avoided in production scenarios. A workaround use case should never dictate the primary use case when spawning multiple engines.

[mv0]

@mv0 To be clear, running multiple instances of the homescreen app is not the proper approach, it's a workaround use case, and should be avoided in production scenarios.

I'm not aware of any other potential approaches when trying to run applications independently of each other. And it is not specific to AGL either, browsers are using the same approach to sandbox (running tabs in a separate process) their applications.

In terms of UI and UX, Wayland does actually favor having a single application, capable of managing multiple surfaces, quite differently from X11 (and this is what the shell client is actually all about), where you have multiple applications to "compose" your entire UI. Further more, the transfer of surfaces between applications happens only with some private extension, as Wayland by nature doesn't allow clients to know about each others surfaces.

The embedder should be able to run independently of each other (so like first paragraph), but also as the shell (where it capable of managing multiple surfaces using those flutter views you've added). In a platform managed entirely by the flutter embedder, you will have an instance of the flutter embedder running as shell client (as an independent application), along side other/multiple flutter emebedder instances, which are actually your regular applications. You'll have at least one instance of the flutter embedder as the shell client. But you can mix and match html5/web platform or the qt one, the requirement being that you can only have a single shell client.

Hope this clarifies where actually we're impersonating a shell client and when we're running as processes, entirely separately of each other.

I vote using flutter. It's not present in Linux, and it would be completely valid to claim it.

Isn't flutter the engine itself? I explicitly avoided using that name, and it is actually present in linux. You use to build applications. I think Scott Murray was also said something similar just that the name is already taken by the engine itself. I'm quite OK with flutter if that's what we decide.

[jwinarske]

I'm not aware of any other potential approaches when trying to run applications independently of each other. And it is not specific to AGL either, browsers are using the same approach to sandbox (running tabs in a separate process) their applications.

The flutter engine design comes from the mobile world where the current app suspends when leaving the foreground. So anything around processes and behavior are up to the application that implements the flutter engine. For a best case system load scenario a single process should be responsible for spawning engine instances, similar to the Chromium zygote process. Today starting multiple instances of homescreen is the naive scenario that does not optimize system resources, unlike Chromium. This could be a milestone goal in the future, as it currently works, and the design is not formalized.

Chromium Zygote Description

As mentioned on Linux there is a special process called Zygote process. It is created at the startup and later the requisite Renderer processes are forked from it. It has an inner loop as any other Chromium process. The performed operations are called from the loop's TaskAnnotator or TaskRunner object.

In terms of UI and UX, Wayland does actually favor having a single application, capable of managing multiple surfaces, quite differently from X11 (and this is what the shell client is actually all about), where you have multiple applications to "compose" your entire UI. Further more, the transfer of surfaces between applications happens only with some private extension, as Wayland by nature doesn't allow clients to know about each others surfaces.

Around composition of screen via Wayland, I see a pattern of using Weston combined with SoC hardware alpha blending to composite an image. Multiple fullscreen apps (multiple process) - UI layer 0, Video Layer 1, etc. The key benefits of this approach are that the hardware alpha blending involves no CPU, and that it allows a given layer an optimal format pipeline (no conversion overhead), which presents best case CPU loading. In the case of a SoC that doesn't support hardware alpha blending the single process scenario might be better. Using Flutter render to texture also presents a compromise on the system. There are a number of tradeoffs however we slice it, and taking the best case technical path also presents a different compromise (possibly cost of ownership). So knowing all the elements in the design allows us to make a better decision. Rarely does one size fit all.

Adding the concept of a Zygote process would resolve the system loading problem of running multiple instances of homescreen today. As having a single process to manage engine instances is the correct design.

The embedder should be able to run independently of each other (so like first paragraph), but also as the shell (where it capable of managing multiple surfaces using those flutter views you've added). In a platform managed entirely by the flutter embedder, you will have an instance of the flutter embedder running as shell client (as an independent application), along side other/multiple flutter emebedder instances, which are actually your regular applications. You'll have at least one instance of the flutter embedder as the shell client. But you can mix and match html5/web platform or the qt one, the requirement being that you can only have a single shell client.

I would make the zygote process the AGL shell client. Two additional features I am adding are multiple display, and a FlutterView control API. The control API would live in the zygote process.

Isn't flutter the engine itself? I explicitly avoided using that name, and it is actually present in linux. You use to build applications. I think Scott Murray was also said something similar just that the name is already taken by the engine itself. I'm quite OK with flutter if that's what we decide.

It's actually the SDK tool (script) name used for building/running/debugging a bundle.

At some point I would like ivi-homescreen to be a generic Linux app capable of replacing the GTK embedder. Initially homescreen was selected, as that's what the app is providing. flutter-ivi doesn't apply to desktop, but might be fine for now.

[mv0]

The flutter engine design comes from the mobile world where the current app suspends when leaving the foreground. So anything around processes and behavior are up to the application that implements the flutter engine

I'm not that familiar with it (i.e., mobile applications), but I don't think suspending here means freezing the tasks, although I suppose that might be case, in some situations, some sort of scheduler that wakes up the task to handle things it might need.

Anyway, if you happen to remember the link I've posted from freedesktop related to how to correctly design your game loop, it would come of no surprise what happens if you tie your logical application loop to your drawing loop, so assuming you still need to handle or any other events (networking, other events from other buses), you most likely do not have the application suspended. What is more likely to be suspended is the drawing. And obviously, it makes no sense whatsoever to draw anything on the screen if the application is, like you said "suspended", or put in some kind of a background. What I'm trying to emphasize is that separating these two loops would make the biggest impact in a resource constrained environment.

With that in mind, and assuming that your application in the foreground isn't doing any alpha blending with what's under, (for instance a maximized surface, that's basically transparent), the compositor will discard from the beginning that background surface. It can attempt to update itself a few thousand times a second, it will just eat CPU.

To me it seems that all the these are actually under the influence of the client itself, or better said how those clients are written, rather than the design we're choosing to have here, where we deliberately want to have isolation between clients.

The fact that the flutter platform is heavy, on its own, in terms of resources, is orthogonal to our multiple process approach, and your proposal of using something similar to that zygote process would actually be very useful, so I'm all for that.

There are a number of tradeoffs however we slice it, and taking the best case technical path also presents a different compromise (possibly cost of ownership). So knowing all the elements in the design allows us to make a better decision. Rarely does one size fit all.

Yeah, totally. For sure. We kind of try to be as generic as possible but some times that's really not viable. Customers can choose to create their own shell plug-ins or, add additional private extensions to favour some code paths. Which might work well for some SoC, but detrimental for others.

It's actually the SDK tool (script) name used for building/running/debugging a bundle.

At some point I would like ivi-homescreen to be a generic Linux app capable of replacing the GTK embedder. Initially homescreen was selected, as that's what the app is providing. flutter-ivi doesn't apply to desktop, but might be fine for now.

Yep, I agree. Hope we can settle for a name on our next call.

[bernardcraddock]

thank you both for the deep conversation, don't fully follow but will keep re-reading until i do

[jwinarske]

@mv0

For optimal scenario for multiple process FlutterViews I'm considering the following scheme:

1. Zygote process (currently App class) -owns Wayland Dispatch -handles spawning of FlutterView (when configured spawn as process; see Flutter View section) -API handler wrapping lifecycle of FlutterViews (init/destruct) -Any direct engine/view callbacks would need to switch to IPC. -use an application loop here, libuv or glib. GStreamer has glib dep already, which would also solve (Dart D-Bus/application id scenario).

2. Flutter View -dedicated thread or process spawned based on config view flag. Default would be dedicated thread (minimalist) for engine message and texture render loop. -IPC with Zygote (process config only)

Using two phases for implementation. Process/IPC being the last phase.

Currently the same loop is used for Wayland dispatch, engine message queues, and frame render. This doesn't scale well when running multiple custom textures. I will be segmenting between Housekeeping vs Render. Wayland dispatch, engine message queues equals housekeeping. Any texture render calls would have independent loops per Flutter View. Note that when using a custom texture widget, the frame drawing is gated if the texture widget is not in the widget hierarchy (visible). The sleep in this render loop is dynamic based on timing measurements, targeting 60 FPS.

The segmentation between housekeeping vs render will also benefit Navigation implementations. You only want to call the Navigation render loop when the texture widget is active. The Navigation housekeeping needs to run regardless.

In the case of multiple fullscreen windows, a render loop suppression scheme could be adopted using enter/leave calls. On leave simply gate the render loop.

The flutter engine can be stopped, and started. I haven't played around with this, but it does incur visual delay. Each Flutter Engine has its own internal Render loop.

For Flutter engine the CPU consumption using the Vulkan backend is 50% less than when using EGL back end. Definitely the preferred scenario for commercial scenarios, as who doesn't want 50% more from their CPU? Not all SoCs have this luxury, which is why the EGL backend will not be deprecated.

Why exactly do we need multiple process support for homescreen?

In the case of opening a URI via Dart (url launcher plugin), it forks: https://github.com/toyota-connected/ivi-homescreen/blob/main/shell/static_plugins/url_launcher/url_launcher.cc#L46

[mv0]

Why exactly do we need multiple process support for homescreen?

Because that's how the entire AGL platforms works. The idea is that OEMs/users create their own applications and deploy that on the target. We do not pile up everything into a giant huge monolithic app. That's being discussed for a while now. For that, we support multiple platforms where these can be done: Qt, Chromium/HTML5, and Flutter.

The optimization of the flutter platform/embedder is IMO a good thing, no one is against that, but that should be a something being done in parallel, or rather after we have a way to make the flutter embedder capable of being shell client and capable of switching between applications. While the recent changes makes the flutter embedder capable of handling multiple surfaces, it can't really be used as it is now. We basically use homescreen from the Qt platform and run the embedder as a regular application.

So IDK, I think probably spending time into making this work with similar to the chromium browser might be a huge, potential detour, and for the time being in the project we're aiming for functionality, rather for than speed/lower resource usage.

[jwinarske]

@mv0 I understand you're not blocked. I'm laying out the design for what needs to happen for a proper solution.

[jwinarske]

@mv0

Around multi-process I think the issue is you just need a way to launch a new instance using the same app. How it's implemented (zygote model or not) is irregardless. It could spawn a thread, or it could fork a process. The lighter weight scenario for using threads would be preferred.

You have the ability to run multiple instances of ivi-homescreen now, so until the performance becomes a problem the zygote design will remain on the shelf.

[jwinarske]

@mv0 As of today main is moving to a quarterly release cadence. As such I have created an AGL branch. In it I will be renaming the app from homescreen to flutter-auto. Meta-flutter and downstream AGL patches will be adjusted for this.

[mv0]

Understood, alright, thanks!