Support multiple screens

[bwikbs]

Background?

There are requirements for multiple screens. There's nothing specific, everything can change, but multiple screens don't change. 😄 So I checked how multi-screen support is going on in Flutter. And I found the following:

https://docs.google.com/document/d/11_4wntz_9IJTQOo_Qhp7QF4RfpIMTfVygtOTxQ4OGHY/edit
https://github.com/flutter/flutter/issues/30701
https://github.com/google/flutter-desktop-embedding/issues/98

IMO, Supporting multiple windows didn't seem to happen soon at this moment. So I decided to look for Plan B. That is EngineGroup feature.(https://flutter.dev/docs/development/add-to-app/multiple-flutters) Technically, It’s a little different but I thought It would be able to produce similar looking with multi screen app. So, I aimed the first step of this task to port sample app(https://github.com/flutter/samples/tree/master/add_to_app/multiple_flutters) at tizen.

---

EngineGroup

As you can see in the example, this is part of how to embed Flutter in a native app. The best way is to embed Flutter in tizen as one of the GUI widgets , but this is not easy In the current implementation. (Many of our rendering behavior are based on window) Here is my rough plan based on these.

1. Make windows externally and use them. This will allow us to create and place partial windows at single app. (https://github.com/flutter-tizen/engine/pull/98)
2. Create spawn API in embedder.
3. Create FlutterEngineGroup API at native API
4. Port sample app to tizen

(Of course, I am not insisting that this feature should be done this way. Any ideas are welcome. 😄 )

[bwikbs]

Currently, My goal is to get the following screen with a new FlutterEngineGroup API. Because I think this can satisfy the requirements what I understand.

[bwikbs]

@xuelian-bai @swift-kim Any comments?

[xuelian-bai]

I'm not sure about this, but I think, if we want to support multi-screens, we can just create multiple flutter engines, each engine has its own window, this may consume many resources, but it's easy to do this way.
I think multiple-flutters solution is actually from hybrid application on android(some pages are written in kotlin, some pages are written in dart). I will check more details tommorow.

[xuelian-bai]

for multiscreen requirement, is it for multiple monitor? One monitor display one window? Only one window can handle input events? If that's the scenario, we can use eom to present another window(image) on another display. https://code.sec.samsung.net/confluence/display/WSI/EOM

[swift-kim]

@xuelian-bai Yes, I heard the second monitor wouldn't accept touch input and only display visual contents (mostly ads) without user interaction for now. Probably the API you mentioned will be used (I don't know the details). However, the scenario may be changed in the future and I think @bwikbs is preparing for possible future requirements.

[bwikbs]

is it for multiple monitor?

yes

One monitor display one window? Only one window can handle input events?

Nothing has been confirmed yet. I wish I want to know too.. :smile:

eom is the solution that other teams have been telling me to use for this. So we will probably use it. But I'm not sure if we can guide this to flutter app developers.

There are many ways that how app developers use this, and I hope you will focus on that and share and discuss our ideas.

[xuelian-bai]

We may refer to this plugin https://github.com/VNAPNIC/presentation-displays.git

[xuelian-bai]

Here is my initial idea, didn't consider much details

[bwikbs]

Thanks for giving me a good idea! I have a question. Is this the same process? Could you explain a bit more running the engine? I have no idea where the dart is and where the native is.

[xuelian-bai]

it's just a rough process, here are more details, almost the same as the referred plugin: dart app --> define external widget, main widget build() --> init eom plugin --> get outputid from eom --> create new flutter engine --> set window to outputid --> set initialroute as defined external widget for flutter engine in dart. The referred plugin can use flutterview, but we can't, since we don't need to handle events for window on external display, I think it's ok to create two windows using eom, but we need to confirm this.

[xuelian-bai]

[bwikbs]

The referred plugin can use flutterview, but we can't, since we don't need to handle events for window on external display, I think it's ok to create two windows using eom, but we need to confirm this.

Unfortunately no one can do this. :smile:

---

I think your approach is pretty good.(Especially,app developer doesn't need a native app interface) but still need a new engine instance and window instance.(Am I right?) FlutterEngineGroup API has resource and performance advantages. so.. I think it is still necessary to apply this to Tizen.

IMO, Now is the time to prepare for the unclear requirement, If we can afford it, it would be meaningful to POC both and integrate them later.

[xuelian-bai]

I think your approach is pretty good.(Especially,app developer doesn't need a native app interface) but still need a new engine instance and window instance.(Am I right?)

Yes, it needs a new engine instance, and I'm not sure if this engine instance can access "route" in main widget. We need a new window because eom can only set window to external display.

FlutterEngineGroup API has resource and performance advantages. so.. I think it is still necessary to apply this to Tizen.

I checked android patch, lots of work, when will you start?

IMO, Now is the time to prepare for the unclear requirement, If we can afford it, it would be meaningful to POC both and integrate them later.

I will assign xiaowei to investigate and implement eom related plugin, is that ok?

Actually I think there is another easier approach for the requirement, but it's kind of work around

1. Start app as usual
2. Implement the UI part on external display as offscreen widget, and make it as image
3. Create efl window inside app, attach the image to efl window, set efl window to eom output. In this way, we only need one engine.

[bwikbs]

I checked android patch, lots of work, when will you start?

Yes, you're right! And I haven't figured out what to do yet. My current plan is to create an API that uses the shell's spawn API first, and then slowly add the rest of the work and what we need.

After the word came out, Let me share what I've checked so far... Currently, When our embedder uses two engines at the same time, the only problem is the vsync waiter. This is not a problem with our implementation, but rather a problem that the upstream is not supposed to use two at same time.

Therefore, no special modifications are required at this time, and we can test 2 instances by adding spawn api to the embedder part.

I will assign xiaowei to investigate and implement eom related plugin, is that ok?

Maybe @haesik will answer this. :smile:

Actually I think there is another easier approach for the requirement, but it's kind of work around

If we don't have time and the scenario is finalized, we can consider it...But we still have time, so let's leave it as the last resort. :smiley:

[xuelian-bai]

Currently, When our embedder uses two engines at the same time, the only problem is the vsync waiter. This is not a problem with our implementation, but rather a problem that the upstream is not supposed to use two at same time.

Why two vsync are not supposed to use at the same time? I didn't check code of this part, but theoretically, If there are two engines, then there are two vsync, they are in different thread, why can't co-exist?

[haesik]

@xuelian-bai You can make the EOM related plugin of course, but we can change the direction or plan to support multiple screens

[bwikbs]

we found that two engines are needed for eom plugin yes, if only one engine with two windows, performance won't be good

@xuelian-bai Can you tell me something about this?

[bwikbs]

I think I've shared what I'm trying to do. @swift-kim @xuelian-bai How do I handle below commit? https://github.com/flutter-tizen/flutter-tizen/pull/126, https://github.com/flutter-tizen/engine/pull/98

IMO, there are two options.

1. Working on a separate dev branch.
2. Put it in main as an experimental feature.

Plz, give me your opinion.

[swift-kim]

@bwikbs If the design and API haven't been fixed yet, the first option looks better for me. If not many API changes will be made in the future, you might also consider the second option.

[bwikbs]

Thanks for your comment. I think the first one is better too.

[bwikbs]

update I tested the shell's spawn API. It seems that simply using the API does not help with memory management. (Except for the shell API, everything else is the same.) 1) Use shell's Spawn API

PID	S(CODE)	S(DATA)	P(CODE)	P(DATA)	PEAK	PSS	3D GEM(PSS)	GEM(RSS)	SWAP	oom_score_adj	COMMAND
8957	73560	4	10032	111512	195108	152094	8552	1224	2448	0	200	/opt/usr/globalapps/com.example.multiple_flutters_tizen/bin/runner

2) Use shell's Create API

PID	S(CODE)	S(DATA)	P(CODE)	P(DATA)	PEAK	PSS	3D GEM(PSS)	GEM(RSS)	SWAP	oom_score_adj	COMMAND
3731	73676	4	9976	111468	195124	152050	8624	1632	3264	0	200	/opt/usr/globalapps/com.example.multiple_flutters_tizen/bin/runner

[xuelian-bai]

@bwikbs did you run in release mode?

[bwikbs]

No, only debug mode. Is there any difference?

[xuelian-bai]

not sure if it's related, but you may see in function DartIsolate::CreateRootIsolate(), it will also check DartVM::IsRunningPrecompiledCode() if spawning_isolate. anyway, worth a try..

[bwikbs]

not sure if it's related, but you may see in function DartIsolate::CreateRootIsolate(), it will also check DartVM::IsRunningPrecompiledCode() if spawning_isolate. anyway, worth a try.

Here is result of release As you said, the results is little different. Thanks for your tip! :+1: There seems to be some gain at PSS, and the launch seems to be a little bit quicker.(by human sensor! :smile: ) I feel it's worth moving on.

---

1. Use shell's Spawn API

PID	S(CODE)	S(DATA)	P(CODE)	P(DATA)	PEAK	PSS	3D GEM(PSS)	GEM(RSS)	SWAP	oom_score_adj	COMMAND
9105	13200	4	8400	10736	32340	20151	8520	1632	3264	0	200	/opt/usr/globalapps/com.example.multiple_flutters_tizen/bin/runner
9973	16544	4	8408	12020	36976	22448	8552	1224	2448	0	200	/opt/usr/globalapps/com.example.multiple_flutters_tizen/bin/runner
10636	16544	4	8408	11976	36932	22404	8552	1224	2448	0	200	/opt/usr/globalapps/com.example.multiple_flutters_tizen/bin/runner

---

2. Use shell's Create API

PID	S(CODE)	S(DATA)	P(CODE)	P(DATA)	PEAK	PSS	3D GEM(PSS)	GEM(RSS)	SWAP	oom_score_adj	COMMAND
9104	22140	4	5600	13632	41376	24047	8624	2040	4080	0	200	/opt/usr/globalapps/com.example.multiple_flutters_tizen/bin/runner
10853	22132	4	5600	13612	41348	24027	8624	2040	4080	0	200	/opt/usr/globalapps/com.example.multiple_flutters_tizen/bin/runner
11495	20768	4	6964	13680	41416	24973	8688	2040	4080	0	200	/opt/usr/globalapps/com.example.multiple_flutters_tizen/bin/runner

[xuelian-bai]

hmm. launch is definitely quicker, as I tested, release mode will decrease about 500ms when preparing dart isolate, since it's using AOT.

[xiaowei-guan]

https://github.com/xiaowei-guan/engine/tree/dart_entrypoint I have finished sample code for multiple screens based on EOM in flutter engine. if we want to full support multiple screens, we need :

1. support dart entrypoint.
2. support engine group.
3. fix vsync issue

[bwikbs]

Oh, thank you! If things go well, it might be possible to run an example by referring to this.

[JSUYA]

https://github.com/flutter-tizen/embedder/pull/3 A patch to support EOM has been merged. Note please.