Preliminary progress on Stride-Maui integration ... What is a good workflow to follow from here?

jonmdev commented 7 months ago

INTENTION

I an trying to integrate Stride into Maui, allowing Stride to run in the background of a Maui application and render to a custom native Maui View (ie. using UIView in iOS, TextureView in Android, perhaps SwapChainPanel or a WriteableBitmap perhaps applied to Image view in Windows).

I think this should be reasonably simple due to Stride's code only approach in Community Toolkit and them both being C#/.NET.

ANDROID/IOS ISSUES

As noted here:

There are Android and iOS specific bugs preventing the Community Toolkit from being successfully added as a NuGet in these build targets.

Next Steps:

Next steps for these operating systems will be trying to fix the bugs in the root Stride libraries causing them. Then I can continue from there.

WINDOWS PROGRESS

Windows seems to behave better with no bugs on adding the Community Toolkit NuGet.

Using a default Maui project and the suggested Code-Only approach for Community Toolkit, I can thus already get both running together, with no errors. However, they start in different Windows by default as would probably be expected:

This was managed by the simple ~70 lines of code here.

Basically, I am just running the Stride Game function inside a Task.Run operation, started from inside the Maui code, so it runs on a different thread (Maui gets the primary UI thread as it is the main overall application, Stride runs on a separate thread).

Next Steps:

Next steps for Windows would be trying to stop Game from creating a new Window as I don't want a new Window. I want it running like normal but without outputting the visual render anywhere.

I want to then just get the visual output of Game (bitmap, whatever) at a certain framerate presumably to update my native UI elements on the other thread with it.

I think this is the right approach - Maui runs on primary thread, Game runs on secondary thread, and we just basically need to get a render/Bitmap or some form of visual stream out of Game at a certain frame rate back into Maui.

Make sense? I have not done this before but it seems logical.

WORKFLOW?

I have no experience with Libraries or NuGets besides installing packages others have made. So I am not sure how I can keep working on my test project while also directly manipulating the Stride source code at the same time.

I would like to add all the necessary Stride code to my existing "MauiStride" test project: https://github.com/jonmdev/MauiStride

But not as NuGet packages (which I can't edit).

I think I maybe need to clone all the Stride packages to my computer. Then in Visual Studio from my "MauiStride" project go File > Add Existing Project > select csproj and do that for all the Stride packages? That way I can see them all in the same place.

Then I need to right click on Dependencies for my "MauiStride" test project and Add Project Reference to add each as well as a Dependency.

Is this correct?

Alternatively, do I just copy and paste all the Stride code folders directly into my project folder? Would that work as well?

I am not sure how I am supposed to go about this. What is the typical workflow if you want full code access to Stride in this manner (so I can test and change and fix things in Stride) while working directly on my test project?

Thanks for any thoughts or suggestions. Hopefully this will not be too bad. If I can get the workflow going hopefully I can get Windows running first as proof of concept. Then fix the others after.

Doprez commented 7 months ago

Mobile: I wish I had more knowledge of the mobile builds of Stride to help you. The one thing I will say is it may be worth directly referencing the Stride nugets instead of the toolkit, in case its causing an issue for mobile. also if you havent seen this discussion it might give some more context for you https://github.com/stride3d/stride/discussions/1657

Windows: Nothing to really add here just wow, Im happy you got this working and huge props for sharing!

Workflow: This is a great video if you want to run Gamestudio Source and debug your game https://www.youtube.com/watch?v=bXSC9o-EaR8&t=110s but I know you want to use a code first approach. You would do essentially the same thing but instead of running gamestudio, you would just make the direct references to the runtime libraries from the below folder in source.

These should be located in the folder ..\stride\sources\engine

The only thing thats a bit annoying here is you would have to do the same thing for the Toolkit if you continue to use it, otherwise you can just recreate the extensions you need by copying the required code like SetupBase3DScene or game.Create3DPrimitive(PrimitiveModelType.Capsule);

jonmdev commented 6 months ago

No time for this. Was able to integrate another project much quicker. Best wishes for Stride. Seems great but needs help.

tebjan commented 6 months ago

Was able to integrate another project much quicker.

Glad it worked out for you. But which one was it? maybe we can have a look at their MAUI integration.

jonmdev commented 6 months ago

ThreeJS with react js fit into a maui hybridwebview. Very easy and kind of cheats the entire system for simple needs at least. Very basic though. Stride would be great by comparison.

The problem with integrating a true game engine like Stride is you really have to understand the whole rendering and window creation process. I was thinking I would have to study Stride for months to understand how all that works so I can take control of it, since I know nothing of opengl or any of that.

The closest thing that I know is urhosharp where they took urho3d and mapped it to native xamarin views.

That is what would need to happen with stride in maui. You need to stop the new window creation of Stride (since you don't need a second window), and have some rendering data stream out of Stride to apply to the native element displaying it.

Again I think if someone understands the system this will not be hard for them to put in some toggles to turn on/off such a configuration in stride but otherwise quite challenging.

You can see UrhoSharp's UrhoSurface here for how they mapped game engine output to the native views using a class UrhoSurface:

https://github.com/search?q=repo%3Axamarin%2Furho%20%20urhosurface&type=code

That was a system that integrated beautifully but got abandoned. Urho was not as good a game engine as Stride in terms of quality features or design.

If you can do something similar, and run the Stride game on a separate thread like I was doing above with any UI updates to the surface perhaps invoked back to the main thread this would allow the main thread to handle the native ui and a second thread for the game system.

It would be quite optimal I think. Or if the cross thread communication becomes too bothersome maybe they would have to share a thread, though I think that might be worse to try to coordinate and keep both running smoothly especially with the Srride game update loop trying to maintain steady frame rate. I honestly think having them on separate threads is the right way to go. But I am not sure.

Anyway, that was the direction I would have gone with this.

1) Get blank Maui app in Android and iOS at least building with Stride nuget (ie. Fix the two basic bugs i reported).

2) Add toggle to disable window creation when Stride engine starts so it can start from inside Maui as shown but without any visible outcome.

3) Figure out where in the code some sort of rendered data stream exists and try to map it to the native surfaces in some fashion like UrhoSurface did.

I definitely think it's possible. For someone who knows the Stride rendering and App/Window system it might even be easy. But for me it would just be too much as I don't know how any of that works.

I hope someone will do it.

It would be a solid way to add complex UI's, native systems, and get cross platform support to all major OS's. Eg. For mobile games on Android and iOS as well as Windows apps.

Stride is clearly a well designed system but it needs a way to access the cross platform and mobile market. To me this is the most logical way to do it as I believe someone who really knows the system could probably do it reasonably easily and it keeps everything native with deep access to the various OS's.

jhimes144 commented 6 months ago

Hello,

I'm currently working on a game idea where I want to use something like Avalonia or WPF for my game's UI.

I was able to get WPF to run on top of the game and vise-versa by doing some tricks using win32 api calls to take the WPF window and make it a child of the game window, with transparency and no border. Then I wrote code to sync the wpf window's position with the game windows position. It works pretty flawlessly. I plan on trying the same with Avalonia. The Stride Game Studio does the same thing but in reverse. If you want to see how Stride does it, take a look at Stride.Core.Presentation.Controls.GameEngine class. The code isn't too hard.

The limitation I'm running into is that this forces the game to be in windowed mode instead of fullscreen exclusive, and I'm experiencing screen tearing which I just opened a bug ticket for.

I'm a little fuzzy on MacOS api calls, but I think the same might be possible there too. I can't imagine this technique would work on mobile though.

stride3d / stride