Requirements for fully custom Game runtime management

[Doprez]

PR Details

The amount of internal features for Stride makes it very difficult and cumbersome to make any custom root/base features for the engine. This PR opens up a lot of those features to be accessible to any dev just referencing the released Nugets.

Now devs should be able to:

• create completely custom windows using any UI framework (Examples will be needed but probably in an external repo for a POC)
• create custom game lifetime management (hopefully for both headless and new window run Stride game)
• generally more availability to make Stride customizable without scaring users by requiring them to touch the source
• ~I need to look into it more but with the changes here to the GameBase, can we use this for separating game and window logic?~ (Nope, need to look into changing the current Gamesystems due to rendering requirements currently built into the main loop. out of scope for now, at least for this PR.)

Related Issue

https://github.com/stride3d/stride/issues/870 https://github.com/stride3d/stride/pull/1315 (Takes the initial work done by @TheKeyblader and adds a few more) https://github.com/stride3d/stride/pull/1474 https://github.com/stride3d/stride/issues/1629

Types of changes

• [ ] Docs change / refactoring / dependency upgrade
• [ ] Bug fix (non-breaking change which fixes an issue)
• [ ] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist

• [ ] My change requires a change to the documentation.
• [ ] I have added tests to cover my changes.
• [ ] All new and existing tests passed.
• [ ] I have built and run the editor to try this change out.

[Eideren]

I think the internal windowing and some of the internal game logic needs to be refactored, this PR might introduce API that would become deprecated once this is on the way. But, at the same time, refactoring those areas properly might also require deprecating large swathes of the current public api as well, so not 100% set on that one, I'll let someone else chime in on this.

[Doprez]

Maybe when I have more time I can take the time to create a mind map of all of the connections to and from the Windowing API. I know last time I dealt with this was due to the Window Exiting bug and it was a pain to walk through all of the seemingly random calls.

[Kryptos-FR]

We need samples that showcase the use of those APIs before merging this PR. Otherwise it's hard to know what justify making one API visible and not one another. It could also be that more refactoring becomes required as to hide some implementation details that shouldn't be public because we can't guarantee stability between versions.

[Doprez]

But that's the hard part lol!

In all seriousness, I can finally take a proper look at this with my main PC. What I can do is create a project template recreating what was done for the SDL implementation but as an external Stride project to start.

[Doprez]

OK, I have recreated a POC for the SDL example in a separate project and the good news is it works fine. I did miss one more area with the InputManager since it uses the AppContextType and didnt have a case for there being no InputSource being added.

I added a simple case for Custom where it would just return null and added the Source manually in a custom Game implementation in the Initialize method:

    protected override void Initialize()
    {
        base.Initialize();
        var sdlContext = (GameContextSDL)Context;
        Input.Sources.Add(new InputSourceSDL(sdlContext.Control));
    }

Also due to how GameContext and Window are tied together, I had to do something a bit funky but functional in the constructor:

    public GameWindowSDL(string title, int width = 800, int height = 600)
    {
        window = new(title);
        GameContext = new GameContextSDL(window);

        // Setup the initial size of the window
        if (width == 0)
        {
            width = window.ClientSize.Width;
        }

        if (height == 0)
        {
            height = window.ClientSize.Height;
        }

        windowHandle = new WindowHandle(AppContextType.Desktop, window, window.Handle);

        window.ClientSize = new Size2(width, height);

        window.MouseEnterActions += WindowOnMouseEnterActions;
        window.MouseLeaveActions += WindowOnMouseLeaveActions;

        var gameForm = window as GameFormSDL;
        if (gameForm != null)
        {
            //gameForm.AppActivated += OnActivated;
            //gameForm.AppDeactivated += OnDeactivated;
            gameForm.UserResized += OnClientSizeChanged;
            gameForm.CloseActions += GameForm_CloseActions;
            gameForm.FullscreenToggle += OnFullscreenToggle;

        }
        else
        {
            window.ResizeEndActions += WindowOnResizeEndActions;
        }
    }

I will publish the project in a public github repo in a bit for anyone to take a look at. and if all looks good I will look into adding it as a project template like the FPS example.

[Doprez]

here it is: https://github.com/Doprez/custom-stride-window-poc

[Doprez]

Thanks - looking over the POC, I don't see any use for TickLock or ImageSerializer, why are those made public ?

I was hoping to create a secondary demo for that. The ImageSerializer was needed public because it has a specific reference in the Game class where I want to focus for the second demo. Same thing with TickLock I need to do some more research on how it can be used by regular users.

The second demo goal is to have a fully custom GamePlatform and GameBase implementation and depending on what I can learn from that I will also see about having separated window and frame rate logic.

I may want this PR to be draft for now, I had some time when I originally started this PR but I am starting to get a bit swamped with work. If my time frees up this weekend I will get back on it in a couple of days.

[Doprez]

Just finished up some more work on this. I decided to try and go for the big fish with creating an Avalonia control that can render Stride in this branch of my external repo https://github.com/Doprez/custom-stride-window-poc/tree/avalonia-custom-platform

Avalonia control

This gives an example of the Stride systems below:

• custom GamePlatform which manages the Game windows lifetime
• custom GameBase implementation that can configure and run the GamePlatform
• custom GameWindow that will create the relevant GameContext and adds the Avalonia Control for reference

Whats missing:

• A custom Input system to look for Avalonia events (this should be fairly easy to add from what I have seen in Avalonia)

Whats broken???: The most important part... rendering. Currently, I am stuck in the creation of the GraphicsDevice within my GamePlatformAvalonia class at line 59.

At this point I am well beyond my comfort zone of knowledge so this will definitely be the longest part for me to research and look into. If anyone is willing to take a look maybe it can go a bit faster with some better knowledge but I will still be looking into it as I find time of course.

API improvement findings and thoughts

This also gave me a lot of knowledge on some of the systems in place that definitely need to be cleaned up. Currently the flow is a mess to work with due to overlapping references.

• GameContext needs a GameWindow with a valid control (I think this is "fine" it does require the user to know to create the context beforehand and add it in the Game.Run(GameContext) call) Example:

  public GameWindowAvalonia(Control control)
  {
  GameContext = new GameContextAvalonia(control);
  Initialize(GameContext);
  }

• GameBase needs a valid GamePlatform (This was the more painful to deal with. since GamePlatform also needs a valid GameBase which had me stuck until I just created a separate constructor that allowed me to update the protected reference. In my eyes GamePlatform should be able to be created without a GameBase and maybe should be attached to the GameContext instead for the user to add in themselves.) Example: This is where I addeded the other constructor option

• GamePlatform needs a valid GameBase (Same as above, just added for list clarity) Example: Where I need to add the GameBase reference

• GameSystems are tied to rendering logic (This isn't the end of the world but it does mean that having a headless instance of Stride that uses the same update logic as existing systems is not possible without some changes to some of them. I think the issue was mostly debugging rendering if I remember correctly but that was the reason I decided to go with the Avalonia work.)

[Doprez]

Probably the last thing Ill get done at least for this weekend but a few updates in case I forget.

API improvements

I cleaned up the creation of GamePlatform and GameBase a little bit. As I mentioned before, one of the issues was the tight coupling of each class to one another. Now you can create the GamePlatform without the need of the GameBase reference which should be better than before. Instead you now have a new virtual method called ConfigurePlatform that sets itself up when constructing the inherited GameBase class. creation example:

gameWindow = new GameWindowAvalonia(this);
gamePlatform = new GamePlatformAvalonia(); // now you can make the game platform before the game is set up.

game = new AvaloniaGame(gamePlatform);

game.Run(gameWindow.GameContext);

Previously this was impossible since one could not exist without the other.

Remaining known issues

Swapchain issue

I resolved the problem with the creation of the SwapchainPresenter. It turned out that I was missing the logic in the GameWindow that told the PresentationParameters what the Backbuffer width and size should be.

New serializer issue

I stumbled onto another problem that I haven't been able to find out what I did. I somehow broke the ContentManager to not be able to find the serializer for my main scene. I'm not sure if the issue is that I did not properly configure the ContentManager or if I am missing some logic somewhere to add the required serializers.

Stride.Core.Serialization.Contents.ContentManagerException: 'Unexpected exception while loading asset [MainScene]. Reason: No serializer available for type id f5f46c6b5d0f4bd02f74731460ad7a13 and base type Stride.Engine.EntityComponent. Check inner-exception for details.'

The good news is, I did not break everything, only the Avalonia project is broken so I am assuming the issue lies within the custom Game implementation or custom GamePlatform.

[Doprez]

Ok, I lied... The fact that this wasn't working drove me nuts so I did a bit more testing and now it actually renders in Avalonia!

Serialization issue

Turns out the issue with this is that the GameBase class can not be outside of where the Assets folder is in the csproj. I am guessing the reason for this is how the Asset DB is created? this is the class that fails and is in the separate Avalonia dedicated project. I copied the exact same code in the new class GameCopyTest.

I dont think anything I would have done would have broken this so it might be out of scope for this but I may want to fix it anyway since it messes with what I need to do for input handling.

Rendering

Turns out the problem with rendering was that the backbuffer image was being covered by the Avalonia Window background. Funnily enough I found this out by angrily dragging the size of the window only to notice that I could see the blue of the scene peeking behind the window.

The good news is that means that controls should be able to be rendered on top of the scene. Im saying this without testing as Im just happy that anything I have done works lol.

I also need to do a better job of handling events between Stride and Avalonia.

• Resizing kind of works, it just does not currently resize the backbuffer properly so it only stretches and squeezes the image.
• It does not exit gracefully when closing the window
• framerate is limited as it doesnt know when its in focus
• minimizing seems to work
• these are just the ones I have tested

Next steps

Input

This will be the next big change. As I said before the content manager gets in the way, but otherwise this should be as simple as registering Stride events to the ones that exist in Avalonias Window class, this will be more busy work that I can tell.

Avalonia improvements

for testing I went with the easy approach of just adding the logic directly to the window. This is perfectly fine unless the user wants to create a dockable control like Kryptos-FR will likely need once the gamestudio rewrite is stable.

For this I will likely not need to change too much to make it work with a control. I will just need to tie resizing events to the control position and bounds as well as create Input events that are tied to the control again for positioning and bounds logic.

[Doprez]

@tebjan would you be able to take a look at what I have done here? I am pinging you as you mentioned something in another post about using a handle from Angle so Im wondering if what I have done here so far is valid in terms of maintainability or long term use.

Link reference in case you dont have it: https://gist.github.com/westonsoftware/a3fa982397fe1817ece4a27d3cbc5a89?permalink_comment_id=4976118#gistcomment-4976118

Also the link to my test project: https://github.com/Doprez/custom-stride-window-poc/tree/avalonia-custom-platform

[manio143]

Turns out the issue with this is that the GameBase class can not be outside of where the Assets folder is in the csproj. I am guessing the reason for this is how the Asset DB is created? this is the class that fails and is in the separate Avalonia dedicated project. I copied the exact same code in the new class GameCopyTest.

That's a very interesting situation - the error you got says that the serialization system cannot find a registered serializer for EntityComponent class. I would imagine this were to happen if you haven't loaded the Stride.Engine assembly into memory yet before trying to perform the deserialization. But that shouldn't be the case since your class has fields of types from that assembly. Which means there's some other source of confusion. I would suggest to do a clean rebuild (removing the compiled assets, the obj folders in all projects) in case it's caused by some change in dependencies while a cached asset compiled with previous dependency version remained.

[Doprez]

Turns out the issue with this is that the GameBase class can not be outside of where the Assets folder is in the csproj. I am guessing the reason for this is how the Asset DB is created? this is the class that fails and is in the separate Avalonia dedicated project. I copied the exact same code in the new class GameCopyTest.

That's a very interesting situation - the error you got says that the serialization system cannot find a registered serializer for EntityComponent class. I would imagine this were to happen if you haven't loaded the Stride.Engine assembly into memory yet before trying to perform the deserialization. But that shouldn't be the case since your class has fields of types from that assembly. Which means there's some other source of confusion. I would suggest to do a clean rebuild (removing the compiled assets, the obj folders in all projects) in case it's caused by some change in dependencies while a cached asset compiled with previous dependency version remained.

So I just tried now with a clean and removed all of the folders manually for bin and obj. Its the same problem though. As soon as the GameBase inheritted class is not in the main project, it fails to find serializers.

It might be important to note that it seems like it doesn't fail when loading the GameSettings serializer? I was stepping through and I noticed that it loads the GameSettings first and then fails after with the Scene serializer.

[Doprez]

funny enough, when I inherit the Game class from the main project and add some logic to it in the Avalonia project it works fine...

[Doprez]

I took some time to get most of the inputs working between Avalonia and Stride. Its mostly working but its important to keep note of some limitations:

• GamePad support does not exist AFAIK (maybe we can keep using SDL bindings?)
• Locking the mouse is not possible with Avalonia currently (Again maybe SDL can be used for this?)
• I think touch inputs are handled by the same API as mouse inputs? This should be fine I think but worth noting depending on how this should be integrated.

[Doprez]

Access changes and questions

Fullscreen

I did some small cleanups in this PR, SetIsReallyFullscreen is back to internal since it seems like a sort of hack work around for the graphics manager to update the game window without calling the events again. Saying this though, this should probably be done better if this was being called to fix a self inflicted issue.

To set true fullscreen there are currently 3 ways of updating it from the user.

• GraphicsDevice.Presenter although this seems unstable
• GameWindow.IsFullscreen calls events to multiple different areas including using true fullscreen in winforms and crashes in SDL and my Avalonia example.
• GameWindow.BeginScreenDeviceChange abstract so this may not work depending on the implementation of GameWindow.

I have added one new method for setting windowed borderless fullscreen called SetBorderlessWindowFullScreen. This is virtual so it is up to the user to manage it if inheritting GameWindow otherwise it will just change the accessible bool FullscreenIsBorderlessWindow.

I think I am happy enough with this but some extra opinions would be nice. The only reason GameWindow.BeginScreenDeviceChange is public is because it is accessed through the Stride code and since its abstract it depends on the user implementing logic so it cant be internal.

Set size

There are currently 3 ways of setting the screen/window size.

• GameWindow.SetSize This is the typical set size method that also updates the backbuffer through an event.
• GameWindow.Resize This is an abstract method that contains the users implementation of resizing window borders.
• GameWindow.PreferredWindowedSize annoyingly seems to do nothing but is again updated by the GamePlatform so it cant be a protected set which seems ideal to me.

Extra things

Im going to ignore gamepad support for this Avalonia POC only because it would just be a copy paste of the SDL input but without the window.

I'm also going to ignore touch for the same reason. It fully depends on knowledge of the UI framework someone is using so the SDL example should be enough for people to work off of from Strides POV.

I got the mouse input lock working but it seems to have bad stutter so if someone has a better way of what I'm doing I would love to hear it. https://github.com/Doprez/custom-stride-window-poc/tree/avalonia-custom-platform/MyGame3/MouseHelpers (edit: looks like I was moving the mouse twice, it should be a lot better now.)

[Doprez]

This is going to sit as draft while we look into the new Silk.NET.Windowing 3.0 proposal as a replacement.

If anyone waiting on this has any concerns, questions or ideas please feel free to add your comments to the discussion about this https://github.com/stride3d/stride/discussions/2524