Add support for embedding the game window in the editor

[reduz]

Describe the project you are working on

Godot

Describe the problem or limitation you are having in your project

Several users who want to switch to Godot (or switched to Godot already) miss the functionality provided by engines such as Unity for "exploring" the running game.

While Godot can show you all the running game contents in the remote scene tree and its sort of possible to override the remote camera, it still leaves a lot to be desired.

Engines like Unity or Unreal let you run the game within the editor window. They achieve this by running inside the same editor process, which lets you explore the game while you run it.

This approach has the clear advantage of allow you to explore the running game more intuitively, but it also has several disadvantages:

• Non-destructive editing of the running game is impossible or in a very limited fashion.
• The game has to be saved, the cleared before playing.
• The game has to be reloaded after playing.
• The editor stability can be easily compromised if the running game crashes.

None of these are problems with Godot however. Godot runs the game in a separate process and lets you explore its contents via IPC, but as the game window is not embedded, exploring the content becomes more unintuitive and annoying.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

The idea is to have an embedded game mode in Godot. This proposal describes specifically how this could be done in a way that is feasible and should allow most of the expected behavior.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

The main goal is to have a new mode in the top bar called "Game", where the game can run embedded in the editor. It would look something like this:

(Note: Pretend that the icon in game is a joystick icon and I am a very skilled graphic designer).

The idea is that the game runs in this window, at either native resolution or letting you stretch the window to fit it (zooming options can be discussed further, but you get the idea).

What functions are supported and what are those icons?

• Pause: Ability to pause the running game
• Next Frame: Ability to move to the next frame while paused.

• ---

• 2D/3D selection mode: This lets you select which kind of nodes and view will be interacted with while exploring the scene.

• ---

• Game input: When this icon is toggled and the game is focused (click on it to focus), all input is captured and goes to the game until user clicks elsewhere. (Of course if the game captures the mouse, we will need some way to un-capture it with some special keybinding).
• Selection Mode: This mode no longer sends input to the game and it lets you select 2D/3D nodes (depending on the selection mode described before) by clicking on them. They will appear on the remote scene tree and inspector so you can see their content.
• List Selection Mode: Same as the mode above, but it presents a list of nodes you can choose to select instead of selecting the topmost one.

• ---

• Camera override: This lets you override the game camera and pan/zoom (if 2D is selected) or WASD/similar (if 3D is selected).

Implementation notes:

The main idea of this feature is that Godot will remain running the game as a sub process, with the main difference being that the game window is embedded into the editor. This can be done with relevant Vulkan extensions (and worse case if a platform does not support this, it just does not let you embed the game window, but most should support it).

FAQ:

Q: Are there not other 10 proposals about this? A: Yes, but this explains how it would be done in a way that is feasible technically.

Q: Will the 2D and 3D editors show the game objects moving around? A: No, this is not possible and probably never will. Exploring will have to happen from the game window itself. We can add tools to make this as comfortable as possible.

Q: Will I still be able to edit the scene while the game runs in a non-destructive way? A: Yes, unlike other engines, you will not lose any change made to the scenes you are editing and the changes will also replicate to the running game.

Q: Will the editor stability be compromised by the game? A: No, it should not be.

Q: Is this feature optional? A: This feature is not only optional but probably only present for platforms and rendering drivers where it is possible to do this kind of embedding. I suppose users will be able to configure whether they want to use it by default or not in the editor settings.

If this enhancement will not be used often, can it be worked around with a few lines of script?

This is core

Is there a reason why this should be core and not an add-on in the asset library?

N/A

[aaronfranke]

This would significantly increase the usability of the editor camera override. Currently you need to have both the editor and the game visible on screen, then enable the camera override, and you need to have the editor focused to move the camera in the game. This proposal significantly improves the situation so that you only need the editor visible.

[yosoyfreeman]

Could be an option to not save the changes made while running the game in this mode? I personally miss it to quickly test things, spawn objects or change values being sure it will revert when i stop it.

[aaronfranke]

@yosoyfreeman You would not be spawning objects into the running game directly. You can only spawn objects in the saved scenes, and in many cases Godot can sync that to the running game. This proposal does not change this behavior, it only improves the usability by allowing the game window to embed into the editor.

[bitbrain]

I really like this proposal! How would this behave if I were to set a breakpoint somewhere? I suspect since it still runs in a sub-process, the game would be set to paused if it reached a breakpoint?

[Bastienre4]

As a professional game developer currently working with Unity and wanting to switch to Godot, this is one of the feature I miss the most.

If I have an issue about the UI, I need to be able to inspect (both in terms of data AND visually, in the scene/game) the node/gizmos to understand what's wrong. Debugging without this is a pain 😅.

[BoneDaddy64]

Coming from Unity, all I really want is to have a scene view (probably should be embedded) that has a separate camera from the game view but renders the same scene state, as in same objects (transforms, models, animation state etc. Unity allows for different shading which is good) that I can fly around with, and to be able to pick (as in click in that view on the rendered object's model) and have the Inspector show that node (and be able to change it, like when in Remote in the Scene list currently). Whether the game view is embedded or not doesn't matter that much to me.

Basically I just want to be able to quickly fly around, select objects, have those objects appear in the Inspector, be able to real-time modify the objects (which Godot has the functionality for now). That's what's really lacking right now for me in Godot, makes it really cumbersome to "touch" my game as it's running

[aaronfranke]

@BoneDaddy64 Depending on the details of how the implementation of copying the texture to the editor works, it might be possible to have the editor camera override add a new viewport + camera and read the texture from there, instead of overriding the existing camera's transform, so the game view could keep the same camera. That would be a separate proposal made after this one, but it would help if this feature was designed with that use case in mind.

[allenwp]

As a professional game developer currently working with Unity and wanting to switch to Godot, this is one of the feature I miss the most.

If I have an issue about the UI, I need to be able to inspect (both in terms of data AND visually, in the scene/game) the node/gizmos to understand what's wrong. Debugging without this is a pain 😅.

This comment about gizmos brings up the question of whether gizmos could be drawn over top of this new Game view mode to visually show what is selected. Further to this, there are questions about whether these gizmos could modify the remote node properties, such as using the transform gizmos to translate, rotate, or scale remote nodes.

Personally, I suspect that this is something that should be kept in mind when implementing this new feature so that full gizmo support could be added at a later time. I think it would be great to get something like this implemented without gizmo overlays sooner, rather than delaying the feature to have full-featured gizmo overlays.

I also think that just being able to see gizmos in the scene would be immensely helpful to visually distinguish what node is selected and shown in the inspector panel. Being able to modify the node's properties using gizmos is a less valuable feature because this can be done through the inspector panel.

[sosasees]

i need to test window transformations often — like moving and resizing — especially when they are not triggered in the OS but in the game.

so the current way of running the game in its own window saves me much time.

so if people want the game window embedded, this should be an option and not the only way.

[allenwp]

I previously started a discussion regarding "Spacial Editor - Focus Selection should focus on remote nodes". I think this new proposal should address this feature: When in the new Game view mode in "Selection Mode", pressing the F key to Focus Selection should focus the camera override on the currently selected remote node.

This is especially helpful when you have nodes that are far away from your game or override camera, such as an off-screen world object. With this Focus Selection feature implemented for the Game view mode, it would allow the user to select the node in the remote scene tree, Focus Selection, and have the override camera focus on this node.

I also noticed that the original post mockup screenshot does not show a remote scene tree view at all, and it seems that the proposal suggests a "List Selection Mode" instead. How does this List Selection Mode compare to the existing remote scene tree? I'm curious to know more about this, especially in the context of how it will interact with node selection and the Focus Selection action.

[m4gr3d]

This would be a welcome improvement for the Android editor, given the limitations on tablets and phones to have multiple windows open side by side.

[robbertzzz]

I'd personally really like if the UI would allow for switching between different performance views. I sometimes want to see wireframes or overdraw in my game, but can currently only do it by adding additional code to the game. Some UI for it like Unreal has would be amazing!

[NathanLovato]

Overall, it sounds great, and it's been requested time and time again, as you know. This is especially big for professionals coming from Unity and Unreal.

UX-wise, do you plan to unify the play scene functionality (the transport controls in the top-right) with this? Here are some thoughts to complement the proposal regarding making this view work with the existing transport control, UX-wise.

Unifying the transport controls with this new Game main screen:

• By default, playing the scene or game with F5 or F6 shows the running game in this new embedded Game main screen.
• Pressing F8 stops the game view and takes you back to the last main screen you were working with (2D or 3D mainly).
• An icon similar to the one to pop out the script editor is present on the Game main screen and lets you float the Game main screen view to another screen.
• When the game window is floating, the toolbar stays attached to it (I understand this may not be feasible, just an idea to have the pause and stop and other controls right by the game)
• An icon on the Game view allows you to toggle it full-screen (to have this game view full-screen on another display, e.g. for two-people test sessions where one person plays the game with a gamepad and a developer or level design makes changes with hot reload).

These are just quick ideas to spark discussion, to see if that's possible or the intention at all.

The idea would be to unify the current play scene functionality with this new view, so that users have one consistent and powerful experience when running their scenes or games in the editor.

If we can do anything to help, please let me know. We now have an experienced UX/UI designer specialist in the team, we can help with mockups, icons, and focused user testing moving forward if the team needs extra hands.

[reduz]

@NathanLovato yeah it does make sense to do this if you are using the embedded mode.

[yosoyfreeman]

* By default, playing the scene or game with F5 or F6 shows the running game in this new embedded Game main screen.

* Pressing F8 stops the game view and takes you back to the last main screen you were working with (2D or 3D mainly).

* An icon similar to the one to pop out the script editor is present on the Game main screen and lets you float the Game main screen view to another screen.

* When the game window is floating, the toolbar stays attached to it (I understand this may not be feasible, just an idea to have the pause and stop and other controls right by the game)

* An icon on the Game view allows you to toggle it full-screen (to have this game view full-screen on another display, e.g. for two-people test sessions where one person plays the game with a gamepad and a developer or level design makes changes with hot reload).

These are just quick ideas to spark discussion, to see if that's possible or the intention at all.

The idea would be to unify the current play scene functionality with this new view, so that users have one consistent and powerful experience when running their scenes or games in the editor.

As @sosasees said, the current workflow is solid and clean and should be maintained. I don't think is a good idea to default to this new mode and needing extra clicks to get to the floating window that was default. I think specific shortcuts for this mode would make more sense from a usability perspective, so if its not possible to disable it, at least do not interfere.

I think that the toolbar should remain where it is. Is the editor the one who play or stop scenes, so for me it makes more sense that is there, always in the same place.

I think this is good, specially for new users, but should not replace what is already there.

[Bastienre4]

The best answer to this "Should it be default or not" is to add an editor setting to define this. So that everyone is happy :)

[sosasees]

• When the game window is floating, the toolbar stays attached to it (I understand this may not be feasible, just an idea to have the pause and stop and other controls right by the game)

there are multiple ways this could be done:

toolbar inside window title bar

• ⭕️ it's very clear that the toolbar belongs to the window

but it has several issues that make it a bad fit for Godot:

• ❌️ if users want to add their own buttons to their game's window title bar (if that ever becomes a feature), they can't see and use them in editor builds
• ❌️ borderless windows have no window title bar, so the toolbar disappears in borderless windows
• ❌️ if the window is too narrow, the toolbar has to be truncated with an overflow menu (and with extreme small window sizes like 64×64px, even the overflow menu becomes inaccessible because it's overlapped by the minimise/maximise/close buttons)
• ❌️ Windows (and probably iMac, i don't know enough about iMac to know for sure) have good support for additional window title buttons like this, but not Linux (at least not without hacks)

toolbar under window title bar

this seems similar to toolbar inside window title bar, but it's much better like this:

• ⭕️ much more room for window title
• ⭕️ window title bar stays as-is, so on Linux we don't need to use hacks to change it
• ⭕️ toolbar is outside the window title bar, so it doesn't disappear in borderless window mode

of course, it still has some of the same advantages and disadvantages of toolbar inside window title bar:

• ⭕️ it's very clear that the toolbar belongs to the window
• ❌️ if the window is too narrow, the toolbar has to be truncated with an overflow menu (but this time the overflow menu stays accessible in extreme small window sizes like 64×64px because nothing overlaps)

and we need to be cautious of

• ❌️ Godot needs to subtract the toolbar height from the window height (just like it subtracts the window border) if we want to get the real game viewport height value

toolbar in own window

• ⭕️ toolbar does not truncate if the game window has an extreme small size like 64×64px
• ⭕️ window title bar stays as-is, so on Linux we don't need to use hacks to change it
• ❌️ connection to the game window is more abstract
• ❌️ we need extra window management programming so that the toolbar window moves as expected:
  • if it has not been moved by the user since the game started, it should move together with the game window
  • if it automatically moves to a position where it goes (at least partially) off-screen, it should move down until it's fully visible
  • if it has been moved by the user, it should not move automatically anymore until the game is restarted
  • it should be always on top if the game window is focused

toolbar under window title bar, but as its own borderless window

¿why didn't i think of this much earlier? 🤦‍♂️ (i added it when editing the comment 3 days later)

this combines the best of toolbar under window title bar and toolbar in own window:

• ⭕️ it's very clear that the toolbar belongs to the window
• ⭕️ toolbar does not truncate if the game window has an extreme small size like 64×64px
• ⭕️ window title bar stays as-is, so on Linux we don't need to use hacks to change it
• ⭕️ toolbar is outside the window title bar, so it doesn't disappear in borderless window mode
• ⭕️ we don't need much window management programming to keep the toolbar in place: just always move it to the same position relative to the window
• ❌️ Godot needs to subtract the toolbar height from the window height (just like it subtracts the window border) if we want to get the real game viewport height value

[NathanLovato]

An editor setting is good, but the default experience will still be everyone's first experience even if you add a setting, and this has a massive impact on discoverability if not handled carefully.

It should at least be thought about seriously what should be people's default experience, weigh the pros and cons. The current pop out window is seen as one of Godot's weaknesses by many users coming from Unity and Unreal, and having worked professionally with both, I understand why.

Godot's pop out window is amazing for two people, with one playtesting on one display, and the other working on the project live in the editor on another display.

However, for debugging, the ability to move your camera in Unity to a given place and switch immediately to the game view in-place, test the bug instantly, and tap out to the editor mode, is something you end up using constantly in production.

---

Note that I personally don't mind either, with what we do at GDQuest, we're not too concerned by discoverability problems, because we always keep up with the latest development in Godot. This point is about taking the time to consider the wider user-base, current and future, and thinking about what's the best experience for them.

[Bastienre4]

An editor setting is good, but the default experience will still be everyone's first experience even if you add a setting, and this has a massive impact on discoverability if not handled carefully.

It should at least be thought about seriously what should be people's default experience, weigh the pros and cons. The current pop out window is seen as one of Godot's weaknesses by many users coming from Unity and Unreal, and having worked professionally with both, I understand why.

Godot's pop out window is amazing for two people, with one playtesting on one display, and the other working on the project live in the editor on another display.

However, for debugging, the ability to move your camera in Unity to a given place and switch immediately to the game view in-place, test the bug instantly, and tap out to the editor mode, is something you end up using constantly in production.

Agreed. Also, I'm currently working on a single 21:9 display, and having the floating player window everytime I want to test the game is a pain if, as you said, I need to go back and forth in the editor.

[NathanLovato]

Bringing a good point/suggestion from a Twitter user: for mobile game developers, but also app developers, it's useful to be able to quickly test the view at different sizes matching popular mobile devices, to see if the game/UI scales properly on different displays.

So they pointed out that Unity, but we could add web browsers, offer a dropdown to quickly change the preview size to these devices' resolution.

This may be out of scope for this proposal, but seeing that the game view would be resizable, would it be possible to have a dropdown or something to select common presets? For various mobile phones, tablets, and typical desktop display ratios (16:9, 16:10, and the likes).

Please let me know if it should go into a separate proposal.

[yosoyfreeman]

An editor setting is good, but the default experience will still be everyone's first experience even if you add a setting, and this has a massive impact on discoverability if not handled carefully.

It should at least be thought about seriously what should be people's default experience, weigh the pros and cons. The current pop out window is seen as one of Godot's weaknesses by many users coming from Unity and Unreal, and having worked professionally with both, I understand why.

Godot's pop out window is amazing for two people, with one playtesting on one display, and the other working on the project live in the editor on another display.

However, for debugging, the ability to move your camera in Unity to a given place and switch immediately to the game view in-place, test the bug instantly, and tap out to the editor mode, is something you end up using constantly in production.

Note that I personally don't mind either, with what we do at GDQuest, we're not too concerned by discoverability problems, because we always keep up with the latest development in Godot. This point is about taking the time to consider the wider user-base, current and future, and thinking about what's the best experience for them.

I understand the importance of the first experience and fully agree with you. I may be failing to see the benefits tho. I think that what users miss the most is the fully editable and reversible playback and this that Reduzio pointed out:

Q: Will the 2D and 3D editors show the game objects moving around? A: No, this is not possible and probably never will. Exploring will have to happen from the game window itself. We can add tools to make this as comfortable as possible.

We would still missing the two more famous features of those engines. So in the end, there is still a learning curve towards a different approach.

Just to understand better, you talked about quickly moving the camera to the point and moving things and seeing it in game. Is this not that?

Grabación de pantalla desde 2023-07-07 11-28-51.webm

I do agree that some way to quickly test screen ratios would be great.

PS: Please, remember that i lack social skills and i apology if my tone is not the right one. I'm just trying to give the feedback you asked in an organized way.

[sosasees]

a dropdown to quickly change the preview size to these devices' resolution

some way to quickly test screen ratios would be great.

i just got a new add-on idea: window size panel

i will most likely not make this add-on because i only make add-ons that i need. otherwise i would make too many add-ons once and never update them to newer Godot versions.

[robbertzzz]

An editor setting is good, but the default experience will still be everyone's first experience even if you add a setting, and this has a massive impact on discoverability if not handled carefully.

Unreal has a drop-down menu next to the play button which allows you to pick how the play button behaves. I really like it because it's immediately clear to the user yet works as a project setting.

[lunarcloud]

As long as the current behavior (run it like for real) remains. It's actually something I really like.

[mmMarzban]

I want that!

[geekley]

To me, the most important feature here is this:

Selection Mode: This mode no longer sends input to the game and it lets you select 2D/3D nodes (depending on the selection mode described before) by clicking on them. They will appear on the remote scene tree and inspector so you can see their content.

Godot really needs a way to select nodes (instantiated at runtime) in the Game View by clicking, even if it's in the current implementation with separate window. In fact, I hope that when this is implemented, we can still use all those new features regardless of whether the game view is embedded or detached / separate window like now. If you're gonna implement this step-by-step, this would be the best thing to have first IMO, and if you wanted to back-port just 1 thing from here to 3.x, this would be the most helpful (and the one feature that has no workarounds). (I'm not asking to back-port btw, just saying that to say it's the most important feature)

List Selection Mode: Same as the mode above, but it presents a list of nodes you can choose to select instead of selecting the topmost one.

Hopefully it will also unify with previous feature without requiring the separate mode/button, with alt+right-click.

[geekley]

Camera override: This lets you override the game camera and pan/zoom (if 2D is selected) or WASD/similar (if 3D is selected).

I'm assuming this would allow to have the camera override in the Game View independently of the Scene View's camera, which is great. But you could also keep/reuse the current equivalent button in Scene View to decide if you still want to sync both view's "editor cameras" or move them independently.

[geekley]

However, for debugging, the ability to move your camera in Unity to a given place and switch immediately to the game view in-place, test the bug instantly, and tap out to the editor mode, is something you end up using constantly in production.

Just to understand better, you talked about quickly moving the camera to the point and moving things and seeing it in game. Is this not that?

@yosoyfreeman Yes, but because you can't click to select remote nodes that you instantiate only at run-time, it's not easy to test bugs. In Unity you don't have this issue. That's why I think Selection Mode in Game View is the most important missing feature here, and IMO should be the one with highest priority among those listed.

[GeorgeS2019]

Not sure if someone written, this feature is good for unit testing scene without resort to godot.exe -scene testscene.tscn

[Calinou]

This can be done with relevant Vulkan extensions (and worse case if a platform does not support this, it just does not let you embed the game window, but most should support it).

If I'm understanding right, this means having to implement something akin to https://github.com/godotengine/godot-proposals/issues/5790.

[MajorMcDoom]

Just curious, instead of bringing the game into the editor... why not bring a portion of the editor features into the launched game?

For example, you have the option to launch with "editor mode support" in addition to the default option of launching just a plain game window. In this mode, you'd be able to toggle the "editor mode" (similar to the different input modes described above). But in "editor mode", it also has a 3D or 2D view, play/pause/step, tree view, inspector, etc. This would check all the requested features (live viewing, selecting, and editing).

Is this not feasible? Would those features be too difficult to graft into a context that doesn't have the rest of the editor?

[aaronfranke]

@MajorMcDoom That would involve reimplementing the high-level parts of the editor to work within a game, and would cause the game to have a lot of objects that don't exist when exported, artificially increasing load times and memory usage to load that stuff up every time you start the game from the editor. So I'm against that.

[MajorMcDoom]

@MajorMcDoom That would involve reimplementing the high-level parts of the editor to work within a game, and would cause the game to have a lot of objects that don't exist when exported, artificially increasing load times and memory usage to load that stuff up every time you start the game from the editor. So I'm against that.

@aaronfranke How would this be different from, say, making a dev build that would contain a lot of debug symbols and have more mem usage and worse performance? The user would specifically be opting into launching a game that has extra editing features. This isn't something invisible that would affect users globally. Not to mention, this would replace the current remoting code, since all the inspection would happen within the launched game itself. That's a lot of weight lifted.

As for having to re-implement editor parts to work within a game: These are the same features users are currently expected to implement due to the lack of live-editing. Godot already has the pieces. If it is too much work to implement from existing tech, how could we justify asking users to do it from scratch?

EDIT: My apologies if this is veering off topic. It was an adjacent idea that perhaps should have its own separate issue.

[robbertzzz]

@aaronfranke keep it in a debug build, remove in a release build. Or add it as a setting in the export window. Add an editor setting that controls it on the editor side. I'd love a built-in debug overlay that allows for custom scripts so users can add their own tooling to the in-game debug UI!

I think a good reason to take this approach rather than doing game-in-the-editor like all the other engines is that having the game in the editor provides a very different experience when testing. Playing in fullscreen completely changes the player experience IMO and it allows developers to properly test performance at full scale. I've always hated that about Unity, the fact that you can't really test the game like how it would be in a build. Both Godot and Unreal do allow that option from the editor and it would be a shame to replace it.

[Calinou]

Playing in fullscreen completely changes the player experience IMO and it allows developers to properly test performance at full scale

We could add an option to automatically enable distraction-free mode when running the project (and possibly also make the editor fullscreen), and restore the previous state when stopping the project.

Distraction-free mode still has borders on screen edges, but we could add a dedicated mode (such as "Zen mode") that makes the viewport truly fullscreen. This would also be useful for rendering performance testing.

[geekley]

I've just noticed the screenshot shows support for multiple tabs. Is this on purpose? If it is, I assume it's not for multiple running game instances, right?

Does it mean there is any intention of supporting multiple views in the running game, with independent toolbar selections? E.g. one for the game camera in game input mode and one or more for editor cameras in selection mode? If so, would you be able to split them (horizontally or vertically) so you can have game camera and editor camera on-screen simultaneously without needing multiple detached windows?

When you can/will only run a single game instance, having a tab bar that will have just 1 (or mostly no more than 2) tabs on it is a waste of precious screen space for the running game. It would be better to just drop it and add/incorporate more buttons instead to the toolbar: Split vertically | Split horizontally | Detach window | Fullscreen | Close And make the view draggable by the toolbar, so you can swap its placement with other split views.

If vertical splitting makes toolbar space an issue, you can unify 2D/3D into a single button (unless they're independent), as well as Input/Selection mode (right-clicking would show the 3 options, including List Selection too).

[mystchonky]

I am more interested in what this means for other programs. If the game embedding support is done in an extensible way, we could use the same code to power addons. We should be able to embed arbitrary programs like image editors and music synthesizer into the editor and have a cohesive workflow. Even if it is restricted to godot apps, it would be huge boost to addon support. Immediate benefit would be for apps like Pixelorma which wont need tool scripts anymore and maintain a separate repository if we wanted to support godot editor embedding.

[frkntlr]

is this offer still valid? Any progress on this?

[Calinou]

is this offer still valid? Any progress on this?

To my knowledge, nobody is currently working on implementing this. It's a difficult task, so it may take a while.

[ShlomiRex]

Is there detailed list of things to implement in order for this feature to be completed?

[Calinou]

@follower has started prototyping this (only for Linux using FFmpeg capture), but there is a lot that needs to be done:

• Implement off-screen rendering for Vulkan and OpenGL, so that the game process can render without creating a window. Vulkan has standard extensions for this, but not OpenGL (especially OpenGL <= 4.1, which is the last version available on macOS).
  • Input processing and audio playback/recording must remain functional when in off-screen rendering mode.
• Display the root Viewport's texture in the editor using inter-process communication. This can be challenging from a performance standpoint, as a 1920x1080 frame is already over 2 million pixels - and you need to transfer that 60 times per second or more. Lossy compression shouldn't be used, as the original image should be visible at all times with no loss of quality.
  • We should also avoid relying on hardware readbacks, as these incur at least 2 frames of latency (that's ~33.3 ms at 60 FPS).
• When the game is running and its texture container is focused, forward all inputs to the game process.
  • This is not required for gamepad input as it's available in non-focused windows as per OS behavior.
• Implement editor UI functionality, such as displaying an overlay when the game process is running with various options.
• To be able to select nodes for inspection while the project is running, dedicated code needs to be written for this (the game doesn't have access to editor-specific code), both for 2D and 3D.

Regarding supported platforms, I only expect this to work on desktop platforms. I doubt the Android editor will be able to benefit from this given all the security limitations in place, but we never know. The same goes for the Web editor.

I would recommend looking at implementing off-screen rendering first. If you have Vulkan or OpenGL experience and are willing to help, please drop by the #rendering channel of the Godot Contributors Chat 🙂

[Scherholz]

i need to test window transformations often — like moving and resizing — especially when they are not triggered in the OS but in the game.

so the current way of running the game in its own window saves me much time.

so if people want the game window embedded, this should be an option and not the only way.

The idea, at least for me, is to have both running at the same time, both embedded and the other regular one we already have in Godot. It is easier to understand if you look at how Unity does it.

[ShlomiRex]

It should look like this:

Basically:

• [ ] Add new tab to toolbar called 'Game'
• [ ] Add Play/Pause/Stop buttons to toolbar
• [ ] Add viewport to the game tab
• [ ] Render something onto the viewport (for starters)
• [ ] Render the game onto the viewport
• [ ] Implement split-view or split-screen of game and code/3d editor/2d editor/script editor
• [ ] Implement: when game running, render selected node as wireframe in editor view (like in image)

I can implement the UI stuff (including viewport) but from there it becomes difficult. I think I should be OK implementing the first 3 things.

However I'm looking at the renderer code, its much more complicated than I thought. However, maybe I can use the window framebuffer and draw onto the viewport to test if it works.

To make things easier, here is what I researched:

• renderer_viewports.cpp - renders the viewports
• renderring_server_default.cpp - starts the rendering
• display_server_windows.cpp - for windows only, in process_events() it receives MSG from queue and dispatches the messages to the rendering server
• os_windows.cpp - inside the run() function - inside the infinite while(true) loop - basically processes events (main loop)

I also think we should focus on Forward+ rendering for now, as it is the direction we should go and with less rendering pipelines it should be easier.

@Calinou when implementing small features should I start from the latest commit in the upstream, or start from stable commit, like the tag 4.1-stable?

[Calinou]

@Calinou when implementing small features should I start from the latest commit in the upstream, or start from stable commit, like the tag 4.1-stable?

All development should be done from the latest master branch.

[strich]

I'm quite concerned the approach this proposal seems to be taking is not going to achieve many of the most important aspects this proposal is inspired upon (the Unity/Unreal experience).

The most important aspect of this feature is by far the ability to inspect and edit the active running game. This includes but not limited to:

• Editing any node data in the nodegraph
• Seeing and interacting with the running nodegraph in the scene view
• Updating or replacing various base assets such as textures, meshes or most critically shaders
• Debugging code

I understand there is a lot of "Unity dev demands" flying about at the moment, but I cannot overstate enough the amount of time the latter half of production involves this kind of necessary work across the entire team.

It would be a grand shame for this proposal to continue to have time and effort put into it if the design itself is avoiding and maybe even blocking the implementation of these requirements sooner or later.

[Karl-Jansky]

@strich I'm quite concerned the approach this proposal seems to be taking is not going to achieve many of the most important aspects this proposal is inspired upon (the Unity/Unreal experience).

Correct me if I'm wrong but can't godot already do that with the option to toggle the remote scene as local? I think the only difference is that you can't really see your changes or interact with viewport tools, but from what I understand you can already interact with nodes through code or the editor inspector but those changes aren't reflected in the viewport.

[ShlomiRex]

@strich you mean like this?

https://github.com/godotengine/godot-proposals/assets/17744127/c6fdc26c-5f99-41e1-8483-01507815443c

[strich]

@Karl-Jansky @ShlomiRex yep that looks good! I mentioned that feature for completion, though I suspect the real issue I'm presenting is the connection to assets - can one edit shaders and materials?

[MajorMcDoom]

I made this Tweet a couple days ago, and I'd just like to share it here so people have a deeper understanding of the intent/needs of regular users of live editing. Specifically, part 4 covers why Godot's current features and this proposal do not meet those needs.

https://twitter.com/MajorMcDoom/status/1704625394119213308?s=20

I don't think Unity-style live editing should / needs to be rushed, but like @strich I have concerns that an "embedded window" is missing the point and that the current work actually makes things harder to reconcile down the road.

[ShlomiRex]

I think we can use Viewport and CameraTexture and display the pixel buffer into the viewport.