Integrate Remote ImGui

[VinceBret]

Hey, Not really an issue but have you tried to add the remote imgui stuff in unreal ?

[segross]

Yes, I used it in my last company, and I do have somewhere an old quick version that I did long time ago. I believe that at some point it was mentioned as a possible feature but I had to remove it because I could not find enough time and I did not want to give false hopes.

It might come at some point, but I won't promise anything right now. It is however likely that I will want to add it, as in my current work we use a vanilla version of this plugin and I sort of miss Remote ImGui. So stay tuned... but be patient :)

I will however explore alternatives before moving forward as I think that Remote ImGui is not under active development anymore and I did have some problems when using it in Unreal.

[ElMegalomator]

Is there any update on this? Would be really a useful addition.

[segross]

Unfortunately no, not at this time.

[sammyfreg]

Hi. I am the author of the library netImgui and was thinking of adding Unreal support next. When looking for current Dear ImGui Unreal integration, I came accros this plugin. I was wondering if you'd be interested for me to add support through your plugin, rather than creating my own.

[segross]

Hi. Sure, yes! That would be nice. We can perhaps sync efforts. Although I must warn you that I'm doing things very slowly recently and hardly can find enough time. I was thinking about integrating ImGui Remote and possibly other remote plugins as well but never did it. As a result, there is something that I believe may be partially helpful but you might find it annoying at first ;) The whole input state. The idea is that I can collect input and then switch between UE and remote. I did that in my previous studio but never in this plugin. It worked quite OK and I was able to use the Unreal and remote inputs at the same time - not literally but in the same session, basically whatever windows had a focus was used for input.

Out of curiosity, what platforms are supported? Do you know how easy is to get it working on consoles? And how do you compile it with UE? With the ImGui Remote, I had to use a statically linked library as I was not able to compile it as part of the UE plugin.

EDIT: I look at the netImgui and it looks really good. It is nicely done.

[sammyfreg]

Hi. I have previous experience as a graphisc programmer on UE3/4 and believe I should be able to add it myself.

I currently support Window/XBox/Android/Linux out of the box for the client code, but it was designed to be really easy to add to any platform. Only requires implementing 4-5 short functions to open, close, send, receive, listen data with sockets (can see it in NetImgui_NetworkWin32.cpp). I was thinking of implementing these functions using UE4 network code, so it then works on all platform they support. I am just not sure plugin have this level of access to core UE code. The code should be added alongside your code, so they are both compiled together.

For the inputs, The entire control is shifted to netImgui once connected, then resume local input when disconnected. I re-use the same ImGui context, but can also be started in a new context, leaving the origal intact.

EDIT: I look at the netImgui and it looks really good. It is nicely done.

Merci! :)

[segross]

I checked the SampleDualUI and I will have a few questions and notes. Forgive me if I'm missing something but I only looked briefly: 1) There are lots of options, which always is nice. 2) I noticed some context switching and restorations inside of the NetImgui API. This plugin also does a lot of contexts switching to support PIE. Will that not cause collisions? 3) Later I will also need to add an option to create other contexts for regular non-debug widgets. But those contexts will not be remotely controlled, so that should be fine as your example also has a local context. 4) In your example, I see separate blocks with NewFrame -> Content -> EndFrame flow. One for the remote and one for the local context. This plugin sets the context for the whole frame and then collects debug content from different game objects. And the flow is more less like this: NewFrame (for every context) -> Next UE Frame-> switching between contexts at the start of each world update -> EndFrame and copy render data (for every world context). I suppose using two context for the same objects will be less straightforward but I don't think this is crucial. 4) For dedicated servers and VR, this is obviously not a consideration, but is it possible to have input from the client and server used interchangeably? I'm asking because in my previous project where I used a remote connection, my preferred workflow was to connect from a browser that was running in the same desktop and then debug from browser or from game, whatever was more convenient at the moment. The game used the remote input whenever available (which was the case when the mouse cursor was inside of the browser) or own input otherwise.

[sammyfreg]

1. Not too much though, wanted this to be easy to add to any codebase using Dear ImGui, with almost no changes. :)

2. This will requires to be careful of the approach taken. netImgui can be used 2 ways. (A) Use current context without switching. (B) Use its own context, leaving the original context intact Now, we could either try to tie netImgui to only 1 context or let netImgui use its own context, not affecting any of your context. I might also need to add a new function to re-set the context, since you might have changed it in you world update and I rely on it staying the same after the netImgui::NewFrame(). Maybe this should be as simple as possible, I don't foresee a lot of use for netImgui while in the editor multi-window environement, it's more for a game launched on console/cellphone.

3. Yes, to do that, I do 2(B), which means I 'cloned' the active context on connection, and continue using only the copy.

4. This should be fine, we can have only 1 of the 'world' tied to netImgui, and everything else works the same.

5. Dual input is currently not supported, it would require some consideration on how to handle. It could be a toggle to tell netImgui where to take input from, locally or remotely. Had not considered this usercase, since I was thinking of gaming console where it's obvious you'd want the input from the server.

[segross]

I don't foresee a lot of use for netImgui while in the editor multi-window environement, it's more for a game launched on console/cellphone.

Well, you might be surprised ;) In my last project, where I had a more dirty but in some respect a more complete integration, I (like a few other colleagues) used remote connection it in my main workflow even though for the most part I was using PIE and we didn't have a dedicated server. In-game debug was very useful for recording and remote for most of the regular development. We used keyboard shortcuts to switch to which PIE remote should be bound and looked at the remote flag to figure out what input use. It was a great workflow, despite a bit hackish implementation, but I never found enough time/strength to clean and add this here - partially because at home I have only one screen and no dedicated server, what took the personal incentive out. But the remote capabilities will be very nice, even if only for servers or VR. I'm also starting to use this plugin in a project that has a dedicated server, so I will surely be thankful.

Ad 2. I'm not going to lie. I am not fully sure how that works in netImgui. From my side, I would imagine that those contexts that are for debugging could be safely tied to the netImgui. It makes perfect sense in my opinion. Some people use ImGui for non-debugging, in which case most likely it should be not tied to remote context, but we can always use a switch to control that feature. Later I want to refactor the plugin in a way that there is a clearer separation between ImGui for debugging and for other applications. One of the reasons is that I made/make some effort to isolate my debugging widget and input from the game as much as possible but this is undesirable for some applications. After that it will be clearer, what should be tied and what not. Ad 2b. How that works? Is it possible to bind netImGui to context and unbind on demand? Could it be possible to select which PIE is bound? Assuming that it will be allowed in the editor, of course, because as I understand this is not necessarily on the cards. I see that you can connect multiple clients at the same time, does that mean that I could connect multiple PIE clients? Because that seems very powerful.

Ad 5. I understand the goals and why the design. It seems well-thought and as great as the Remote ImGui was netImui seems cleaner. Paradoxically, a bit hackish nature of the Remote ImGui made it easy for us to bend it - I don't know why but somehow you feel more excused in that situation ;) It would be nice to have that capability but this is my selfish thinking and please don't take that as pushing for a feature - especially if it would undermine other goals. And especially that netImgui have other great features.

In any case, some people feel that this plugin is overwhelming, so I'm happy to explain what and why at any time. But as you probably noticed, I do looong responses (hate myself for that ;), which means they can take some time. And we are a few months before release, so it can be busy at times.

[sammyfreg]

(2) Ah, so you are thinking of using Dear ImGui for final product, gamer facing UI? Interesting. Right now, when not activating the 'cloning' feature, I just use the current context almost normally. On connection, I override the 'input' configuration settings, binding the standard Windows virtual keys ( CmdInput::eVirtualKeys ) instead, so input works the same on all client platform. Toggling which UE context we are using with netImgui is possible, but I will have to add a functionality. Make explicit the input override/restore so we can go from one to the other. Not a problem.

(2b) There is the option to clone the imgui context on connection (if requested) and then just use this copy. This leave the local context untouched and free to do whatever every frame. Not sure we need this for this case, unless you'd like to continue rendering some different Dear ImGui content in the UE window.

(5) It would not be too much trouble to support multi connections per editor. I already have all of the client data tracking encapsulated in 1 class, thinking that at some point people might want to do that :) If you look at the code in netImgui_Api.cpp, you'll see that every api call start by taking the global object gpClientInfo as the 'active' client. To support multiple client, I just need to figure out how to let user specify which one is active instead. This would be required if we want to support multiple remote clients from the same program. netImgui server only display one at a time though, you'd have to toggle between them, so not sure if this is desirable but it's definitly do-able. (5+) I never used remoteImgui, let me know if there were some features you felt were usefull and that I could add.

I just got Unreal compiled and running on my PC, with you plugin. I will slowly look at having a basic Dear ImGui displayed in a game window, and see from there what I can do. I'll probably try a very basic approach first, then we can see about how to seal with multi-windows/context setup. I hope plugin have access to UE network socket class, that way I know it'll automatically work on every platform.

Addentum: Thinking about it, 1 connection with an interchangeable active context could be the best. Your plugin could inject an entry in the MainMenubar, listing all existing contexts, and let the user switch between them to know which one is active remotely.

[segross]

Ad 2. I don't, but apparently some people do :) I think that somebody mentioned about doing some sort of visualisation and using ImGui for UI - I suppose because it is very good for data. And somebody else wanted to use that for editor widgets (thread is still opened I think)... but to be fair we had the same idea in my last company :) This plugin wasn't designed with that in mind, but I'm thinking about improving that a bit.

The toggle could happen in this plugin. Actually, it is happening and it has to happen because every world has its own debug context which needs to be set at the beginning of the world update. So it will not be a problem unless netImgui would somehow need to own the global context pointer. Context proxy is the wrapper for the ImGui context, and the context manager is monitoring creation and switching. I think that hooking the netImgui to proxy should make things easier. In general, I tried to keep ImGui specific stuff in the proxy, manager and one class for handling the demo. The rest is mostly UE related stuff, making sure that it works with hot-reload etc. The plugin tries to be invisible for the game, so it adds an option to register for events and register own textures but other than that game should use the plain ImGui. Delegates are merely to add an option to call debug code and should not be an issue. The texture manager probably will need an extension to support texture upload... although I would say that this is hardly critical. In my last company, I never handled that part and nobody complained... meaning that nobody felt a need to use that.

Ad 5. Yeah, it is not necessary to support multiple clients. And what you mentioned later, one connection with an interchangeable active context is perfectly fine. In fact, this is what I used to do with Remote ImGui.

Ad 5+. I don't remember it exactly, but I don't think that there were many features in Remote ImGui that I would really need. In the end, it is pretty minimalistic and the main feature is to be able to connect remotely. The one thing that I felt was useful I already mentioned, which would be a possibility to switch between inputs. Remote ImGui was designed differently, though - the game was the server and the client was connecting from the browser. This probably made it easy to switch input and control the canvas size to match it to game needs as it was already in the same place.

[sammyfreg]

First hackish test done, and we have a touch down! I mean communications :)

This only send the content of Context 0 and still use the Windows networking code. I'll spend some time this weekend to make a UE4 netImgui plugin that UnrealImgui plugin can use. I have a good amount of experience of low level Unreal graphics programming, but not so much with the higher level / plugin things. So I have to figure things out as I go.

I figure that I'll create a netImgui plugin with the needed functionalities and basic integration in UnrealImgui, and afterward, you could help with finalizing the details of how things are going to be exposed in your plugin (like connections settings and such).

[segross]

Nice. And whatever you feel makes more sense. And sure, I will be happy to help finalising it. Not sure how much or little plugin experience you have, but if anything ask and if I know I will surely share. I have some experience although it has its limits :)

You have a choice between making a separate plugin, adding a separate module or building it with the same module - whatever is more convenient. Separate plugin might be easier to isolate, if that is preferred. Now, I don't fully remember but I think that when use a separate plugin approach, you will need to specify a dependency in the ImGui.uplugin file. My problem with those files is that settings cannot be express dynamically, like based on the engine version or "If found that directory or plugin then enable that and that". For instance, I have this problem with the module type. 4.24 or 4.25 depreciated the "Developer" module type and introduced "DeveloperTool", I would like to switch between them depending on the engine version but this is not possible. So I opted for the old name, since it is still supported. But you get a warning what some people already raised.

[sammyfreg]

I have a prototype working that you can take a look at (it's the source code that goes under "ImGui\Source"). I had to make a few small changes to netImgui and it's all good.

For the moment, everything is compiled inside your plugin, until I figure out what I want to do with the netImgui code. I can make a pull request after that. The changes to your plugin (not too many) can be found when searching NETIMGUI_ENABLED. You need to run the netImgui server and add 2 client config on localhost port 8889 and 8890.

I basically copied your sample to create my test scene. Because the plugin relies on World::Tick() for refreshing, Editor ImGui is only updated when in PIE and with a low refresh rate. The game works fine though. I was wondering if ImGui refresh could rely on something else instead, to update more frequently in editor?

ImGui.zip

[segross]

Yes, I see it working :)

About the refresh rate. Do you mean the reduced refresh rate when leaving the editor to debug remotely? Because there is a setting for that in the Editor Preferences -> General - Performance: "Less CPU when in Background". We used to recommend to turn it off when using Remote ImGui. I would add it to the readme and I think it is a fair requirement for what you get.

The plugin relies on the world tick, so all game objects can send their data from their own tick functions. If it wasn't synced, I think it would not work correctly. I never made any proper examples, but I copy-pasted one of my tests AImGuiDebugOrderTest to this thread: "there's any example to start with this?" It is a bit outdated but it shows the main point. Debugging code from AImGuiDebugOrderTest::ImGuiTick() and AImGuiDebugOrderTest::ImGuiMultiContextTick() will always work but I'm pretty sure that the debugging code from AImGuiDebugOrderTest::Tick(float) would not. Another reason why I have the debug synced with the world tick is that I used to rely on that when recording behaviour of our systems depending on physics or animation. There was always this one to one parity parity between state and output.

Finally, the one thing that I would be a little cautious about or maybe reluctant, is this:

PCHUsage = PCHUsageMode.NoPCHs;

I think that there is a way around this but first the problems: It breaks the compatibility with anything older than 4.22. Yes, 4.22 is old enough but I remember some people fixing issues to have this plugin working with versions that were alder from whatever I supported. So as kind of a goal I try to keep it backward compatible as much as possible and reasonable. Another thing is that I'm not sure how that works when compiling it as the engine plugin as they recommend using PCHUsageMode.UseExplicitOrSharedPCHs to work correctly with IWUY. To avoid modifying ImGui files, I put them in the separate ThirdPart/ImGuiLibrary directory, and then to compile it as part of the module I added the ImGuiImplementation.h and .cpp. This may be not the most elegant solution but it allows adding all the necessary includes, wrappers, etc. without touching the original files. In my opinion, this also makes it easier when upgrading as it isolates ImGui source files. And as a bonus, ImGui credits and license have good visibility. And when the Unreal changes its requirements, the implementation file is the one that needs to be updated. Additionally, you can there use the NETIMGUI_ENABLED switch to completely wipe out that part if it is disabled. On the downside, you need to manually include all necessary cpp files but this is something I can live with that. With this approach, files like _NetImguiNetworkUE4.cpp I would probably add to module, and regular NetImgui headers and source files used in the same way as ImGuiLibrary.

[sammyfreg]

About the refresh rate. ... for that in the Editor Preferences -> General - Performance: "Less CPU".

That was it, good to know. We could also turn that off automatically when connecting, or offer a toggle option directly in the netImgui menu options (alongside the Context switch).

The plugin relies on the world tick

At my previous company, we had integreated ImGui as part of the update loop (directly in engine code) and we were updating it per View, not per World. In any case, there seem to be 1 problem to fix, when I switch active Context from 'Editor' to 'PIE', netImgui stops refreshing (when not running PIE, makes sens). Will have to force a NewFrame/EndFrame if world is disabled, when drawing the MainMenu bar for context switch.

Finally, the one thing that I would be a little cautious about or maybe reluctant, is this: PCHUsage = PCHUsageMode.NoPCHs;

Good to know, I did that change quickly so things work, without worrying of side effect at this point. I'll probably move my plugin to work like ImGui. When I first started, I didn't understand what was going on with the Imgui module :) It was a separate compile entry, but there was no source files! When trying the same thing with netImgui, shockingly, it was missing symbols. I don't like having to manually include each file, but I like even less having to edit the sources files, so I might skip plugin approach and duplicate your Dear ImGui source code approach.

Note: Pull request created.

[segross]

At my previous company, we had integreated ImGui as part of the update loop (directly in engine code)

I see. Don't think it is possible from plugin though.

and we were updating it per View, not per World.

Yeah, that can work. Did it work with the debugging code from Tick functions? It was a goal of mine to allow that and I would think that ImGui and game frames need to be in sync for that to work.

there seem to be 1 problem to fix

I'm not sure if I fully understood the problem here (I'll try to look at this when looking at the pull request) but yes, updating frame for otherwise inactive contexts should not be a problem. The manager needs some cleanup anyway, so it can be handled. I can also look at this. On a side note, I'm not sure whether the editor context is really needed. I've added that because without having some default instance that could be used in absence of other contexts, there was always somebody trying to call ImGui API in some weird places or states and than coming and asking why is he getting a crash ;) So I just added that as a backup to guarantee that there is always some context to draw to. I don't fully remember whether this is needed for something else as it did evolve over time a bit and it was a while ago.

We could also turn that off automatically when connecting

Not sure how to do it, but yes that could be an option.

Thanks for that pull request. If you don't mind I'll take a look and take it for a test run later today or tomorrow if today proves impossible. But of course it will be merged. Is the default "squash and merge" fine? Any other questions will continue in that request thread.

[sammyfreg]

Yeah, that can work. Did it work with the debugging code from Tick functions? It was a goal of mine to allow that and I would think that ImGui and game frames need to be in sync for that to work.

The ImGui drawing were only allowed from callback functions, so no Tick :) I should have been possible though, by always having a Context open for draw instead of our Begin -> Callback -> End setup.

I'm not sure if I fully understood the problem here

I already fixed that problem before creating the pull request. The things left are in the description of the pull.

Is the default "squash and merge" fine? Any other questions will continue in that request thread.

Good and good. I am not well verse in Git, beyond using it mostly like I use Perforce, so feel free to take the approach you want for the merge. It's only 2 submits (main submit + fix) so, I think it's better to squash like you mentionned.

[segross]

Current integration thanks to @sammyfreg is in the _netimgui branch. It will be merged into master but anybody interested can give it already a go.

[Frigerius]

Hey :D Do you plan with net_imgui to also support dedicated servers?

[sammyfreg]

What do you mean by 'dedicated server' ? NetImgui was designed to have Dear ImGui run in your code (in this case, under Unreal) with the Server application running on your PC, controling and displaying the UI locally.

Another version of remote access is this project : RemoteImgui. It allows accessing the Dear ImGui UI on the web, but is unrelated to NetImgui and not supported with this integration.

[Frigerius]

I mean that right now you need slate to have ImGui running. What I am talking about is having ImGui working for Unreals Dedicated servers e.g. for a commandline exe (no rendering) for multiplayer games. this way you could also show data on the server without the need of using the Gameplaydebugger to send this data to the clients.

[sammyfreg]

Ah, I understand.

After integrating NetImgui to this excellent UnrealImgui plugin, I created a new standalone one : UnrealNetImgui. It's a barebone Dear ImGui implementation without local rendering. I will be updating these 2 plugin in the upcoming weeks, after a big Server refactor, but it should still be working.

[Frigerius]

Nice, I'll have a look, thx!

[sammyfreg]

I haven't tried it in a 'Dedicated Server', let me know if there's some issue. It might not correctly detect the end of a frame.

[Frigerius]

Alright, will do that, but most likely won't have time for that today :( maybe tomorrow

[Frigerius]

XD I took a code snipped from your repo, basically FNetImguiModule::StartupModule where you build the fonts etc. add this there where this plugin loads the textures and by disabling code that requires Slate, if we are on a dedicated server and changing GFrameNumber to GFrameCounter it works xD edit Ok works fine so far, now I need to add more ports^^

[sammyfreg]

Nice!

If you have some questions / problems, you can open a ticket in the UnrealNetImgui Github project.

[sammyfreg]

I recently updated NetImgui / UnrealNetImgui and was planning to do the same to this plugin next. I was wondering if you'd like me to also bring in Dear ImGui 1.80 in the netimgui branch?

[Eragon-Brisingr]

@sammyfreg Do you have any plans to upgrade this branch to NetImgui 1.5? I tried to compile the NetImgui 1.5 Server but I can’t connect to the editor, I need change source code to support 32-bit vertex indices.

[sammyfreg]

I'll try to take a look this weekend. Shouldn't be too hard.

On Wed., Jul. 14, 2021, 14:13 Eragon-Brisingr @.***> wrote:

@sammyfreg https://github.com/sammyfreg Do you have any plans to upgrade this branch to NetImgui 1.5? I tried to compile the NetImgui 1.5 Server but I can’t connect to the editor, I need change source code to support 32-bit vertex indices.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/segross/UnrealImGui/issues/28#issuecomment-879596591, or unsubscribe https://github.com/notifications/unsubscribe-auth/AESBPY6UPGMTUF7MWTCNJ2LTXUMFZANCNFSM4MICRHDQ .

[Eragon-Brisingr]

I'll try to take a look this weekend. Shouldn't be too hard. … On Wed., Jul. 14, 2021, 14:13 Eragon-Brisingr @.***> wrote: @sammyfreg https://github.com/sammyfreg Do you have any plans to upgrade this branch to NetImgui 1.5? I tried to compile the NetImgui 1.5 Server but I can’t connect to the editor, I need change source code to support 32-bit vertex indices. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#28 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AESBPY6UPGMTUF7MWTCNJ2LTXUMFZANCNFSM4MICRHDQ .

That's great, Thank you for this amazing NetImgui!

[sammyfreg]

Pull request #53 as been created with the update

[Eragon-Brisingr]

Pull request #53 as been created with the update

Nice! Thank you~

[HAL-XP]

Hello, thanks so much for these plugins! FYI Anyone who'd like to connect multiple clients, just modify this to add a custom port by commandline: else if (FApp::IsGame()) { uint32 CustomPort; if (FParse::Value(FCommandLine::Get(), TEXT("imguiport"), CustomPort)) { return CustomPort; } return NETIMGUI_LISTENPORT_GAME; }

And start your game with -imguiport 8895 or something, then configure the server for a second game on that port :)

[willievaldez]

Quick question, hopefully devs are still active. I have the net_imgui branch integrated into our project, and it works perfectly with the out of the box netImguiServer.exe, but when I try compiling and making one from the https://github.com/sammyfreg/netImgui repo itself, I'm not getting connections. I copied the imgui.ini and netImgui.cfg files into the same directory as the newly generated NetImguiServer.exe but still had no luck. Any ideas on what could be going on? How did you generate the original netImguiServer.exe?

EDIT: To add to this, I'm also noticing that when I switch the NetImguiServer to listen to my actual local IP (192.168...) instead of localhost, I'm failing to get a successful connection

[sammyfreg]

This should be easy to fix. The newer server is not compatible with he older client code. You will need to replace the client code in your branch.

• Copy the client code in the package you downloaded, to UnrealImgui\Source\ThirdParty\NetImguiLibrary\Client.
• Make a small change to UnrealImgui\Source\ImGui\Private\ThirdPartyBuildNetImgui.cpp (adding/removing a space is enough) so that the new code is detected properly and recompiled.
• There might be a few API function differences that you need to fix, but nothing major

Note: The ImUnrealCommand code might also be a bit older (not needed for NetImgui update).

• This is the code to have a GUI for the unreal console commands
• You can get the latest version from https://github.com/sammyfreg/UnrealNetImgui.
• The 2 files that need replacing are in UnrealImgui\Source\ThirdParty\ImUnrealConsole\Private

[coderoid]

@sammyfreg @segross I know you'll have regular day jobs and regular things to do but as a community member I value what you have done here with UnrealImgui and NetImgui. Now I come bearing some ideas, which kind of goes against what ya'll have done thus far. I my previous company IMGUI used to render in the web and I would connect to a port and the web would render the game's UI. This was far easier than having a standalone app and allowed me to look at debug data regardless of platform. I just need the IP and a browser. I already made changes to @segross 's UnrealImgui app to allow this to work and it works as a proof of concept but I have some remaining issues to figure out. https://github.com/Eragon-Brisingr/ImGui_WS Does exactly this and I have ported that code using unreal websockets and layered it into Sebastian's system.