Missing texture replace

[lemantisee]

Hi! In our project client app provides the video stream witch I would like to embed in ui on server side. Previously we implement it very roughly:

1. On client app draws texture with ImGui::Image and send texture and texture id via third party tool (NDI)
2. On server side receive texture and texture id from third party tool
3. Provides to netimgui on server side texture data by texture id which was used in ImGui::Image on client side

So, I interested is netimgui project interesting in this feature? I can make PR with describes the idea. Seems it very close to https://github.com/sammyfreg/netImgui/issues/10 but difference in that I'm want to provide texture in each frame on server side.

[sammyfreg]

The idea of #10 was to have the changes happen only on the client side. The netimgui client should let the library user know that a new texture is being used, so they can decide to send it to the netimgui server.

If I understand correctly, you want to update a texture every frame on the client, send it to your Dedicated Server and have it forwarded to the NetImgui Server?

So the NetImgui server would need to handle opening a new data pipe to receive texture from your 3rd party server, and somehow know it belongs to a specific client?

Any reason you don't try to update the texture directly between the NetImgui client and server?

Is it because this is a video that is still compressed on the client side, then decompressed on your dedicated server?

On Thu., May 18, 2023, 00:15 Kostiantyn Chernenok @.***> wrote:

Hi! In our project client app provides the video stream witch I would like to embed in ui on server side. Previously we implement it very roughly:

1. On client app draws texture with ImGui::Image and send texture and texture id via third party tool (NDI)
2. On server side receive texture and texture id from third party tool
3. Provides to netimgui on server side texture data by texture id which was used in ImGui::Image on client side

So, I interested is netimgui project interesting in this feature? I can make PR with describes the idea. Seems it very close to #10 https://github.com/sammyfreg/netImgui/issues/10 but difference in that I'm want to provide texture in each frame on server side.

— Reply to this email directly, view it on GitHub https://github.com/sammyfreg/netImgui/issues/44, or unsubscribe https://github.com/notifications/unsubscribe-auth/AESBPY7QQ6EKWO7XI574W7LXGTTQLANCNFSM6AAAAAAYFH4RVQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[lemantisee]

Oh, sorry for bad describing our pipeline:) Our case is more common. We have two applications - server(local or dedicated) and client. On server we use imgui (and NetImgui) to draw interface and shows it on client app with NetImgui. Server continuously renders some image which we want to show in client app. For that we use third party tool to stream this image to client app. We don't want use transferring texture via NetImgui (NetImgui::SendDataTexture) because video streaming requires complicated compression and obliviously not a task for NetImgui.

[sammyfreg]

Ok, if I understood correctly (because there a lot of server and client mixed in :) )

You have 'Server A' application with a 'Netimgui Client B' connection. You have 'Client A' PC that's running 'NetImgui Server B' application.

Your Server A is running some logic, then sending the imgui results to the NetImgui Server B.

Given that it's a video stream, I agree that sending it uncompressed would be a problem. An easier solution than what you propose, could be to send the compressed image frame as a new texture format, and then have the NetImgui server decompres it, maybe by using a known library, or what you are already using.

[lemantisee]

You have 'Server A' application with a 'Netimgui Client B' connection. You have 'Client A' PC that's running 'NetImgui Server B' application.

Yes, totally correct.

Given that it's a video stream, I agree that sending it uncompressed would be a problem. An easier solution than what you propose, could be to send the compressed image frame as a new texture format, and then have the NetImgui server decompres it, maybe by using a known library, or what you are already using.

Actually we made something similar. We send video stream from 'Server A'. After receiving it on 'Client A' we upload it to GPU and obtain 'Video texture id'. After that when NetImguiServer (on 'NetImgui Server B') replaces imgui texture ids by own texture ids (NetImguiServer_RemoteClient.cpp:250) we check if requested texture id is for our video stream and replace it with 'Video texture id'. It's implemented in way which breaks smooth update of NetImgui to next version because we changed NetImguiServer_RemoteClient.cpp file. Maybe it's possible to add some callback to NetImguiServer which will requests missed textures id (which not found in NetImguiServer::Client::mvTextures vector)?

[sammyfreg]

The Callback idea was limited to the NetImgui Client code. When the NetImgui Client detects an unkown texture, it would use a user specified function callback to request some actions on its application side. Having the callback between the NetImgui Server and NetImgui Client would be more complex, and I feel that there would be not point to detect this at the NetImgui Server level.

I also do not want to add too much complexity to the NetImgui codebase, for a very specific usercase, that I would then have to keep maintaining.

However, I might have an idea of something that could work for you

• Add a new 'Custom' texture format (right now we have A8 and RGBA8).
• Every frame, the NetImgui Client send the 'Custom' texture to the NetImgu Server as a 'video compressed frame'
• The NetImgui Server receives the 'Custom' Texture update
• When detecting the Custom format, it calls a user defined function to process it and creates the normal GPU texture
  • The custom Processing function could be in a separate file, or a callback
  • This require the user to compile their own NetImgui server (like you are doing)

This is a rough idea, let me know what you think and please let me know if you have better suggestion that involves few changes and maintenance to the codebase to maintain it.

On Thu, May 18, 2023 at 1:52 AM Kostiantyn Chernenok < @.***> wrote:

You have 'Server A' application with a 'Netimgui Client B' connection. You have 'Client A' PC that's running 'NetImgui Server B' application.

Yes, totally correct.

Given that it's a video stream, I agree that sending it uncompressed would be a problem. An easier solution than what you propose, could be to send the compressed image frame as a new texture format, and then have the NetImgui server decompres it, maybe by using a known library, or what you are already using.

Actually we made something similar. We send video stream from 'Server A'. After receiving it on 'Client A' we upload it to GPU and obtain 'Video texture id'. After that when NetImguiServer (on 'NetImgui Server B') replaces imgui textures ids by own texture ids (NetImguiServer_RemoteClient.cpp:250) we check if requested texture id is for our video stream and replace it with 'Video texture id'. It's implemented in way which breaks smooth update of NetImgui to next version because we changed NetImguiServer_RemoteClient.cpp file. Maybe it's possible to add some callback to NetImguiServer which will requesting missed textures id (which not found in NetImguiServer::Client::mvTextures vector)?

— Reply to this email directly, view it on GitHub https://github.com/sammyfreg/netImgui/issues/44#issuecomment-1551750485, or unsubscribe https://github.com/notifications/unsubscribe-auth/AESBPY3DVJTWCCCHOD5UEWDXGT65VANCNFSM6AAAAAAYFH4RVQ . You are receiving this because you commented.Message ID: @.***>

[lemantisee]

This solution is good. It actually much better than my callback idea for our future plans about video streaming. But can we not create texture every time in NetImgui Server when it receives 'Custom' Texture update and just update existed? On first call for custom texture user will creates it, for next calls just updates existed texture.

[sammyfreg]

The idea would be that the NetImgui Client keep sending video frame using the same textureid. Then, the NetImgui Server would recreate a new texture GPU ressource, but at the same TextureID. This would be simple.

It might be possible to setup the GPU texture to be updated rather than recreated, but this is something that would need to happen in the custom texture creation function (so handled by you).

[lemantisee]

Sounds good. It's possible also to pass some user data to custom texture creation function? Or it will be to specific? User data can helps to determinate is creating or updating needed.

[sammyfreg]

I was thinking of passing the current GPU texture object of found and then the texture command data.

On Sat., May 20, 2023, 01:42 Kostiantyn Chernenok @.***> wrote:

Sounds good. It's possible also to pass some user data to custom texture creation function? Or it will be to specific? User data can helps to determinate is creating or updating needed.

— Reply to this email directly, view it on GitHub https://github.com/sammyfreg/netImgui/issues/44#issuecomment-1554866124, or unsubscribe https://github.com/notifications/unsubscribe-auth/AESBPY6CYYIQ7HIDWO3PNSDXG6PHTANCNFSM6AAAAAAYFH4RVQ . You are receiving this because you commented.Message ID: @.***>

[lemantisee]

Maybe it's too specific case, but some times user can wants to change processing texture command data depends on some external parameters. In this case some user data in custom texture callback may helps.

[sammyfreg]

I was thinking that this would be up to the user to create their own data format inside the texture data.

I'm doing a custom texture branch, and once I have something, you can try it out and let me know what works and doesn't

[sammyfreg]

Submit bc7d34e452d71cc0dbcff6a748cb3a6713fb5b7c in the Task_CustomTexture branch give an idea of how I was imagining the feature to work.

The ProcessTexture_Custom function will be moved to a standalone file, so user can freely modify it, and name might change.

Let me know if it looks like something that would work for you.

[lemantisee]

Thank you! It's totally works for me. But want to describe how exactly I use this feature now. I still use third party lib to transfer texture data. Can't send texture data via NetImgui because texture compressing is not yet done in our code. From NetImgui Client I send some small dummy texture data only for calling ProcessTexture_Custom callback. In callback I just set valid texture id to serverTexture which I created on NetImgui Server side after receiving texture data. It's works perfectly because there are no checks (size, bits per pixel and etc) for data validation in NetImgui texture synchronization process. So if it possible to keep it in same way it will be perfect.

My ProcessTexture_Custom implementation looks like

bool ProcessTexture_Custom(const NetImgui::Internal::CmdTexture& cmdTextureUpdate, ServerTexture& serverTexture)
{
    auto eTexFmt = static_cast<NetImgui::eTexFormat>(cmdTextureUpdate.mFormat);
    if (eTexFmt != NetImgui::eTexFormat::kTexFmtCustom) {
        return false;
    }

    if (cmdTextureUpdate.mTextureId != remoteTextureIdReplace || serverTexture.mImguiId != remoteTextureIdReplace) {
        return false;
    }

    serverTexture.mpHAL_Texture = reinterpret_cast<void *>(static_cast<uint64_t>(localTextureIdReplace));
    serverTexture.mSize[0] = 1024;
    serverTexture.mSize[1] = 1024;
    return false;
}

where remoteTextureIdReplace is a texture id which I use in NetImgui::SendDataTexture to send dummy texture localTextureIdReplace is a texture id with valid texture data which I created on NetImgui Client side

In some future I will use hardware h264 encoding for video and send texture data via NetImgui as it designed now.

[sammyfreg]

I have updated the Sample in the CustomTexture branch. Let me know if this solves your issues. I will then clean it up and integrate it to the DevBranch.

if (cmdTextureUpdate.mTextureId != remoteTextureIdReplace || serverTexture.mImguiId != remoteTextureIdReplace) {
        return false;
    }

So I guess you send a texture update everyframe, but only update the texture if you detect you have received a new one from the video server that you are using? I guess there's not much choice, if you can't be sure on the netimgui client, when the video server has received a new texture and is ready to display it on the NetImguiServer.

So for now, your code won't change much, but you have the option of streaming directly from the Texture command now, and your code changes will be isolated to a separate file (once I move the function).

[lemantisee]

Everything good. Thank you! Please correct me if I understand it wrong. Will ProcessTexture_Custom function goes to HAL section on feature release?

[sammyfreg]

I finalized the work on custom texture support and imported it in the 'dev' branch. (see beb5dd4).

The custom texture handling is done inside NetImguiServer_App_Custom.cpp and there's a sample demonstrating the usage inside the Texture Sample, and the Server Application. (look for the TEXTURE_CUSTOM_SAMPLE define)

Once you have confirmed that everything works as expected, I will close this issue.

[sammyfreg]

Note: I never worked with textures updated every frame (like a video frame), but there might be a better way of handling this than Destroy/Create texture gpu resource each time (like the sample is doing). It might be that creating the texture gpu resource as 'updatable' and then updating it instead of recreating it, could be faster.

This is something that you can explore, but would need to implement your own HAL_CreateTexture to handle it.

Just a thought.

[lemantisee]

Yeah. In OpenGL there is glTexSubImage2D for updating whole texture or only a part of it. Thank you for this feature!