Integrate UIs into 3d world

[a-teammate]

UIs are 2D, the world is 3D. We have 3D editing. We have a great 2D UI framework (CEF).

We don't want to invent a 3D UI since its

• hard to control
• hard to look at

UIs need to be easy.

We can combine both by pinning UIs onto the faces of cubes.
Those UIs should exhibit the settings of the cube its pinned to. *E.g. you modify the texture of the selected cube. (instead of operating globally a button could exist like 'repeat on all similar'. Possibly with a filters like "on all cubes with same texture/same shader/same shaderparameter/same xy).**

Usecases:

• in world shader editor
• in world texture editor

Todo

Control

• [ ] Input handler: if cube is selected, open inworld menu (or is it the only menu?)
  • e.g. 2 texture menus: one if you selected a cube before, one otherwise

Rendering

• [ ] Camera ("Fly onto the cube, until you see the whole UI as usual")
  • [ ] zoom onto cube face
    • until its using the whole screen
  • [ ] shift the view angle
    • needs to be normal to the cube face
• [ ] Render UI "at cube face"
  • only for you, only after you pressed the button (as usual)

Possible follow up issues:

• share input (mouse movement, keystrokes) among players spectating you when you are looking at these UIs.

Possible integration into Visual Scripting System

• you can exhibit specific inputs/outputs in the UI
  • will be rendered in 3D out of that cube
• same logic ents as in 3D could be available in 2D
  • (probably against the concept of 3D scripting) and hence a not-Todo (or is it?)

[IAmNotHanni]

Great! This feature is also essential for the implementation of our visual scripting system! What we need are billboard surfaces with a HTML5 texture that has been rendered by CEF. In this picture you can see one of my experiments from ui-prototype\hanni\entity-editor.

[MartinMuzatko]

I am not too sure if CEF is capable of rendering websites on top of 2D faces of a cube. We would need to calculate where on which surface a CEF layer is pinned to - is that possible @a-teammate ?

I imagine other use-cases as well. UIs always used to be private - per client. Only the user having the menu open can look at it.

If you could have a public shared UI like an ingame texture browser that you can look at together with others, you could also collaboratively select textures, edit textures.

This would make collaboration a lot easier and fun but possibly also a bit more chaotic.

For maps that are finished, I also imagine a few use-cases. You could also integrate a publicly visible scoreboard into an arena.

Generally, binding websites to cubes sounds very fancy and opens opportunity to many things, but I imagine that implementation (especially with the shared input) and performance might be very tough.

Public UIs bound to textures could also still spawn private UIs if you click on them or select them or interact them in a way, depending on gamemode. For this, it is important for the UI to know in what mode the focused player is, in order to interact with.

[a-teammate]

CEF gives back a texture (this is btw also the reason for issue #387: the different textures/layers of different browser windows can be drawn at different positions as usual OpenGL textures)

Pinning it on a cube is generally just a matter of drawing that texture at the right position, nothing special. A publicly visible scoreboard as part of an arena (a HUD as an element of the map) would be straight forward then: Just render another HUD layer at a different position.

Sharing the UI is in so far a different kind of story, that probably the mouse input must be shared among players (alternatively the state changes of the website must be shared or the texture CEF gives back). So a limit to only those people ultimately involved is probably recommended for that (network performance wise).

For this, it is important for the UI to know in what mode the focused player is, in order to interact with.

True, the player would go into a 'viewing UI xy'-state. Then the camera acts accordingly and other players also viewing 'UI xy' would be notified to share their input.

Generally the number of concurrently visible layers should be limited in case we find out that it impacts performance.