Add a GUI solution to LWJGL

[Spasi]

This is meant to be a meta-issue in which discussion can take place about potential solutions to GUI layout & rendering. Currently LWJGL only supports text rendering (via stb) and may soon support antialiased 2D graphics (via nanovg).

Topics to discuss:

• Rendering GUI components on top of 3D content (e.g. game menus, HUDs, etc).
• Rendering GUI components inside 3D content (e.g. monitor/panel in 3D space displaying 2D content).
• Embedding 3D content in a standard application/tool. The 3D content should be lightweight, i.e. standard menus should be renderable on top of it.

The ideal solution should be hardware-accelerated, i.e. there shouldn't be any framebuffer copies from the CPU to the GPU or vice versa.

Approaches that have been explored already:

• JavaFX integration via PixelWriter & PixelReader. See the LWJGL-FX project (uses LWJGL 2 but the same issues apply to LWJGL 3). Not hardware-accelerated.
• Adding bindings to CEF and using the offscreen rendering API. Not hardware-accelerated.

[SpinyOwl]

Hi all, currently I working on GUI oop solution with api little bit similar to swing... Here is my repository: legui

Here is result:

Radio button and text input	Widgets	TextArea
Button, togglebutton, checkbox, selectbox	Sliders	ScrollPanel

[Barteks2x]

Someone actually made JavaFX work with lwjgl3, but it seems like it doesn't really use opengl core profile and it uses opengl 3.2 core profile: https://bitbucket.org/cuchaz/jfxgl

As it says in readme, it's a giant hack but it works and has good performance.

[SpinyOwl]

@Barteks2x it looks like it's hard to make it work with Oracle java as standalone library, am I right?

[Barteks2x]

@ShchAlexander I'm not the author so beyond what it says in the readme and what I see I can't tell you much. Based on my testing, it works as standalone library, but it uses implementation-specific hacks, including modifying the bytecode using asm library.

Obviously when using JFXGL you can't use normal JavaFX without lwjgl at the same time (and why would you?)

[cuchaz]

Hi There. I'm the JFXGL author. It's still very much a work in progress, but it's getting better all the time. I've been talking to Spasi about it too, so it's already on the radar.

If you have any questions about JFXGL, ask away. Although maybe don't spam this issue thread?

[j-ibarra]

Doing some research I got with this post!

https://engineering.riotgames.com/news/under-hood-league-client%E2%80%99s-hextech-ui

Here mention the use of javascript to create the client of this game, it would be interesting to be able to create GUI with HTML

OpenGL GUI using Chromium Embedded Framework (CEF)

[jjYBdx4IL]

SWT uses native OS widgets, why would one want to use that?

Did anyone evaluate whether it is possible to extend Graphics2D class/override repaint() methods and thereby redirect Java Swing UI painting to OpenGL/NanoVG?

Besides that, one could set JFrames as transparent and place them on top of the OpenGL window... :)

However, the best solution would be some generic Java UI lib that can be plugged into NanoVG, ie. a rewrite of Nuklear. CSSBOX provides html/css rendering, though I'm not sure how efficient animations would be. And interactions? In general, I'm not sure how efficient GUI animations would be with pre-written GUI libs. It would be an interesting task to think about the required structure of such a library to efficiently support all sorts of GUI animations.

BTW: because Vulkan support is now quite widespread, is NanoVG compatible with it?

[elect86]

Hi guys,

I'd like to propose also another possibility..

I'm doing a full jvm port of imgui here (nuklear is based on its concept), it's almost complete, text input/handling needs still to be addressed but all the rest is done

plus, it's also libgdx compatible :)

See the Screenshots Thread for some user creations. Also see the Mega screenshots for an idea of the available features.

[dustContributor]

I was reading some Java articles and it popped up that Gluon published release/support plans for OpenJFX. For those who don't know, JavaFX is being removed from core JDK, and it will be released separately. So to use it you'd need it to import it like any other dependency (maven, raw jars, etc). And now it will be called OpenJFX.

https://gluonhq.com/javafx-11-release-and-support-plans/

They have a GitHub mirror of the OpenJFX repository and they say they pay attention to the issues there. https://github.com/javafxports/openjdk-jfx

Moreover, they say they have their bug tracking on the JDK thingy (although the site works like ass for me and I cant find it) https://bugs.openjdk.java.net/secure/Dashboard.jspa

Anyway, I'm wondering if that given the changes JFX is going through, if it would be possible to push for direct access to the GL internals from it? As a means to integration with other GL rendering.

All the stack is open source now so you got in that repo their "Prism" rendering engine and all, plus the HLSL/GLSL shader transpiler (Decora I think?) and some other nifty things.

In any case, they went a different route for it so instead of making bindings and "lower" the layer of native interop, they instead "raised" it by making the rendering part in C/C++ (doing all their multi platform context management on their own and all) and exposing higher level operations to Java. That doesn't sounds like it'd interact well with how LWJGL works but I wouldn't know.

It gets complicated because OpenJFX uses three backends, OpenGL ES 2, Direct3D and software. So what happens if you want to use OpenGL Core? Or Vulkan?

It's pretty complex, and I personally believe that The One True Way™ to do an UI lib for games is doing it like Nuklear does, handing out buffers of vertices/uvs/indices at rendering stage and have the user be responsible of displaying that to the screen, but I thought someone might be interested.

[Spasi]

@dustContributor Yes, OpenJFX being decoupled from the JDK opens up opportunities, you can read some details in this JGO post.

(No work has been done yet, I grossly overestimated the free time I'd have this summer)

[dustContributor]

I see you know already how to tackle it.

I inspected a bit OpenJFX code and it seems to have a rather direct rendering, in the sense that it has fairly hard codes operations in a bunch of classes that do specific things.

I'm guessing that achieving Nuklear-style of integration (ie, making it spit out vertices and texture bindings, API agnostic) would be pretty hard.

[SpinyOwl]

Hello everyone again. I have several changes since last comment here. Here is image of my gui library's current state:

If anyone want to contribute, request any changes - you are welcome. GitHub repo - https://github.com/SpinyOwl/legui/

[dustContributor]

If anyone want to contribute, request any changes - you are welcome.

I'll repeat what I said above (except longer!). I dont think having your own rendering backends is a good idea. You don't know how I, library user, do my rendering, you don't know what API I run, you don't know what version I run, and you really shouldn't.

Things that did their own drawing like Scaleform on big AAA games were phased out, because you have a complete subsystem trying its best to group rendering commands, group state and shader changes, and drawing all of that efficiently, and then you stuff your code with some other renderer that has some other API with some other concepts that just doesn't fits on how your engine does its drawing.

You have libraries like NanoVG, Nuklear and (now defunct) libRocket that are really really easy to integrate because they have a clear distinction between what is UI handling, what is rendering and what is input handling.

Without the library setup code, this is my rendering loop for Nuklear:

try ( MemoryStack mem = MemoryStack.stackPush() ) {
    // setup buffers to load vertices and elements
    final NkBuffer vbuf = NkBuffer.mallocStack( mem );
    final NkBuffer ebuf = NkBuffer.mallocStack( mem );
    nk_buffer_init_fixed( vbuf, tmpVertices/* , max_vertex_buffer */ );
    nk_buffer_init_fixed( ebuf, tmpIndices/* , max_element_buffer */ );
        // converts buffers to the layout specified in the config
    nk_convert( context, commands, vbuf, ebuf, bufferLayoutConfig );
}
// Upload data to the GPU
glAttribBuffer.bind().updateUnsafe( tmpVertices );
glIndicesBuffer.bind().updateUnsafe( tmpIndices );
// Process task commands.
int offset = 0;
// Compose render tasks out of nuklear's draw commands.
for ( NkDrawCommand cmd = nk__draw_begin( context, commands ); cmd != null; cmd = nk__draw_next( cmd, commands,
        context ) ) {
    final int elmCount = cmd.elem_count();
    if ( elmCount < 1 ) {
        continue;
    }
    // Submit task to the queue.
    // Matrices computed in shader.
    renderer.addToQueue( taskBuilder.reset()
            .type( RenderTask.Type.NUKLEAR )
            .meshId( glVao.id )
            .material( materialsById.get( cmd.texture().id() ) )
            .vertexCount( elmCount )
            .indicesOffset( offset )
            .build() );
    offset += elmCount * 2;
}

I just ask for Nuklear to fill vertex/index buffers, upload them to the GPU, then compose render tasks that use them and that's it.

I know that will best fit my rendering scheme because it is my rendering scheme. It will use the API I use, it will use the version I use, it will track state changes like I do, it will batch UBO updates like I do, and run without switching shaders unnecessarily like I do, and if I wanted I could make it use my instanced rendering by grouping by material first.

I dropped things like NiftyGUI because literally the first time I use them it turns out that there was some GL context interaction that they didn't account for so I had to spend time debugging what state my renderer was leaving GL with, what state the UI lib really needed (that may or may not be what the documentation says) and what state the GL context ended being in that made things not work correctly.

The GPU is a big fucking thing, wouldn't it be nice if you (lib author) didn't have to account for these things? Wouldn't it be a concern better suited for myself the library user since I know what API I target and how I want to use it?

I'd say: Write an UI lib, or a renderer, but not both in the same thing. They're separate concerns, and big enough to warrant their own separate libraries.

The same argument can be made for input handling but that'd make this too long. I have an input system that translates GLFW's stuff for my own scheme, and those events get pushed to Nuklear, Nuklear doesn't cares how do I map my keys, devices or anything, it just cares for being fed input events in a format it understands.

At least that's my POV after trying out integrating JavaFX, Swing, TWL and NiftyGUI multiple times over time. Nuklear, being the pain in the ass of a library it is to use, was way, way easier to integrate. And since the API is so clean cut on that regard, it was easier to debug. I know when it renders because I render it. I know when it processes input because I tell it to.

That said, I know UI is hard, that's why I'm always in the lookout for a lib that does it for me. I understand making such a simple integration point isn't easy, and may require rethinking a lot of abstractions. For example, it's really hard to move TWL as-is to such scheme, or even to using a core GL context (nevermind Vulkan) with it because not only believes it should draw itself, it believes it should draw itself like they did with OpenGL 1.1 in 1998, which for any kind of recent project is a problem. The right abstractions just aren't there.

Moreover, I will argue that moving to such scheme not only opens up the integration possibilities, it has no drawbacks. Nothing prevents you from implementing a GL 1.1 renderer for Nuklear, or a Vulkan one, or a software renderer, nothing prevents you from supplying separately your own backends to draw it if you really really want to code an UI renderer. Hell, it wouldn't surprise me if data-driving it that way made the whole thing faster via handling flat buffers of draw commands and vertices. It just makes the whole ordeal way easier for everyone involved.

I'm not writing this for you specifically anyway, so dont take it as a big ass critique of your code, I've only skimmed through your library and it probably does a lot of things right that aren't even on my mind right now. I'm just writing it for anyone who is interested in doing/using an UI library in Java for videogame (or any real-time graphics) purposes.

[SpinyOwl]

I understand what you mean, @dustContributor, and of course big projects often use their own decisions that fit better to the project and work faster.

I'm not saying that my decision is good, but I tried to make it as much extensible as it could be. Renderers could be implemented using OpenGL, Vulkan or and any other technology.

Any way every solution is good for it's purposes. If you need best performance - you implement something in your own way, if you need fast development without pain - you can choose existing solution. 👌

[kappaOne]

Interesting new feature for JavaFX https://twitter.com/johanvos/status/1128627206295629824

[cuchaz]

Sounds like it still uses texture copies though. =( See: https://twitter.com/tomsontom/status/1128698978520977411 DriftFX still looks much better. Just wish it were more portable.

[cuchaz]

Looks like some momentum is starting to build for integrating DriftFX into LWJGL.

See: https://github.com/eclipse-efx/efxclipse-drift/issues/9

[hYdos]

would there be a way to surround a 3d scene with a gui i made using a flat quad?

[zugaldia]

Now that https://github.com/eclipse-efx/efxclipse-drift/issues/9 has been closed, what's next for the integration branch (https://github.com/LWJGL/lwjgl3/tree/driftfx)? Is it available for testing as a snapshot build?

[Spasi]

The latest snapshot (LWJGL 3.2.4 build 4) includes DriftFX. It's a build of the wip_performance branch.

Since the implementation is unstable, performance is suboptimal and only Java 8 is supported, lwjgl-driftfx is considered experimental. Like lwjgl-bullet, it will be available in snapshots only until the situation improves.

[gudenau]

Is there a way I can skip building this or get it to build with java11-openjfx installed? It can't seem to locate classes related to openjfx.

[Spasi]

Hey @gudenau,

As of 92c88d461df60e1017322ea3f6372c00ded3892a there is support for JavaFX 11. The build should download the necessary OpenJFX jar files automatically via Maven. If it doesn't work for you, please open a new issue and paste the error you're getting.

You can also skip the DriftFX bindings by setting binding.driftx to false in config/build-bindings.xml.

[Displee]

I've been working on a 3D RuneScape map editor using JFXGL and threekt:

RuneScape map editor

The performance is not bad at all. It does have some UI issues but there are workarounds for that :)

However, I have yet to figure out how I'm going to render popups and child windows and such.

[cuchaz]

Holy cow, that's awesome!

[Spasi]

Update on the DriftFX integration:

I'm about to remove the DriftFX bindings from LWJGL, because vanilla LWJGL + latest official DriftFX work great together! Details here: https://github.com/eclipse-efx/efxclipse-drift/issues/17#issuecomment-951313520

[lqyaos]

How do I run javafx and lwjgl on mac

Now I just need to set -XstartOnFirstThread to run javafxApplication.launch(HelloApplication.class); The thread seems to be blocked and has no response

[DeafMan1983]

The performance is not bad at all. It does have some UI issues but there are workarounds for that :)

However, I have yet to figure out how I'm going to render popups and child windows and such.

If you want create native sub or child windows ( dialogs ) if I understand correctly.

You need understand native bindings like glfw3 with X11, WinApi or Cocoa.

Example SubWindow.java

Example long subwindow = Subwindow.Create() Create loop with native Api X11, WinApi and Cocoa Convert to custom glfw3 window as subwindow via glfw3 native Create loop with glfw3 input with native events Example trying to work for show, hide or resize ... Etc....

If you want drop & drag native window to window - you need write engine with native Api.

I recommend you JNA to help with glfw3 windowing manipulation. If you try first C++/C how do you understand to write glfw3 with native api then you will port to Java.

I hope you understand me.

[Displee]

Thank you for your comment. I am now using DriftFX. That is way easier and I can run mulitple GL renderers in one or multiple windows. I have implemented DriftFX into threekt with multiple windows here: https://github.com/markaren/three.kt/blob/master/examples/src/main/kotlin/info/laht/threekt/examples/javafx/HelloDriftFX.kt

[tlf30]

I would like to point out for anyone reading this thread, https://github.com/SpaiR/imgui-java is a java binding to Dear ImGui, which is a great gui library. I use imgui-java in a lot of my projects.

[enesaltinkaya]

Can you take a look at RmlUi https://github.com/mikke89/RmlUi It renders html+css and gives back vertices, indices, and textures. Used for game uis.

[tlf30]

@enesaltinkaya I don't see a C api for it, only a c++ api. It will require a C api for mapping to LWJGL.

[enesaltinkaya]

If I were to wrap a C api around this library (somehow), what would be the next step for integrating into LWJGL? Is there an automated way?

[dustContributor]

If I were to wrap a C api around this library (somehow), what would be the next step for integrating into LWJGL? Is there an automated way?

Nope. LWJGL does has docs on how to contribute and so on.

https://github.com/LWJGL/lwjgl3/tree/master/doc

After reading that, best thing you can do probably is pick some simple lib already in LWJGL3 and look how its bindings are done.

I also made a bunch of questions about this in this issue here https://github.com/LWJGL/lwjgl3/issues/428 that may help you, it was a long time ago though.

[Displee]

Update: https://github.com/SpaiR/imgui-java seems to be the most stable at this moment

The only nice thing that DriftFX seems to have and imgui not is having multiple main windows. But DriftFX doesn´t work on Linux, only on Windows. Which is the only reason for me to use imgui over DriftFX at this moment.