Add a GUI solution to LWJGL

[Spasi]

This is meant to be a meta-issue in which discussion can take place about potential solutions to GUI layout & rendering. Currently LWJGL only supports text rendering (via stb) and may soon support antialiased 2D graphics (via nanovg).

Topics to discuss:

• Rendering GUI components on top of 3D content (e.g. game menus, HUDs, etc).
• Rendering GUI components inside 3D content (e.g. monitor/panel in 3D space displaying 2D content).
• Embedding 3D content in a standard application/tool. The 3D content should be lightweight, i.e. standard menus should be renderable on top of it.

The ideal solution should be hardware-accelerated, i.e. there shouldn't be any framebuffer copies from the CPU to the GPU or vice versa.

Approaches that have been explored already:

• JavaFX integration via PixelWriter & PixelReader. See the LWJGL-FX project (uses LWJGL 2 but the same issues apply to LWJGL 3). Not hardware-accelerated.
• Adding bindings to CEF and using the offscreen rendering API. Not hardware-accelerated.

[httpdigest]

Also I have a running version of Themable Widget Library, which works really awesome with LWJGL 3 and kappa's lwjglx library. I would really love for someone to do the porting to native LWJGL3 without needing an adaptation layer. The render code of TWL is really nicely decoupled from everything else and can be swapped once a new LWJGL3 implementation is available.

[kappaOne]

Do agree that TWL is a great option for rendering GUI on top of 3d content. It only supports LWJGL2 atm, though looking at the code base it doesn't seem all that much work to port to LWJGL3.

However for standalone ui's, SWT is probably the best and most mature option out there. It integrates nicely with the native platform and is small and lightweight. It also strips down pretty nicely (to a few KB's, the Avian example are good examples of how small, less than a meg for SWT support and the Avian JVM!).

So looking into getting LWJGL3 nicely integrated with SWT should be explored. IMO a much better investment then spending time on AWT or JavaFX.

Interestingly, l33tlabs (who host TWL), work on a really interesting project called Bling. They've somehow integrated SWT with TWL! I think they also use LWJGL2. They already have a full IDE running on the same, so pretty far along. There is a pretty long video here explaining how it works and theres a quick preview of what it looks like here.

I'm not sure what l33tlabs plans are but if they released the underlying SWT/TWL library or if something similar could be written, it could probably be the ultimate and complete solution for LWJGL3 i.e. a small lightweight API that can render ui inside OpenGL applications, create native platform widgets (like dialog's, system tray integration, native controls, etc) or even enable full blown native applications that can render opengl content inside it (e.g. particle editors, game engine editors, etc).

[renannprado]

I wonder if it would be possible to somehow render HTML+CSS (and possible code the UI logic in javascript via nashorn maybe) for HUD? I've researched quite a bit some time ago but couldn't find anything like that or even a reference. It would also be nice to check how games such as Heroes of Storm and Starcraft 2 (which, IMO, have nice HUD) are doing their HUD.

Another thing that maybe could help is the project dukescript.

[httpdigest]

I think using Nashorn and JavaScript would not be the problem; the library implementation and its API would be, as it sounds rather like the tasks of browser engines like WebKit or Blink and the domain of the Web platform in general. However it would sure be great. :) IMHO when layouting with HTML/CSS, I would then switch to WebGL (like with three.js), Canvas "2d" based 3D frameworks such as phoria.js or CSS 3D Transformations, instead of developing a desktop application with Java/LWJGL. dukescript however sounds fascinating! :)

[renannprado]

Yeah, that's the thing. I didn't mean to create something like WebKit, but if we could create a binding for it and use it to somehow render the HUD, that is what, IMO, would be awesome. (I don't have any clue if it's possible, maybe I'm saying some non-sense here. I'm just trying to think from the library end user POV and, for me, HTML+CSS (and possible javascript) is the most flexible way to write UI today)

[httpdigest]

This "awesomium" seems to be what you were looking for:

• http://gamedev.stackexchange.com/questions/47361/draw-user-interface-in-opengl-using-web-technologies
• http://awesomium.com/
• http://wiki.awesomium.com/general-use/working-with-offscreen-web-views.html

It also allows to render into an OpenGL surface (one can define its own surface factory). That could be an option.

[dustContributor]

In my humble opinion its more of an issue if a GUI lib doesn't supports core OpenGL than not supporting LWJGL 3 (TWL is really nice but it's renderer is GL 1.1 exclusively).

NiftyGUI is there, has a core OpenGL backend, but doesn't supports LWJGL 3 (mainly Display/Keyboard/Mouse related changes and some function name changes).

Also another concern of mine is the state in which the GUI lib leaves OpenGL. This should be very well defined, ie, what state OpenGL needs to be before the GUI lib can do whatever it needs to do, and what state is OpenGL left with once it finishes. This is vital for efficient state tracking.

[JustGregory-zz]

Unlike @dustContributor I'm not concerned with supporting "core OpenGL" standing as much as if it fits in the "lwjgl wheelhouse" of libraries and functionality, and if the general API fits thematically with other, existing lwjgl API calls. I don't have a specific recommendation on how to do that, so long as I would be able to pass in some UI component hierarchy, and just tell it to "show the gui".

What would be wrong with basically "copying" the AWT and making it render for lwjgl?

[httpdigest]

Supporting OpenGL 1.1 and not >= 3.2 core is an issue if you yourself want to use a "modern" OpenGL context and functions therein, which would both not be possible in a single context. Either TWL would not function correctly because it used deprecated OpenGL functions or you would not be able to use "new" OpenGL functions. This is an issue with Mac OS X, which either supports OpenGL <= 2.1 if you don't ask for a new context specifically, or you would only get a "core" context if specifically asked for, but then TWL would not work.

One solution to this problem, and also to the problem of touched OpenGL state, could be multiple sharing contexts. One "deprecated" OpenGL context for TWL and another one for your own rendering. TWL could be issued to render into a texture using a Framebuffer Object and then that texture can be shared between both contexts to simply blit the texture onto your own "new" context's window.

Sharing between an "old" deprecated context and a "new" core/forward compatible context does work at least under Windows with Nvidia.

However, only "requiring" OpenGL 1.1 is the smartest thing one can do for such a widespread library like TWL, in my opinion. Whenever you require anything higher than OpenGL 2.0 it will not work on some person's machine. Compatibility is a very big thing here!

[dustContributor]

That sounds both slow and bound to hit driver specific issues. No, I'd rather handle the state change myself.

At the very least it should have a renderer interface that doesn't assumes fixed-function pipeline working underneath, TWL has that issue in various places last time I checked, which makes it a pain in the butt to code a core renderer for.

I still propose NiftyGUI as a better alternative, right now it just needs a couple of classes using a LWJGL 3 backend. I opened an issue about it a while ago https://github.com/void256/nifty-gui/issues/304

There is literally no other GUI library that has a core renderer that I know of (for Java that is), thus why I think its an important matter. OpenGL 3.0 has been out there for 7 years, with hardware supporting it going as back as 2006 (GeForce 8xxx, Radeon HD2xxx), that was almost ten years ago.

[Spasi]

However for standalone ui's, SWT is probably the best and most mature option out there. It integrates nicely with the native platform and is small and lightweight. It also strips down pretty nicely (to a few KB's, the Avian example are good examples of how small, less than a meg for SWT support and the Avian JVM!).

So looking into getting LWJGL3 nicely integrated with SWT should be explored. IMO a much better investment then spending time on AWT or JavaFX.

Do we even need anything special for SWT integration? Looks like @httpdigest already has a sample here.

[httpdigest]

The only downside of SWT is that it still (even in the very latest nightly build) cannot produce a GL >=3.2 core context. I had a look at the sources and found that the WGL solution at least is devised but commented out... But this problem can be solved by letting LWJGL3 do the GL context creation. So we would need a cross-platform way to create GL contexts in LWJGL3 based on a window handle. This handle is easily API-accessible in SWT. Another solution would of course be to hack SWT, but I heard that their OS X backend is really heavily outdated, so it would be quite some hacking. I also dug out an old LWJGL forum entry about this: http://forum.lwjgl.org/index.php?topic=5531.0

[dustContributor]

I recall digging through the sources and finding no clear way to hijack the SWT's component so I could use it with my core context (although my experience with window management APIs is limited).

Both JavaFX and SWT (since 4.2 I believe) are theme-able with CSS, which is a pretty cool thing to have.

I wonder if we could nag SWT developers about this...

EDIT: Still no interest in NiftyGUI?

[httpdigest]

I am going to build an own GLCanvas, something like a org.lwjgl.opengl.swt.GLCanvas or org.lwjgl.swt.opengl.GLCanvas, based on the platform API exposed by LWJGL3. The native/JNI code of SWT is really horrible. They do like a thousand (*env)->GetField(...) invocations and other slow/nasty hacks. Personally I think, an own custom SWT Canvas would be the best solution, and I don't think it will require much maintenance, since it is only about creating the GL context and that should be common knowledge how to do that.

[crisp00]

So any update on lwjgl native gui rendering methods? Any news on nanovg implementation? Please keep us updated!

[Spasi]

@httpdigest is making good progress with SWT integration, see his latest update.

I experimented a bit more with JavaFX and have a working OpenGL backend that runs on top of LWJGL. This may seem like a huge step forward, but it's not enough. It would be easy to implement context sharing with the JavaFX context and access the textures directly. The problem is, JavaFX is architected in a such way that the rendering layer has no idea which textures correspond to which GUI elements. I don't have a solution to this yet, I'll explore some more ideas when I have time.

I don't know if nanovg bindings will make it to the final LWJGL 3.0 release, but I'll try.

[Spasi]

The latest nightly build has jawt bindings, which in theory allows for the AWT integration in LWJGL 2 to be implemented in terms of LWJGL 3. Sample code can be found here (simplest possible implementation, Windows-only).

[dustContributor]

I've been thinking. Now that NanoVG bindings are up, it might be possible to use it as backend for other GUI libs that lack an OpenGL core context, like TWL for example.

I'm looking at TWL's sources, seems a lot of work though -_-

[dustContributor]

I completely forgot about libRocket: http://librocket.com/

Its C++ though.

[ShadowLordAlpha]

This is kind of late but a while ago I did manage to make TWL run with LWJGL 3 though it wasn't good by any means and TWL does use old OpenGL functions.

I should also bring up that one of the main problems I was having was figuring out a good way to get the event data from GLFW and processes it. As there can only be one listener it would be easy to make a version that just splits that up however it gets more complicated for the end user and taking over isn't a very good option as its not very easy to detect if your listener has been removed and could lead to problems with the end users code.

[Spasi]

https://github.com/vurtun/nuklear

[dustContributor]

Sweet, it even leaves rendering up to the user by just giving you commands to execute, perfect.

[Caritas-Dei]

Hey, if at all interested, I'm making a library for lwjgl, but for a broader spectrum*, from scratch. It'll allow direct support for GLFW in LWJGL3, support for the latest OpenGL (4.5+), and a way to build shaders on the fly during runtime (in GLSL). The window component system will be made to support XHTML and XML based layouts for quick building of GUIs and easy management, so you could simply build it in a browser or other development environment.

(I don't have a repository yet, and I have a(n empty) repositoryI have a repository, but don't have internet. ~~ but I'll get one up as soon as possible. It's pretty bare bones right now anyways...)~~

*By which I mean it's for more than just lwjgl. It'll support GLFW, but it'll support other GUI systems as well: AWT, SWT, Swing, etc.

[SpinyOwl]

Hey! It would be great if you add bindings to this library NanoGUI Can you please take a look on it? :)

[arisona]

NanoGUI is C++ and probably hard to wrap. But it could make sense to port NanoGUI to Java on top of lwjgl's NanoVG. Edit: NanoGUI is quite decent, btw. IMHO would be a good choice.

[Caritas-Dei]

SGL now has code in it

[dustContributor]

https://github.com/glegris/nuklear4j

Dafuq, never saw what O_o

EDIT: Oh, SWIG : /

[Spasi]

LWJGL will very soon have Nuklear bindings, see the nuklear branch.

There's already support for everything, but I'm working on a Nuklear+GLFW+stb_truetype demo to understand and properly annotate the API (nullability, autosize parameters, etc).

[dustContributor]

Awesome! I didnt see that branch. I better read up on Nuklear then :smile:

[arisona]

Very cool!

@Spasi: It seems to be pretty fast to add bindings. I'd be interested contributing by adding bindings (in particular Valvoe / OpenVR). I couldn't find much documentation how to get this started. What's the normal procedure to start with bindings for a new library? Cheers :)

[Spasi]

@arisona

The best approach is cloning the repository, opening the IntelliJ project and exploring the Templates module. There are some important differences between bindings (loaded from a shared library vs baked into LWJGL, custom functionality like the capabilities classes, etc), but the general structure is the same:

• The org.lwjgl.<library_package> package contains the type, struct and callback definitions. The type definitions provide an important level of type-safety and "refactorability".
• org.lwjgl.<library_package>.templates package contains the actual binding templates, a file per generated class. These might look a bit scary, but it's mostly the documentation that makes it verbose. The most important thing is getting the "modifiers" right, but there is a ton of examples you can work with.

You should start with studying the GLFW bindings, I think that's the cleanest API. The LibOVR bindings should be fairly similar to OpenVR. You could also look for commit messages in the commit log that look like "Add <library name> bindings". These should give you a fairly good idea of everything required to add a new binding.

With that said, you don't have to do everything right. Simply contributing the types/structs/callbacks, templates and documentation will be a major help. I can do the rest of the work easily.

On OpenVR in particular: The C API has these FnTable structs that contain function pointers. They are loaded with the global VR_GetGenericInterface function (example here). No other library uses this scheme and LWJGL does not support it currently, so some customization will be required. Please open a new issue if you'd like to discuss this further.

[erodozer]

Will the nuklear bindings be moved into separate module that can be included optionally instead of being part of lwjgl3? I just feel that something like this, while valuable to the project and those who wish to use barebones lwjgl3, adds bloat and redundancy to other libraries that rely on lwjgl but provide their own gui solution, such as libgdx.

[Spasi]

@nhydock

Yes, see #100. The first 3.0.1 builds will have Nuklear built-in (adds about 200kb to the binaries) so that users can start working with it, but the plan is to split heavyweight bindings to separate artifacts.

[arisona]

@spasi thanks re generator. this should get me started.

[Spasi]

Demo code here.

[j-ibarra]

i have a problem is the code of the demos,

import static org.lwjgl.nuklear.Nuklear.*;

in the version of lwjgl 3.0.0 build 90 no have the package

[httpdigest]

Like @Spasi said:

The first 3.0.1 builds will have Nuklear built-in

You are using the 3.0.0 Release, which does not have Nuklear included. The sources of the Nuklear module, as well as the demo you are referring to, live in the nuklear branch, and you could build LWJGL with Nuklear from that branch manually.

[Spasi]

The first LWJGL 3.0.1 nightly build is now available and it includes the Nuklear bindings.

[pierr3]

Great job, thanks!

[dustContributor]

Awesome man, thanks a lot for your work!

[jediminer543]

Rendering GUI components inside 3D content (e.g. monitor/panel in 3D space displaying 2D content).

Is this something that can be done with nuklear?

[sriharshachilakapati]

@jediminer543

You can render the GUI to a texture using a framebuffer and then use it to render the monitor panel triangles in the 3D space. I never worked with Nuklear, but this is how you will be doing normally.

[codemuse]

Are there still plans to further explore JavaFX integration "natively" in lwjgl3, or is LWJGL-FX the current best alternative?

Most discussion in this ticket is about adding UI in a native GL display, what I have is an SWT based GUI that embeds lwjgl through an SWT GLCanvas to display a 3D representation that can be inspected using the SWT UI components around it, but we are looking at rewriting the GUI with JavaFX.

[Spasi]

Hey @codemuse,

Have you tried JavaFX Interoperability with SWT?

As for a pure JavaFX solution, I will probably try again when I have time, but failure is the most likely outcome.

[kappaOne]

IMO any proper solution is really going to need help from the JavaFX team at Oracle, such as them adding an API for this sort of stuff or for them to expose certain internals of JavaFX which we can use to implement it properly.

Without the above, any solution is just going to be a hack, likely non-optimal and subject to breakage. JavaFX at present just isn't designed to allow this sort of integration.

[kappaOne]

Interestingly, planning for JavaFX.next (Java 10+) is currently taking place over at http://mail.openjdk.java.net/pipermail/openjfx-dev/2016-December/020025.html and Oracle are asking for suggestions.

Might be a good opportunity to chime in and request support for OpenGL/Vulkan via LWJGL.

[httpdigest]

It seems that has already been requested:

• http://mail.openjdk.java.net/pipermail/openjfx-dev/2016-December/020058.html
• http://mail.openjdk.java.net/pipermail/openjfx-dev/2016-December/020059.html

Also for JDK8:

• http://mail.openjdk.java.net/pipermail/openjfx-dev/2013-May/007451.html

But I fear, because one of JavaFX's design goals being "abstract all and everything away" so that JavaFX is able to run on phones, on the Desktop, on the Browser platform and everywhere while being accelerated by either D3D, OpenGL or Vulkan (or what happens to be the hardware acceleration on that particular device), that Oracle will not break with this abstraction to actually expose an "OpenGLNode" (or even just a proposed "3D Canvas Node" telling the client that everything is currently being accelerated by D3D, OpenGL or Vulkan) on the API layer.

[codemuse]

@Spasi, @kappaOne and @httpdigest: thanks for all your quick replies!

Your comments are similar to what I could find myself as I was investigating this. I'm looking for a "pure" JavaFX solution, i.e. embedding it in SWT wouldn't really make sense, since the idea is to move our application away from SWT.

@httpdigest thanks for the pointers to the OpenJDK mailinglist, I'll follow that to see what comes out of it.

In the meantime, I'll toy around what sort of hack would -- although suboptimal -- work for our specific situation.

[sriharshachilakapati]

Now that we have JAWT bindings, can't we have it embedded into Swing and embed it into JavaFX using SwingNode?

[JustGregory-zz]

I will second what sriharsha said above, except I fear that Swing/JavaFX/SwingNode adds altogether more of the abstraction-complexity to the mix. If it could be done easily, it would have been already; but maybe it will take a complex implementation to simplify it somewhat.