Nifty GUI doesn't display correctly after context restart

[ghost]

It seems JME's context restart feature is flawed because with the sample code below, the Nifty GUI refuses to display at all:

import com.jme3.app.SimpleApplication;
import com.jme3.material.Material;
import com.jme3.niftygui.NiftyJmeDisplay;
import com.jme3.scene.Geometry;
import com.jme3.scene.shape.Box;

import de.lessvoid.nifty.Nifty;
import de.lessvoid.nifty.builder.LayerBuilder;
import de.lessvoid.nifty.builder.PanelBuilder;
import de.lessvoid.nifty.builder.ScreenBuilder;
import de.lessvoid.nifty.controls.button.builder.ButtonBuilder;
import de.lessvoid.nifty.screen.Screen;
import de.lessvoid.nifty.screen.ScreenController;

public class NiftyJme3RestartTest extends SimpleApplication implements ScreenController {

    public static void main(String[] args) {
        new NiftyJme3RestartTest().start();
    }

    private NiftyJmeDisplay niftyDisplay;

    @Override
    public void simpleInitApp() {

        // this box here always renders
        Box b = new Box(1, 1, 1);
        Geometry geom = new Geometry("Box", b);
        Material mat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
        mat.setTexture("ColorMap", assetManager.loadTexture("/com/jme3/app/Monkey.png"));
        geom.setMaterial(mat);
        rootNode.attachChild(geom);

        niftyDisplay = NiftyJmeDisplay.newNiftyJmeDisplay(assetManager, inputManager, audioRenderer, guiViewPort);

        Nifty nifty = niftyDisplay.getNifty();
        nifty.loadStyleFile("nifty-default-styles.xml");
        nifty.loadControlFile("nifty-default-controls.xml");

        ScreenController ctrl = this;

        new ScreenBuilder("start") {
            {
                controller(ctrl);
                layer(new LayerBuilder() {
                    {
                        childLayoutVertical();
                        panel(new PanelBuilder() {
                            {
                                childLayoutCenter();
                                width("100%");
                                height("50%");
                                backgroundColor("#ff0000");
                            }
                        });
                        control(new ButtonBuilder("RestartButton", "Restart Context") {
                            {
                                alignCenter();
                                valignCenter();
                                height("32px");
                                width("480px");
                                interactOnClick("restartContext()");
                            }
                        });
                    }
                });
            }
        }.build(nifty);

        guiViewPort.addProcessor(niftyDisplay);
        nifty.gotoScreen("start");

        flyCam.setDragToRotate(true);
    }

    @Override
    public void bind(Nifty nifty, Screen screen) {
    }

    @Override
    public void onStartScreen() {
    }

    @Override
    public void onEndScreen() {
    }

    public void restartContext() {
        // even without changing settings, stuff breaks!
        restart();
        // ...and re-adding the processor doesn't help at all
        guiViewPort.addProcessor(niftyDisplay);
    }

}

The code was tested with jMonkeyEngine 3.2.2-stable. When run, the output log is pretty standard - there aren't any exceptions thrown or anything. Also, it seems the flyCam 'drag to rotate' functionality breaks after restart as well, unable to... well, drag at all.

On a slightly more complicated GUI setup that I've made (with nested panels and a few more buttons written in XML) the screen became green after restarting the context.

Perhaps the problem can be related to this post?

I haven't explored the engine much, so I'm hoping somebody who knows what they're doing can investigate this issue.

[tonihele]

This seems to happen only with LWJGL 3, right? At least for me the LWJGL 2 seems to work just fine.

[tonihele]

With LWJGL 3 the keyboard and mouse listeners are also tied to the window. When the context is restarted, the window is re-created. This seems to break the keyboard and mouse listeners as they still refer the old window handle. That explains the problems you get with the flycam. This one is easy to fix.

For our game the restart results in the whole scene gone, both Nifty and the regular scene context. The Nifty is there still alive and kicking, but I just can't see it. Any buttons etc. seem to work.

[stephengold]

Pretty please may I see a pull request?

[tonihele]

I pretty please need help :) I was able to fix the keyboard & mouse issues so that it only slightly feels like a hack (re-init them to the new window). But the graphics.... I don't understand, I tried all kinds of stuff. I carefully compared the LWJGL 2 code from jME. And tried to add some renderer update code etc. what was missing. I tried to have the previous window still open and have a shared context between them... I guess the buffers are just wrong or something in the new window.

I just don't understand it. I don't have enough competence for this I'm afraid. I haven't yet give up on this but I recognize that I'm just taking stabs at it blindly. I can't grasp the actual problem.

One crappy workaround would be that we just change the full screen status and the resolution on LWJGL 3 restart. Since that is all you can do without actual context restart.

[stephengold]

Okay, I read "easy to fix" and assumed you had a complete fix. I'm a bit behind the learning curve on LWJGL 3 myself.

@Ali-RS you can assist @tonihele ?

[Ali-RS]

Sorry, unfortunately this is out of my knowledge :( Also I have no experience with Nifty gui.

[tonihele]

Just to be clear, I don't think this is just related to just Nifty GUI. We just both happen to use it with the op. We have in app graphic settings made with Nifty GUI. But anyway, the example provided by op should be enough. I didn't actually tried it, but I assume the whole screen goes blank as in our game. That means also normal scene graph stuff just vanishes, right?

[tonihele]

Hmm, in the example the scene graph holds, just the Nifty and render statistics disappears.

[pspeed42]

In JME nifty uses its own viewport and stuff, I guess.

If Nifty is not involved then someone should be able to make a test case that illustrates the issue that doesn't use nifty. And if that non-nifty test case works fine then is probably useful information, too.

[tonihele]

What happens to our game, is probably just related to the complexity of our Nifty, it somehow hides(?) the regular viewport scene after the restart. But we can forget that for now. Here is in pictures what happens with the test code:

Before:

After:

[tonihele]

The statsappstate is in GUI view port. If I remove the Nifty, the statsappstate still vanishes. So I can conclude that it is not only Nifty that is feeling bad. And having Nifty doesn't cause this.

[stephengold]

Regarding the interference with StatsAppState, see issue #801.

[tonihele]

I can try the solution provided in https://hub.jmonkeyengine.org/t/problem-with-gui-z-order-after-graphics-restart/40985/4. To me it sounds something that we perhaps need to do anyway for consistency. Although again, I don't understand it :) But lets see what it does.

[tonihele]

This https://hub.jmonkeyengine.org/t/problem-with-gui-z-order-after-graphics-restart/40985/4 seems to solve the https://github.com/jMonkeyEngine/jmonkeyengine/issues/801. But this issue remains.

[ghost]

Well, I was able to get the Nifty GUI to show after the context restart by doing something like this in restartContext:

niftyDisplay.cleanup();
restart();
enqueue(this::setupNiftyDisplayInstance);

...where setupNiftyDisplayInstance contains all the code in the original example that sets up the GUI.

However, input does not seem to work: clicking the button again does nothing, nor does hovering over it change its style.

[tonihele]

https://github.com/jMonkeyEngine/jmonkeyengine/pull/1268 - will fix the inputs https://github.com/jMonkeyEngine/jmonkeyengine/pull/1269 - will fix the stats app state

The original issue still remains. Interestingly after restart if one just says: nifty.gotoScreen("start");

The restart button draws a green rectangle and when pressing it, briefly the original screen flashes on the screen. I dunno does this help at all.

[tonihele]

This only happens with the JmeBatchRenderBackend. The legacy RenderDeviceJme works just fine.

[stephengold]

I believe this is fixed. Correct?

[tonihele]

I believe this is fixed. Correct?

Nej! Not yet fixed! Only a bit improved. The mouse & keyboard now works but the screen is still blank.

I did discover a fix I last time played with this. But it requires some work.

[stephengold]

Okay, I'll leave it open.

[fba-rio]

I have something to add to this issue. It seems like restart() issues are not limited to Nifty and keyboard/mouse input on LWJGL 3. The following code changes AppSettings resolution from 1920x1080 to 1280x720 on a window (not full screen). It works on LWJGL 2, but on 3 the window size changes after restart(), but everything inside the window behaves like the resolution hasn't been changed. Gamma correction is also ignored after calling restart() and behaves like it's always false.

Run the code and press Enter to swap between 1920x1080 and 1280x720 resolutions.

import com.jme3.app.SimpleApplication;
import com.jme3.input.KeyInput;
import com.jme3.input.controls.ActionListener;
import com.jme3.input.controls.KeyTrigger;
import com.jme3.light.DirectionalLight;
import com.jme3.math.ColorRGBA;
import com.jme3.math.Vector3f;
import com.jme3.scene.Spatial;
import com.jme3.system.AppSettings;

public class TestWindowSizeChange extends SimpleApplication implements ActionListener {

    static AppSettings[] settings = new AppSettings[2];

    private int i = 0;

    public static void main(String... args) {
        // App settings for resolution 1920x1080, windowed app (fullscreen = false).
        settings[0] = new AppSettings(true);
        settings[0].setFullscreen(false);
        settings[0].setGammaCorrection(true);
        settings[0].setResolution(1920, 1080);

        // App settings for resolution 1280x720, windowed app (fullscreen = false).
        settings[1] = new AppSettings(true);
        settings[1].setFullscreen(false);
        settings[1].setGammaCorrection(true);
        settings[1].setResolution(1280, 720);

        TestWindowSizeChange app = new TestWindowSizeChange();
        app.setSettings(settings[0]);
        app.setShowSettings(false);
        app.start();
    }

    public TestWindowSizeChange() {
        super(null);
    }

    @Override
    public void simpleInitApp() {
        DirectionalLight sun = new DirectionalLight();
        sun.setDirection((new Vector3f(-0.5f, -0.5f, -0.5f)).normalizeLocal());
        sun.setColor(ColorRGBA.White);
        rootNode.addLight(sun);

        Spatial spatial = assetManager.loadModel("Models/Jaime/Jaime.j3o");
        spatial.setLocalScale(3);
        rootNode.attachChild(spatial);

        inputManager.addMapping("Change Window Size", new KeyTrigger(KeyInput.KEY_RETURN));
        inputManager.addListener(this, "Change Window Size");
    }

    @Override
    public void onAction(String action, boolean isPressed, float tpf) {
        if (isPressed) {
            if (++i > 1) {
                i = 0;
            }
            setSettings(settings[i]);
            restart();
            System.out.println("\nwindow size changed");
            System.out.println(cam);
        }
    }
}

The output of System.out.println(cam) is the following:

On LWJGL 2:

window size changed
Camera[location=(0.0, 0.0, 10.0)
, direction=(0.0, 0.0, -1.0)
res=1920x1080, parallel=false
near=1.0, far=1000.0]

window size changed
Camera[location=(0.0, 0.0, 10.0)
, direction=(0.0, 0.0, -1.0)
res=1280x720, parallel=false
near=1.0, far=1000.0]

On LWJGL 3

window size changed
Camera[location=(0.0, 0.0, 10.0)
, direction=(0.0, 0.0, -1.0)
res=1920x1080, parallel=false
near=1.0, far=1000.0]

window size changed
Camera[location=(0.0, 0.0, 10.0)
, direction=(0.0, 0.0, -1.0)
res=1920x1080, parallel=false
near=1.0, far=1000.0]

As you can see on LWJGL 3 it's always res=1920x1080.

[tonihele]

I also think that it is not just limited to Nifty. In your example though, I'm unsure why it behaves like that with LWJGL 2, since both have actual delay after restart() for things to happen. They both do it on the next frame update. Maybe the key listener runs on the same update queue... But LWJGL 3 introduces a new delay also, it triggers the reshape listener from a callback.

While this is a behavioral change for sure, I don't think the restart() is never supposed to be trusted like that. You should hook on to the listener instead to get notified of the change.

[tlf30]

Perhaps this is related: https://hub.jmonkeyengine.org/t/white-screen-on-context-restart/43866/7 Upon a context restart for LWJGL 3 the screen is just white, nothing is rendered, but key input still works.

[stephengold]

@tonihele Please re-test with JMonkeyEngine v3.4.

[tonihele]

Well, the situation is far better than it was! With 3.3 I would be left with a totally black screen after restart, with 3.4: (the textures maybe just disappeared but at least some of the content stays intact) LWJGL3, Before restart:

LWJGL3, After restart:

[stephengold]

Thanks for testing. I'll leave this issue open.

[stephengold]

I suspect this is related to issue #1793.

[tonihele]

Still happens on the master (jme3test.niftygui.TestIssue1013)