Calling RayHandler.updateAndRender() confuses ScreenUtils

[haimat]

I take screenshots as usual in libGDX via ScreenUtils.getFrameBufferPixmap(), which works fine. Hoever, when I call RayHandler.updateAndRender() during rendering, then the screenshot only contains those assets rendered after this call. All other pixels in the screenshot are transparent.

What is the best way to deal with this issue?

[rinold]

I've tried simple example:

    ...
    rayHandler.updateAndRender();
    /** BOX2D LIGHT STUFF END */

    Pixmap pixmap = ScreenUtils.getFrameBufferPixmap(0, 0, Gdx.graphics.getWidth(), Gdx.graphics.getHeight());
    batch.begin();
    batch.draw(new Texture(pixmap), 0, 0, Gdx.graphics.getWidth() / 50, Gdx.graphics.getHeight() / 50);
    batch.end();
    pixmap.dispose();

And got it working fine - mini-screen at center-bottom:

Need some example on which this could be reproduced.

[haimat]

That's strange, I don't understand why it does work for you, but not in my case. So I have commented nearly everything out in my render() method, leaving only this:

camera.update();
spriteBatch.setProjectionMatrix(camera.combined);
spriteBatch.begin();
// render assets...
spriteBatch.end();

rayHandler.setCombinedMatrix(camera);
rayHandler.updateAndRender();

box2daccumulator += delta;
while(box2daccumulator >= BOX2D_TIME_STEP) {
    box2dWorld.step(BOX2D_TIME_STEP, 6, 2);
    box2daccumulator -= BOX2D_TIME_STEP;
}

Pixmap pixmap = ScreenUtils.getFrameBufferPixmap(0, 0, Gdx.graphics.getWidth(), Gdx.graphics.getHeight());
spriteBatch.begin();
spriteBatch.draw(new Texture(pixmap), camera.position.x, camera.position.y);
spriteBatch.end();

That doesn't work, I only see the normal game view, but not the pixmap. However, as soon as I comment out the line rayHandler.updateAndRender(); the pixmap shows up in the top right quarter of the screen, as intended.

For the records, I initialize box2dlights like this:

RayHandler.setGammaCorrection(false);
rayHandler = new RayHandler(box2dWorld);
rayHandler.useDiffuseLight(true);
rayHandler.setCulling(true);
rayHandler.setBlurNum(3);

Ohh and by the way: I am using your fork of box2dlights currently, the one you gave me recently to test the pseudo-3d feature. Maybe this has something to do with it?

[rinold]

Ok... pseudo-3d might affect it... It uses frame buffers texture bindings which might produce your issue. Think I need to check for correct 'unbind' method, which I haven't found before.

BTW, if you are using stepped box2d simulation, possibly the box2dlights should be also stepped for performance reasons, like:

rayHandler.setCombinedMatrix(camera);
rayHandler.render(); // do the render

box2daccumulator += delta;
while(box2daccumulator >= BOX2D_TIME_STEP) {
    box2dWorld.step(BOX2D_TIME_STEP, 6, 2);
    rayHandler.update(); // update physics
    box2daccumulator -= BOX2D_TIME_STEP;
}

Also just strange for me, why you update world only after the rendering? imho, the update should preceed the draw procedures, otherwise it means for current frame you render outdated world.

[haimat]

Ok fine, then I will wait for your next update to box2dlights pseudo-3d and test again. Do you have any time estimation when you might have the next version ready for testing?

Ohh and thanks for your info regarding the stepped box2d simulation :-)

[rinold]

You can try the box2dlights from pseudo3d branch (this, not mine :)), however it have some major API changes, but for me the screen utils worked fine:

If you will still face the issue, I will do some additional investigation.

For pseudo-3d you will need importing things from "box2dlight.p3d" package, and P3dLightManager is what was the rayHandler (renamed cause it not uses raycasting anymore). I will try to describe it's usage on a Wiki with details but don't know when will have time :( However, I tried not to change the API gradually.

[haimat]

Thanks. I looked into your last changes and checked the ScreenUtils issue again. I also tried it with the box2dlights lib from Maven. In both cases I can still reproduce the problem.

So I created a minimal Screen class for you to demonstrate the issue. I really hope you can reproduce it with this example:

import box2dLight.RayHandler;

import com.badlogic.gdx.Gdx;
import com.badlogic.gdx.Input;
import com.badlogic.gdx.Screen;
import com.badlogic.gdx.math.Vector2;
import com.badlogic.gdx.physics.box2d.Box2D;
import com.badlogic.gdx.physics.box2d.World;
import com.badlogic.gdx.utils.ScreenUtils;
import com.badlogic.gdx.assets.AssetManager;
import com.badlogic.gdx.graphics.GL20;
import com.badlogic.gdx.graphics.OrthographicCamera;
import com.badlogic.gdx.graphics.Pixmap;
import com.badlogic.gdx.graphics.Texture;
import com.badlogic.gdx.graphics.g2d.TextureAtlas;
import com.badlogic.gdx.graphics.g2d.TextureRegion;
import com.badlogic.gdx.graphics.g2d.SpriteBatch;

class TestLevel implements Screen {
    World box2dWorld;
    RayHandler rayHandler;
    SpriteBatch spriteBatch;
    OrthographicCamera camera;

    TestLevel() {
        spriteBatch = new SpriteBatch();
        camera = new OrthographicCamera(1024, 768);

        Box2D.init();
        box2dWorld = new World(new Vector2(0, 0), true);

        RayHandler.setGammaCorrection(false);
        rayHandler = new RayHandler(box2dWorld);
        rayHandler.setDiffuseLight(true);
        rayHandler.setCulling(true);
        rayHandler.setBlurNum(3);

        rayHandler.setAmbientLight(1f, 1f, 1f, 1f);
    }

    @Override
    void render(float delta) {
        Gdx.gl.glClearColor(0.25f, 0.25f, 0.25f, 1f);
        Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT);

        camera.update();
        spriteBatch.projectionMatrix = camera.combined;

        // load a texture to draw here
        //Texture tex = new Texture(...);
        spriteBatch.begin();
        spriteBatch.draw(tex, -300, -100);
        spriteBatch.end();

        // it doesn't work anymore when I uncomment the next line
        //rayHandler.updateAndRender();

        Pixmap pixmap = ScreenUtils.getFrameBufferPixmap(0, 0, Gdx.graphics.width, Gdx.graphics.height);
        spriteBatch.begin();
        spriteBatch.draw(new Texture(pixmap), 0, 0, Gdx.graphics.width * 0.5f, Gdx.graphics.height * 0.5f);
        spriteBatch.end();
    }

    @Override
    void resize(int width, int height) {}

    @Override
    void show() {}

    @Override
    void pause() {}

    @Override
    void resume() {}

    @Override
    void hide() {}

    @Override
    void dispose() {}
}

[haimat]

Hi, any news on this issue? Did you had some time to experiment with my code sample?

[rinold]

Still can't reproduce it, seems to work fine on my machine. What libgdx version are you using?

[haimat]

I am using libGDX 1.5.4, from Maven. How can the same code lead to such different results for us? :-( Btw. just tried the code I posted above on another machine, with the same bad result.

[rinold]

I will try to run your complete example with the Scene on my machine with same libgdx version. But a bit later, seems my hard drive at home burned :( and I will need some time to restore all the lost data and got my machine work again.

[haimat]

Ohh sorry to hear :( Good luck with that!

Btw. please let me know if there is anything I can do to support you with this issue. I am really interested in sorting this out, but unfortunately I am not sure what I can do.

[haimat]

Sorry to bother you again, but did you had any luck with your hard drive? Have you been able to run my code sample?

[dingjibang]

(sorry for my bad english) I had a same program. On my home computer(driver is the R9280X),use box2dLight with screenUtils is not work,but it works fine on my company's computer(driver is a old Intel's HD Graphics). After,I found when I set RayHandler.useDiffuseLight to false it works fine,but the hover light is awful

[dingjibang]

just now I tried to update my game libgdx version(from 1.5.0 to 1.5.4) But the problem still exists _(:3」∠)_ohhhh……

[rinold]

I've got the new one and setting up the environment, but still wasn't able to recover some important data from old :)

BTW, I've missed some important question - on which OS you are facing this issue?

[dingjibang]

win7 sp1 (home and company 's computer)

[rinold]

Ohh.. I can't even imagine what is happening now :( Just as a guess, looking at the box2dlights internals code, could you check following:

        ...
        rayHandler.updateAndRender();

        Pixmap pixmap = ScreenUtils.getFrameBufferPixmap(0, 0, Gdx.graphics.width, Gdx.graphics.height);
        Texture tex = new Texture(pixmap);
        tex.bind(0); // <-- Does the issues still exists with this code?
        spriteBatch.begin();
        spriteBatch.draw(tex, 0, 0, Gdx.graphics.width * 0.5f, Gdx.graphics.height * 0.5f);
        spriteBatch.end();
        ...

And also does the ScreenUtils.getFrameBufferTexture(...) have the same issue?

[dingjibang]

no…… whatever I add or not add tex.bind(0) the issues still exists(:3」∠)

sorry the CHINESE network is too bad I can't upload result image the code is same as your example(:3」∠)

[dingjibang]

http://blog.rpsg-team.com/?p=187 here are results

[dingjibang]

just now I tried to use vmware ubuntu kylin system and there is no issues! The same code and hardware but differences os are differences result……

BUT my company's PC also use win7 system,eh there maybe some differences I will give you more info tomorrow

[haimat]

Adding tex.bind(0) as you suggested did not solve the issue for me. I am using Linux with an nVidia GTX 660 Ti card and original nVidia drivers. Also tried it on a Windows 7 machine with an AMD R7 200 card, same problem.

[rinold]

It's hard to understand while I can't reproduce it :( as @dingjibang said that it works when diffuse light is disabled - that means the issue might be caused by FBO and diffuse shader, if so it might be not the box2dlights issue but some issue with libgdx stuff it uses. What I can suggest currently - is to create the minimal sample after which the ScreenUtils fails to create screenshot - basing on box2dlights LightMap render method code but with all unnecessary code removed and possibly replaced with some dummy rendering stuff. Looking currently think we could try to reproduce the issue with following scenarios:

1. Create the shader manually by calling:

    ShaderProgram shader = DiffuseShader.createShadowShader();
    // do some draw like in LightMap.render with this shader
    ...
    // Try to grab the screenshot

1. Do the previous step but adding usage the FBO as in LightMap

If one of this small scenarios is failing then we can go with this results to forum/issue tracker and got it solved. Otherwise I'll keep looking for a machine on which I could reproduce it and will experiment with it by myself.

[haimat]

Thanks for your reply. I am sorry though, I have not much experience and knowledge with all this shader and FBO stuff, so I don't really know what to do now.

Could you please be so kind and tell me exactly what you want me to do? Thanks!

[dingjibang]

me too:(

[rinold]

It's ok, don't worry :) I just need some free time to create some minimal samples at work... I will post them here when done.

[dingjibang]

thanx

[haimat]

Super, thank you very much!

[szszss]

I tried @haimat 's example (with a few modifications, of course), and "it opened a gate of new world for me" because it was the first time in my life that SEEING A INTEL INTEGRATED GRAPHICS PROCESSOR WORKS BETTER THAN NVIDIA DEDICATED GRAPHICS CARD. When the Intel managed to snap a wrong picture (the screenshot has lots of blank), the Nvidia was silent totally. Anyway, I noticed that when Nvidia card snapped screenshots without light rendering, it can snapped all opaque objects (in another word, all the screen excluding where is blank), and I also noticed that the ScreenUtils always snaps a screenshot with Alpha, which is impossible displayed in screen. So I copy a getFrameBufferPixmap to my testing class, modifying the format of pixmap from RGBA8888 to RGB888, and modifying the format of glReadPixels from RGBA to RGB. Eventually, it works on my laptop with both Intel and Nvidia card. I guess there is something violates the specification of OpenGL in the shading of light, and it makes the card giving up to read pixels with all four channels. The using of FrameBuffer seems having no problem. Should the shaders of Box2Lights clamp the Alpha of fragment? And there is also a silver bullet if your game only works in desktop and the solution in above is invalid: change the reading buffer. In the default state, glReadPixels only read from backend buffer (you have already known the double buffer, right?). However, you can use _GL11.glReadBuffer(GL11.GLFRONT) to let it read from front buffer. This is real "What You See Is What You Get", but it can only get the screenshot of past one frame, not this frame. And you can't draw the screenshot in your screen in real time (you know why, hehe).

[dingjibang]

thank u, I'll try

[rinold]

@szszss thanks for your investigation! :) Actually, I'm not an expert in everything you described, but:

Anyway, I noticed that when Nvidia card snapped screenshots without light rendering, it can snapped all opaque objects (in another word, all the screen excluding where is blank), and I also noticed that the ScreenUtils always snaps a screenshot with Alpha, which is impossible displayed in screen. So I copy a getFrameBufferPixmap to my testing class, modifying the format of pixmap from RGBA8888 to RGB888, and modifying the format of glReadPixels from RGBA to RGB. Eventually, it works on my laptop with both Intel and Nvidia card.

Might this alpha channel problems be caused by box2dlights render method leaves with the GL_BLEND disabled? And also as @dingjibang mentioned that this happens for diffuse lights, where in shader really nothing is done with alpha channel? How it works if in "DiffuseShader.java" fragment shader we add setting of the alpha?:

                + "{\n" //
                + "gl_FragColor.rgb = (ambient.rgb + texture2D(u_texture, v_texCoords).rgb);\n" //
                + "gl_FragColor.a = ambient.a;" // <-- add settings of alpha
                    + "}\n";

[szszss]

Hmmm... I'm not sure does it caused by blend. However, I built a modified demo which tries to draw a screenshot of left half side in the right half side, and the screenshot was snapped by ScreenUtils. As was expected, it didn't draw. And then I added setting of the alpha in the DiffuseShader, and this time it worked.

(Figure above: Without setting. Figure below: With setting.)

(However, in my computer I didn't use "gl_FragColor.a = ambient.a;", because the screenshot will became transparent. I used "_glFragColor.a = 1.0;". I haven't too much understandings about the shading pipeline of Box2dLights, so the choice is yours. Just don't make the alpha undefined, neither the drivers nor specification like this situation.)

(Although Gaussian hasn't setting of the alpha, too, it isn't the last step, and the ShadowShader and WithoutShadowShader can fix its mistake.)

[rinold]

Great! Seems we found the root cause :) Using 1.0 is absolutely better than ambient, but still I need to think about what the alpha value will be actually correct there.

[haimat]

Sounds great, thanks for your analysis and efforts guys! I am looking forward to testing any update you come up with. Please let me know if I can help you in any way.

[rinold]

May be it will be correctly to use the alpha of sampler texture, but I've leaved the 1.0 value cause haven't find the real usage benefits of it. Updated both branches, should be fixed now.

[haimat]

Great, thank you all very much! Will the fixed version be published via Maven anytime soon? Or alternatively: Is there a nightlies version of box2dlights in Maven?