DirtyOpts support

[DeathPhoenix22]

Hi,

Great job! I know that it's mandatory to deactivate DirtyOpts a.t.m. Is there any possible way to find a solution for this?

My game basically goes from 180+ FPS to 70FPS without dirty optimization.

Any tips or workaround? Thx

[Teragam]

Hi,

Disabling dirty region optimization is currently necessary for shaders that use varying UV coordinates when sampling the input texture. For example, this is required when a shader applies an offset to the coordinates to shift the image. With dirty region optimization, only the changed part of the scene will be rendered, so the shader gets applied only to that smaller portion of the scene. Consequently, the shader cannot access all the neighboring pixels of the full scene. Additionally, the textures themselves are smaller, and some effects might appear scaled down or stretched without proper correction.

The blurring effects of JavaFX avoid this by defining larger shader bounds that scale the dirty region bounds accordingly. Defining custom bounds or disabling dirty regions only for the shader itself is a feature I already planned to implement.

Does your game use shaders that lead to artifacts due to dirty region optimization?

[DeathPhoenix22]

Hi!

Thx for the details. Yes, my game uses a Shader that forces me to deactivate Dirty Region Optimization. Oh yes.... It's a FogOfWar Shader ;) which expends everywhere in the scene.

I had to expend the OutputRegion already on a Blur I've made. I noticed that Shaders will get clipped automatically to the Texture boundaries. For Example, the RadialBlur sample of JFXShader won't work if the projection strength is too big (causes Rays to go cut off)

Next step will be a Shadow shader that will technically lead to a similar problem.

What I've noticed as well is the changes on other Nodes of the scene doesn't cause a proper update on the Shader as well. But this could be a bug in my Shader.

I'm not certain how to solve this cleanely other than deactivating DRO.

[DeathPhoenix22]

Hi,

By looking at the implementation 8n JFXShader, the Peer implementation doesn't offer the possibility to define the Bounds and the DirtyRegion (defined in the GaussianBlur peer, for exemple). Could it be caused by that ? If I'm right, JFXShader relies on a default Peer which for all shaders defined.

I'll take a deeper look at this.

[Teragam]

Yes, JFXShader uses InternalEffect as the default peer, which internally uses the default bounds and dirty region calculation of the JavaFX base effect classes. I could add the option to define custom bounds in the ShaderEffectPeer that get used by the internal peer to extend the shader bounds and dirty regions accordingly. Regarding the fog of war, if a pixel only gets colored and not shifted in the position, it should work with dirty regions if the shader knows which part of the full scene gets rendered. However, I did not experiment with this enough to know if this information can be currently obtained in the ShaderEffectPeer. Additionally, I would be interested to know to what extent a node update does not lead to a shader update, as I have not yet experienced this (for 2D effects at least).

[Teragam]

Update: I figured out how to define custom effect bounds and how to handle dirty regions in shaders. The ZoomRadialBlur, where you pointed out the artifacts when using extreme values, was a perfect starting point for this. When setting the property prism.showdirty to true, JavaFX visualizes the dirty regions with red boxes. Below is a video showing the improved ZoomRadialBlur:

https://github.com/Teragam/JFXShader/assets/67922585/7fcc9b94-f2df-4fcd-b17e-305aee18affc

These bounds now also work when using the effect in an effect chain with different effects that also change the bounds. I delegated the necessary methods getBounds, getResultBounds and getDirtyRegions to the ShaderEffect. The fourth way to influence the rendering regarding the bounds is the RenderState. For the ZoomRadialBlur, rendering the effect for the dirty regions alone is not sufficient, as some region around the dirty region has an influence on the rendering result. Via the RenderState, the bounds of the input textures can be extended without changing the dimensions of the resulting rendered image. Unfortunately, using these methods to define custom bounds in addition to handling dirty regions is not straightforward. I did not abstract or ease this process significantly; however, I could add a section to the wiki that goes into detail on how to support dirty regions. I need to clean up some code before pushing these changes and need to test some edge cases regarding the automatic DPI scaling of JavaFX.

[DeathPhoenix22]

Hi Teragam, That is quick and efficient work! I can't wait to see your changes when you feel satisfied by them. This will send my FPS back to 180 if DRO works again! Awesome :D

[DeathPhoenix22]

Hi Teragam, any news on the integration of your fix? It will most likely change the FPS and the future of graphical effects ;)

[Teragam]

I currently don't have much time on my hands, unfortunately. Improving the DRO of the ZoomRadialBlur to support scaling transforms requires some more work, but it is mostly done. It might be finished this week or early next week.

[Birdasaur]

Hey Teragram I've been following your work and am also excited about it. I am especially interested in your 3D material support. Question: what is the hardest part of implementing these updates within your framework? I have a lot of 3D experience but honestly have never had to work on the shader side of things. I would be interested in helping you if I can understand how and where.

Thanks again for your work!

[Teragam]

Thanks for the kind words.

The most difficult part of implementing a new feature is figuring out how to embed this feature in the JavaFX rendering pipeline. JavaFX is not designed to allow the external replacement of certain rendering parts, and finding ways to implement the feature anyway is very time-consuming. A substantial part of this library is just a collection of reflection utilities, allowing me to replace internal components. Any new rendering feature must be implemented for both the DirectX 9 and OpenGL rendering pipelines, which only expose a limited subset of usable functions. This particularly limits what can be achieved with the 3D materials.

One planned feature I have, for example, is the ability to use multiple render targets for 2D and 3D effects. This could be used to implement deferred shading for PBR support. Because the original JavaFX DirectX rendering code for the MeshViews is written natively in C++, I have already added the Direct3DDevice9 wrapper to be able to call any DirectX 9 function. However, adding the OpenGL equivalent is something that needs to be figured out, as it has to use the same OpenGL context that JavaFX uses and must work for both Linux and macOS. This is also necessary for the OpenGL support of the custom vertex attributes.

Another feature I plan to implement as an exercise to test if the library provides enough utility functions are shadows. The ability to render a 3D scene from another angle and using the depth buffer as an input for a 3D material in the same frame might require a few additional features but should be possible.

Thank you for your interest in this library. Feel free to suggest any feature or problem you would like to work on. It does not have to be anything I listed above; anything that exposes bugs or missing features is good to know. I have some smaller unresolved bugs as well. They are easy to replicate but very time-consuming to debug and would not be particularly fun to work on.

[DeathPhoenix22]

Hi Teragam, I'm a solo dev for an RTSwP game and think that it could be interesting for us if we could work together. I'm heavily experienced in Java and starting with Shaders (planning to learn it to be fully autonomous). The thing is, I'm facing limitations (which is expected as your library is solving a 10+ years JavaFX limitation) and would appriciate your help in creating certain shaders (noise based water, better Fog Of War, Rain, Shadows, Godrays, dynamic lights raytracing, GPU handled layered Sprites, etc) I may be over optimistic, especially about unlocking a raytracing API, but the more is unlocked, the better it'll be for my game. I would like to have a discussion about how we could work together and make this library a Must Have. Eventually, it could be of interest for JavaFX OpenJDK team to simply... update their API with your changes directly in it (no more Reflection at that point) Interested?

[Teragam]

This certainly seems interesting, however we should keep in mind that other frameworks like libGDX are well more suited for graphics heavy applications. JavaFX is primarily designed for UIs and uses caching and DRO to perform best on small scene changes. This library was created to improve the very limited 2D effect and 3D capabilities, but it still uses the existing outdated graphics pipelines. The only viable thing would be to expose the graphics pipelines directly and leave it to the user to implement the desired features. The Direct3DDevice9 wrapper can currently be used in a custom 'IEffectRenderer' to integrate a custom pipeline when rendering effect shaders, however there is currently no example for this in the wiki.

Please let me know what features you need for your planned shaders. Most of them should already be possible for 2D, as I made some more complex shaders for the Demo-Page that need similar requirements. It helps a lot when the library is used and any bugs, missing features or rough parts are reported.

It would be great if the official JavaFX team eased the integration of custom effects. However, the effect system is deeply integrated in the 'com.sun.' packages, which are not meant for to be used when working with JavaFX as a library. The required work for official custom effect shaders would not be worth the benefits.

[Teragam]

Update: I pushed the changes to the develop branch and created the 1.3.2-SNAPSHOT version. It can be accessed by adding the snapshot repository to the pom.xml:

<repositories>
    <repository>
        <id>ossrh-snapshots</id>
        <name>OSSRH Snapshot Repository</name>
        <url>https://s01.oss.sonatype.org/content/repositories/snapshots/</url>
    </repository>
</repositories>

[DeathPhoenix22]

Thanks. I'll take a look at the impact.

I currently run the RTS in 120+ fps (on a GTX970 or Mac M2) with 100+ onscreen fully animated HD (not pixelated) units along with VFX effects. This also includes shadows (with sun projection) and night lightings similar to Factorio.

Shaders will help beautify the result without japerdizing performanced, at least I hope so.

For now, JavaFX is way more than a simple UI toolkit :)

Thanks for the changes !

[DeathPhoenix22]

Hi Teragam,

I've tried the DirtyOpts changes and it's really close in my case. I'm certainly doing something wrong though in the Shader itself. Here's an example of my problem (first ever screenshot of the game):

As you can see, the dirty regions of other entities are causing the shader to partially redraw which (the fog of war is redraw where the shadow of a tree changed)

Here is my GLSL shader:

#ifdef GL_ES
precision mediump float;
precision mediump int;
#endif

varying vec2 texCoord0;

uniform sampler2D baseImg;
uniform vec2 resolution;
uniform vec2 viewport;
uniform int axis;

const float radius = 16.0;
const float radiusPow = 256.0;
const float weightCenter = 0.087568952;

void main() {
    vec2 texel = texCoord0;
    vec4 col = vec4(0.0, 0.0, 0.0, 0.0);

    if(axis == 0) {
        float scaledPixel = 1.0 / resolution.x;
        texel.x -= radius * scaledPixel;
        for(float x = -radius ; x <= radius ; x++) {
            texel.x += scaledPixel;
            float neighborWeight = weightCenter * (1.1 - ((-x * x) / radiusPow))/(abs(x/4.0) + 1.0);
            col += texture2D(baseImg, texel) * neighborWeight;
        }
    }

    if(axis == 1) {
        float scaledPixel = 1.0 / resolution.y;
        texel.y -= radius * scaledPixel;
        for (float y = -radius ; y <= radius ; y++) {
            texel.y += scaledPixel;
            float neighborWeight = weightCenter * (1.1 - ((-y * y) / radiusPow))/(abs(y/4.0) + 1.0);
            col += texture2D(baseImg, texel) * neighborWeight;
        }
    }

    gl_FragColor = col;
}

Here is my Shader:

@EffectDependencies(FogOfWarShaderPeer.class)
@EffectRenderer(FogOfWarShader.EffectRenderer.class)
public class FogOfWarShader extends OneSamplerEffect {

    private static final int RADIUS = 16;

    public static final int HORIZONTAL = 0;
    public static final int VERTICAL = 1;

    private int axis;

    public FogOfWarShader() {
    }

    public int getAxis() {
        return this.axis;
    }

    public void setAxis(int axis) {
        this.axis = axis;
    }

    protected static class EffectRenderer implements IEffectRenderer {

        private FogOfWarShaderPeer shaderPeer;

        @Override
        public synchronized ImageData render(InternalEffect effect, FilterContext fctx, BaseTransform transform, Rectangle outputClip, RenderState rstate, ImageData... inputs) {
            if(shaderPeer == null) {
                shaderPeer = (FogOfWarShaderPeer)ShaderController.getPeerInstance(FogOfWarShaderPeer.class, fctx);
            }

            if(effect.getEffect() instanceof FogOfWarShader shader) {
                shader.setAxis(HORIZONTAL);
                ImageData resultPass1 = shaderPeer.filter(effect, rstate, transform, outputClip, inputs);
                shader.setAxis(VERTICAL);
                ImageData resultPass2 = shaderPeer.filter(effect, rstate, transform, outputClip, resultPass1);
                resultPass1.unref(); // Memory unalloc
                return resultPass2;
            }

            throw new RuntimeException("Invalid Effect Type");
        }

    }

    @Override
    public RenderState getRenderState(FilterContext fctx, BaseTransform transform, Rectangle outputClip, PrRenderInfo renderHelper, Effect defaultInput) {
        return new RenderState() {
            @Override
            public RenderState.EffectCoordinateSpace getEffectTransformSpace() {
                return EffectCoordinateSpace.CustomSpace;
            }

            @Override
            public BaseTransform getInputTransform(BaseTransform baseTransform) {
                return baseTransform;
            }

            @Override
            public BaseTransform getResultTransform(BaseTransform baseTransform) {
                return BaseTransform.IDENTITY_TRANSFORM;
            }

            @Override
            public Rectangle getInputClip(int i, Rectangle rectangle) {
                final BaseBounds inputBounds = FogOfWarShader.super.getInputBounds(0, BaseTransform.IDENTITY_TRANSFORM, defaultInput);
                final Rectangle untransformedClip = ShaderEffect.untransformClip(transform, outputClip);
                final BaseBounds bounds = FogOfWarShader.this.calcInfluenceBounds(inputBounds, new RectBounds(untransformedClip));
                transform.transform(bounds, bounds);
                bounds.roundOut();
                return new Rectangle((int) bounds.getMinX(), (int) bounds.getMinY(), (int) bounds.getWidth(), (int) bounds.getHeight());
            }
        };
    }

    @Override
    public BaseBounds getBounds(BaseBounds inputBounds) {
        return inputBounds.deriveWithNewBounds(this.calcBounds(inputBounds, inputBounds));
    }

    @Override
    public Rectangle getResultBounds(BaseTransform transform, Rectangle outputClip, ImageData... inputDatas) {
        return outputClip;
    }

    @Override
    public void writeDirtyRegions(BaseBounds inputBounds, DirtyRegionContainer drc) {
        for (int i = 0; i < drc.size(); i++) {
            final RectBounds region = drc.getDirtyRegion(i);
            if (region == null || region.isEmpty()) {
                break;
            }
            region.deriveWithNewBounds(this.calcBounds(inputBounds, region));
        }
    }

    private RectBounds calcBounds(BaseBounds inputBounds, BaseBounds region) {
        return new RectBounds(region.getMinX() - RADIUS, region.getMinY() - RADIUS, region.getMaxX() + RADIUS, region.getMaxY() + RADIUS);
    }

    private RectBounds calcInfluenceBounds(BaseBounds inputBounds, BaseBounds region) {
        final BaseBounds result = new RectBounds(region.getMinX() - RADIUS, region.getMinY() - RADIUS, region.getMaxX() + RADIUS, region.getMaxY() + RADIUS).deriveWithUnion(region);
        final BaseBounds totalRegion = this.calcBounds(inputBounds, inputBounds);
        result.intersectWith(totalRegion);
        return result.flattenInto(null);
    }

}

Here is my Peer:

@EffectPeer(value = "FogOfWar")
class FogOfWarShaderPeer extends ShaderEffectPeer<FogOfWarShader> {

    protected FogOfWarShaderPeer(ShaderEffectPeerConfig config) {
        super(config);
    }

    @Override
    protected ShaderDeclaration createShaderDeclaration() {
        final Map<String, Integer> samplers = Map.of("baseImg", 0);
        final Map<String, Integer> params = Map.of("resolution", 0, "viewport", 1, "axis", 2);
        return new ShaderDeclaration(samplers, params, getGLShader(), getD3DShader());
    }

    @Override
    protected void updateShader(JFXShader shader, FogOfWarShader effect) {
        shader.setConstant("resolution", (float) this.getDestBounds().width, (float) this.getDestBounds().height);
        final double scaleX = Math.hypot(this.getTransform().getMxx(), this.getTransform().getMyx());
        final double scaleY = Math.hypot(this.getTransform().getMxy(), this.getTransform().getMyy());
        shader.setConstant("viewport", (float)scaleX, (float) scaleY);
        shader.setConstant("axis", effect.getAxis());
    }

}

Are you able to find what I'm missing?

[Teragam]

I think you need to grow the bounds in getResultBounds as well:

Rectangle resultBounds = new Rectangle(outputClip);
resultBounds.grow(RADIUS, RADIUS);
return resultBounds;

By not overriding getResultBounds at all, this will be done automatically as the ShaderEffect calculates the required bounds from the already grown input bounds. In the effect peer, you might want to use the input bounds instead of the destination bounds. Specifically, you need to use the input native bounds as JavaFX uses texture pooling and might select a slightly larger input texture:

shader.setConstant("resolution", (float) this.getInputNativeBounds(0).width, (float) this.getInputNativeBounds(0).height);

You can also directly use the utility function drc.grow(RADIUS, RADIUS); as this will grow each dirty region by the specified amount without needing to iterate through each of them. It is necessary for the ZoomRadialBlur as the amount varies with the location of the region. If the reason for the special weighted blurring is to blur the underlying fog of war only outwards and prevent any leaks, then you could alternatively chain together the JavaFX GaussianBlur or BoxBlur with a shader that remaps every alpha value above 0.5 to 1 and fades back from 1 to 0 only between 0.5 and 0. This might give you more control of the blur more easily.

[DeathPhoenix22]

Hi Teragam,

For completeness, here is the same code fixed with your guidance:

GLSL:

varying vec2 texCoord0;

uniform sampler2D baseImg;
uniform vec2 resolution;
uniform int axis;

const float radius = 16.0;
const float radiusPow = 256.0;
const float weightCenter = 0.0876;

void main() {
    vec2 texel = texCoord0;
    vec4 col = vec4(0.0, 0.0, 0.0, 0.0);

    if(axis == 0) {
        float scaledPixel = 1.0 / resolution.x;
        texel.x -= radius * scaledPixel;
        for(float x = -radius ; x <= radius ; x++) {
            texel.x += scaledPixel;
            float neighborWeight = weightCenter * (1.1 - ((-x * x) / radiusPow))/(abs(x/4.0) + 1.0);
            col += texture2D(baseImg, texel) * neighborWeight;
        }
    }

    if(axis == 1) {
        float scaledPixel = 1.0 / resolution.y;
        texel.y -= radius * scaledPixel;
        for (float y = -radius ; y <= radius ; y++) {
            texel.y += scaledPixel;
            float neighborWeight = weightCenter * (1.1 - ((-y * y) / radiusPow))/(abs(y/4.0) + 1.0);
            col += texture2D(baseImg, texel) * neighborWeight;
        }
    }

    gl_FragColor = col;
}

Shader:

@EffectDependencies(FogOfWarShaderPeer.class)
@EffectRenderer(FogOfWarShader.EffectRenderer.class)
public class FogOfWarShader extends OneSamplerEffect {

    public static final int RADIUS = 16;

    public static final int HORIZONTAL = 0;
    public static final int VERTICAL = 1;

    private int axis;

    public FogOfWarShader() {
    }

    public int getAxis() {
        return this.axis;
    }

    public void setAxis(int axis) {
        this.axis = axis;
    }

    protected static class EffectRenderer implements IEffectRenderer {

        private FogOfWarShaderPeer shaderPeer;

        @Override
        public synchronized ImageData render(InternalEffect effect, FilterContext fctx, BaseTransform transform, Rectangle outputClip, RenderState rstate, ImageData... inputs) {
            if(shaderPeer == null) {
                shaderPeer = (FogOfWarShaderPeer)ShaderController.getPeerInstance(FogOfWarShaderPeer.class, fctx);
            }

            if(effect.getEffect() instanceof FogOfWarShader shader) {
                shader.setAxis(HORIZONTAL);
                ImageData resultPass1 = shaderPeer.filter(effect, rstate, transform, outputClip, inputs);
                shader.setAxis(VERTICAL);
                ImageData resultPass2 = shaderPeer.filter(effect, rstate, transform, outputClip, resultPass1);
                resultPass1.unref(); // Memory unalloc
                return resultPass2;
            }

            throw new RuntimeException("Invalid Effect Type");
        }

    }

    @Override
    public RenderState getRenderState(FilterContext fctx, BaseTransform transform, Rectangle outputClip, PrRenderInfo renderHelper, Effect defaultInput) {
        return new RenderState() {
            @Override
            public RenderState.EffectCoordinateSpace getEffectTransformSpace() {
                return EffectCoordinateSpace.CustomSpace;
            }

            @Override
            public BaseTransform getInputTransform(BaseTransform baseTransform) {
                return baseTransform;
            }

            @Override
            public BaseTransform getResultTransform(BaseTransform baseTransform) {
                return BaseTransform.IDENTITY_TRANSFORM;
            }

            @Override
            public Rectangle getInputClip(int i, Rectangle rectangle) {
                BaseBounds inputBounds = FogOfWarShader.super.getInputBounds(i, BaseTransform.IDENTITY_TRANSFORM, defaultInput);
                transform.transform(inputBounds, inputBounds);
                inputBounds.roundOut();
                return new Rectangle((int) inputBounds.getMinX(), (int) inputBounds.getMinY(), (int) inputBounds.getWidth(), (int) inputBounds.getHeight());
            }
        };
    }

    @Override
    public Rectangle getResultBounds(BaseTransform transform, Rectangle outputClip, ImageData... inputDatas) {
        Rectangle resultBounds = new Rectangle(outputClip);
        resultBounds.grow(RADIUS, RADIUS);
        return resultBounds;
    }

    @Override
    public void writeDirtyRegions(BaseBounds inputBounds, DirtyRegionContainer drc) {
        drc.grow(RADIUS, RADIUS);
    }
}

Peer:

@EffectPeer(value = "FogOfWar")
class FogOfWarShaderPeer extends ShaderEffectPeer<FogOfWarShader> {

    protected FogOfWarShaderPeer(ShaderEffectPeerConfig config) {
        super(config);
    }

    @Override
    protected ShaderDeclaration createShaderDeclaration() {
        final Map<String, Integer> samplers = Map.of("baseImg", 0);
        final Map<String, Integer> params = Map.of("resolution", 0, "axis", 1);
        return new ShaderDeclaration(samplers, params, getGLShader(), getD3DShader());
    }

    @Override
    protected void updateShader(JFXShader shader, FogOfWarShader effect) {
        shader.setConstant("resolution", (float) this.getInputNativeBounds(0).width, (float) this.getInputNativeBounds(0).height);
        shader.setConstant("axis", effect.getAxis());
    }

}

Thanks for your more than valuable advice. I'll probably change it eventually to support configurable radius, but I'll do the job for now.

I'm back to a 180 FPS ! Water effects will certainly be next ;)

[DeathPhoenix22]

The changes made in Develop are solving the Dirty Region Opts - This is a major extremely noticeable impact on performances !

I can't thank you enough for this change!