Support for custom vertex attributes.

[Kneesnap]

I've got a program I've been working on which edits 3D environments from older video games. I've been planning to move away from JavaFX for a while because of its lack of shader customization & custom vertex attributes. (Using OpenGL terms here) However, I just stumbled across your project, and I think it might save me from moving on from JavaFX. The main thing I notice this to be missing is a way to add custom vertex attribute data, via ObservableFXArray types, just like the TriangleMesh class.

I wrote a really bad system which rendered the vertex coloring onto textures, so every unique polygon had its own texture in a 4096x4096 texture sheet. This worked perfect for a long time, but it's just not realistic to allow editing of the vertex color data with how slow that texture is to update. So this would be HUGE if support could be added for it.

[Birdasaur]

As always great work with this library. I agree that this would be an awesome feature. If someone were to make a vertex color shader I'd throw in support for vertex line width possibly but a concern might be how hard it might be to make that cross platform.

@Kneesnap I'm curious about your current implementation because it sounds like the same system we implemented in FXyz3D examples and derivative tools. I found I could get decent update performance... good enough to be mostly responsive to mouse interaction. (though it does depend on the data sizes)

[Teragam]

Hi! I dived into the JavaFX rendering pipeline again to see if custom vertex attributes might be possible, and I think I found a way of implementing this. I might focus on DirectX first, as it requires less work regarding the current architecture this library uses. If desired, I could release a snapshot version once the DirectX version is done to test it out.

[Kneesnap]

As always great work with this library. I agree that this would be an awesome feature. If someone were to make a vertex color shader I'd throw in support for vertex line width possibly but a concern might be how hard it might be to make that cross platform.

@Kneesnap I'm curious about your current implementation because it sounds like the same system we implemented in FXyz3D examples and derivative tools. I found I could get decent update performance... good enough to be mostly responsive to mouse interaction. (though it does depend on the data sizes)

Interesting. I must admit, I haven't actually profiled my system, since I don't have licenses for any good free ones atm that have IDE integration. However, I can quickly describe how the texture atlas generation worked. There's a BufferedImage representing the image which I edit, and when the texture corresponding to a given face is updated, only that portion of the texture sheet is re-drawn. Then, the image is re-converted to the JavaFX image (I suspect this is the slowdown), and re-applied to the material. It performs at a usable speed on powerful desktop computers, but it's not acceptable on low-end laptops, which I do wish to support.

If you'd like to see my current implementation, here's a link (or two): [https://github.com/Kneesnap/FrogLord/blob/master/src/net/highwayfrogs/editor/gui/texture/atlas/AtlasBuilderTextureSource.java#L92C1-L100C1](The class responsible for creating and updating the texture atlas image.) [https://github.com/Kneesnap/FrogLord/blob/master/src/net/highwayfrogs/editor/gui/mesh/DynamicMesh.java#L275C1-L292C6](The place the texture atlas is converted to a JavaFX image and applied to the material.)

Here's an example of one of the texture sheets to make it clear the scale of how large this image is.

Hi! I dived into the JavaFX rendering pipeline again to see if custom vertex attributes might be possible, and I think I found a way of implementing this. I might focus on DirectX first, as it requires less work regarding the current architecture this library uses. If desired, I could release a snapshot version once the DirectX version is done to test it out.

That'd be awesome! Thanks very much for your work on this.

[Birdasaur]

Hopefully @Teragam doesn't mind us having this sidebar conversation but... @Kneesnap I think I have some performance suggestions for you but first verify if I have the following points correct about your implementation:

1. I see your updateMaterial() method expects a BufferedImage, and the idea is that you've edited the BufferedImage upstream somewhere yes?
2. You're using and manipulating single large BufferedImage to map all the 3D meshes to it (to save on resources likely)
3. All the 3D meshes are getting wrapped with a material that uses this base BufferedImage and I'm guessing you're telling each how to map into that image using U,V coordinates?

Whether all the above are correct or not, you're taking a couple performance hits due to recreating and reapplying a phongmaterial each time you update your material image. The first hit is that you are converting from a BufferedImage to an FX compatible Image. Second is that you have to clear and create a new material object each time (which of course means copy time into the GPU in addtion to the CPU necessary to process everything.

Why can't you make a single Material object based on a single "WritableImage"? That way you would be able to skip both the FX conversion step AND creating a new Material. To update the image, the easiest way would be to simply use the writableImage PixelWriter to update the image in place.

Here is an example I use to draw a crosshair that updates when the mouse moves over a surface: `
public void illuminateCrosshair(Point3D center) { if (null == diffusePaintImage) return; int x = (int) (center.getX() / surfScale); int z = (int) (center.getZ() / surfScale); //Image Y is projected into Z

    PixelWriter pw = diffusePaintImage.getPixelWriter();
    for (int i = 0; i < diffusePaintImage.getWidth(); i++)
        pw.setColor(i, z, Color.WHITE);
    for (int i = 0; i < diffusePaintImage.getHeight(); i++)
        pw.setColor(x, i, Color.WHITE);
}

`

Where the diffusePaintImage is a reference to the original WritableImage that I used for my PhongMaterial. Technically what I'm doing is managing two 3D surface meshes... one is colorized the original color set. The second is an exact copy but slighly above it in the Y axis and the entire diffuse color is Transparent. Whenever the mouse moves I clear all the pixesl (using pixelwriter) and then call this illuminateCrosshair method to draw on the material. The resulting effect is a crosshair moving around with the mouse.

And for what its worth... I am using probably the slowest way to write to the pixels by doing it one at a time in for loops. PixelWriter supports several bulk setPixels() methods but I"m too lazy to make it work for my needs.

[Kneesnap]

@Birdasaur Ty for the tips, I'll try these and see how it goes.

Wanting to touch base about a few things you mentioned though 1) Correct 2) Close. This texture sheet is for the terrain mesh, and contains all of the textures for the terrain mesh. The entity meshes have separate texture sheets but they are extremely small and rarely use shading. 3) Each unique mesh has its own Material, but the only major problem I have is with the full terrain mesh.

So for your suggestions, that's not a bad idea. Frankly, I didn't realize WritableImage existed. I'm optimistic that it has better performance to update than converting a full BufferedImage. Will try it for sure. For clarity though, when you talk about creating a new material each time, I'm not explicitly doing that. Is JavaFX implicitly creating a new Material every time I change the diffuseMap image? If so, then I follow you, otherwise I think we have a misunderstanding.

[Birdasaur]

Ah OK. I misunderstood part of your code... I thought you were creating a new material. So the good news is you still might be able to improve your performance by pixel writing directly to a WritableImage since under the hood it wont have to copy the full byte array of the new image object you edited.

[Teragam]

Custom vertex attributes are now supported for DirectX. Regarding the speed improvement, JavaFX unfortunately does not support partial updates of vertex buffers (it uses partial updates internally but completely rebuilds the mesh for the GPU anyway). For simple geometry, this should still be fast enough, and I might add support for partial updates in the future.

A different approach for vertex shading I can think of is using the texture directly in the vertex shader. Below is a demo I created where an image is used in the vertex shader to offset the vertex position and to set a color that is used by the fragment shader. The fragment shader itself does not sample any textures.

https://github.com/Teragam/JFXShader/assets/67922585/d8a689a9-988c-4684-8958-486f85dedf35

The image is updated via a simple JavaFX Canvas, which is very fast because the image is modified directly on the GPU. The snapshot method can be used to "download" the image from the GPU. In the demo, I linked the internal texture of the Canvas to the texture used by the vertex shader. The texture itself therefore stays on the GPU, resulting in good performance. The demo uses 10,000 vertices, but I get decent performance even for 4,000,000 vertices because the vertex buffer does not need to be altered. If desired, I could create a Wiki entry with the code of this demo in addition to how to use custom vertex attributes.

@Kneesnap I saw your project uses Java 8 with the packaged JavaFX version. Unfortunately, the library needs Java 11 or higher with JavaFX 18 or higher. Supporting the version packaged with Java 8 requires extensive work on many parts of the library, which I don't have any plans for at the moment.

[Kneesnap]

That's okay, I've been planning on updating it for a while and just haven't done it yet.

On Wed, Mar 27, 2024, 7:34 AM Teragam @.***> wrote:

Custom vertex attributes are now supported for DirectX. Regarding the speed improvement, JavaFX unfortunately does not support partial updates of vertex buffers (it uses partial updates internally but completely rebuilds the mesh for the GPU anyway). For simple geometry, this should still be fast enough, and I might add support for partial updates in the future.

A different approach for vertex shading I can think of is using the texture directly in the vertex shader. Below is a demo I created where an image is used in the vertex shader to offset the vertex position and to set a color that is used by the fragment shader. The fragment shader itself does not sample any textures.

https://github.com/Teragam/JFXShader/assets/67922585/d8a689a9-988c-4684-8958-486f85dedf35

The image is updated via a simple JavaFX Canvas, which is very fast because the image is modified directly on the GPU. The snapshot method can be used to "download" the image from the GPU. In the demo, I linked the internal texture of the Canvas to the texture used by the vertex shader. The texture itself therefore stays on the GPU, resulting in good performance. The demo uses 10,000 vertices, but I get decent performance even for 4,000,000 vertices because the vertex buffer does not need to be altered. If desired, I could create a Wiki entry with the code of this demo in addition to how to use custom vertex attributes.

@Kneesnap https://github.com/Kneesnap I saw your project uses Java 8 with the packaged JavaFX version. Unfortunately, the library needs Java 11 or higher with JavaFX 18 or higher. Supporting the version packaged with Java 8 requires extensive work on many parts of the library, which I don't have any plans for at the moment.

— Reply to this email directly, view it on GitHub https://github.com/Teragam/JFXShader/issues/9#issuecomment-2022928745, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSCIYOXNBZZ5TLZYK3EFM3Y2LDE7AVCNFSM6AAAAABEZB6AZOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRSHEZDQNZUGU . You are receiving this because you were mentioned.Message ID: @.***>

[DeathPhoenix22]

It's sad that JavaFX doesn't allow partial updates, it would be good to have the support for it simply to avoid rewriting shaders with vert