New blending sorting methods break spine models within 3D environment

[Kagamma]

The title says it all. With the new way CGE deals with blending sort, spine models simply break when used within 3D environment. This is because the sorting of shapes is now based on world (view?) coordinates instead of local coordinates as in the old scene-based sort.

I have gone through OnCustomShapeSort, but it seems like it won't work if dynamic batching is turned on because ParentScene may be nil, making it uncertain whether a particular shape comes from the spine scene.

[michaliskambi]

You should be able to make the sorting work for Spine layers even in 3D, even if you rotate the Spine models using arbitrary 3D rotations.

(Note: If you don't use arbitrary 3D rotations, then just Viewport.BlendingSort := sort2D should work, even if camera is perspective, looking at some angle etc.)

If you rotate the Spine models using arbitrary 3D rotations, OnCustomShapeSort to sort based on local coordinate should work, as you say. It gets shapes in scene coordinates, so you can sort just like before changes on https://castle-engine.io/wp/2023/06/30/big-renderer-improvements-correct-and-automatic-blending-sorting-more-powerful-batching-now-cross-scene-easier-and-more-reliable-occlusion-culling-and-occlusion-sorting/ .

Indeed you cannot use batching in this case. When batching connects shapes, potentially from different scenes, indeed ParentScene is not available -- because there is no single parent scene for the merged shape. Moreover the transformations are flattened, so the original local coordinates are also lost -- the merged shapes had to be brought into one, common, coordinate system.

So from my perspective:

• you can get the sorting you need with OnCustomShapeSort
• you cannot use batching anymore in this case, that's indeed a problem.

We could introduce an option for batching to not merge things across scenes, and to not flatten the transformations... but it would be hard to maintain, admittedly, as it would be an exception in some code places.

Can you show your exact use-case? Let's make sure you cannot use Viewport.BlendingSort := sort2D, and that you need batching.

[Kagamma]

Can you show your exact use-case? Let's make sure you cannot use Viewport.BlendingSort := sort2D, and that you need batching.

It's a dungeon-crawler type of game where all the NPCs in game are done using Spine2D. And since there will be a lot of NPCs in one place (a village), batching would be great. Screenshots below are from the built-in map editor of the game.

[michaliskambi]

Are the Spine models maybe always parallel to the screen? That is, do you always look straight at them, or can you see Spine models at an angle? Can you show more screenshots -- the answer to this is not obvious to me from your screenshots above :)

• If yes, then we could introduce another sorting method that sorts along the camera direction (so a bit like sort2D, but sort2D assumes camera direction is -Z). It would project each shape on the camera direction vector, and choose closest. In case when camera direction = -Z, it should behave exactly like existingsort2D.

  Bonus: this means you can keep cross-scene batching.

• If no, then we'll have no other choice but to introduce a different batching method that is limited to single scene, and behaves as before https://castle-engine.io/wp/2023/06/30/big-renderer-improvements-correct-and-automatic-blending-sorting-more-powerful-batching-now-cross-scene-easier-and-more-reliable-occlusion-culling-and-occlusion-sorting/ . I'll need to think about it -- it is somewhat uneasy (it means maintaining special batching mode for such purposes).

[Kagamma]

The first solution should be enough to solve the issue. There's no look up / look down and Spine models always face the camera.

[michaliskambi]

On branch https://github.com/castle-engine/castle-engine/tree/blending-sort-direction (not yet merged to master, but I'll merge tomorrow if tests pass):

I improved our blending sorting and billboards.

1. 3D sorting algorithms (sort3D, sort3DOrigin, sort3DGround) now order by projecting shape point on the camera direction (instead of measuring the distance from shape point to camera position).

2. By default TCastleBehavior adjusts to camera direction, not vector from camera to billboard origin. (One can get old behavior by setting MatchCameraDirection = false.)

   The end result of AD 1 and AD 2 is that if you use TCastleScene with TCastleBillboard, it will have perfect sorting even for extremely thin layers, like when loaded with Spine. Out of the box, i.e. default blending sorting and default billboard behavior are good.

   This works as long as AxisOfRotation is zero or equal to camera up. The sorting is 100% reliably correct in this case.

3. Additionally, I fixed when billboards' orientation is set.

   It is now done without any delay, and will work even when scene with billboards is used multiple times (e.g by TCastleTransformReference).

Note that your case, looking at your last screenshot, seems to still need a custom OnCustomShapeSort implementation. Because it seems you look at your Spine models at an angle (they are not like billboards). But it seems they are aligned along one of main axis (X or Y or Z) so you could implement sorting OnCustomShapeSort similar to our default 2D sorting implementation, just adjust the axis?

Let me know if you want me to experiment with this. I'll need a testcase then, to exactly reproduce your situation (possible camera views and Spine models arrangement), to make sure I implement something useful for your use-case :)

If not, I'll assume that current state (on blending-sort-direction branch, tomorrow in master) is enough, and you can implement custom OnCustomShapeSort to cover your needs. And then you can have both blending sorting and cross-scene batching and enjoy everything :)

[Kagamma]

Ah yes my "billboard" code was implemented way before TCastleBillboard was a thing, and it works simply by setting model's look direction to camera position. I will switch to TCastleBillboard instead.

[michaliskambi]

OK, everything on CGE side related to "better billboards and sorting options" is now done and merged to master branch :)

Jenkins is now busy building it, and if nothing will explode, I'll make a news post about it later this weekend.

For now, I'd say this is closed from CGE side -- we support various cases of using blending + batching + billboarded Spine models in CGE now. If something isn't covered, hopefully OnCustomShapeSort can account for it. I will close this issue, assuming this is satisfactory for you --- but please reopen if the solutions outlined above are not good enough :) Thank you!

I'm attaching the contents of news post (to be published later this weekend), so that you know what to expect from the code already in master:

[gallery type="rectangular" link="file" columns="1" ids="4420,4419"]

We've done a number of improvements to blending and billboards, to support more use-cases, and to make some cases "just work out-of-the-box".

Before we start, just download latest version of Castle Game Engine and open a new example examples/viewport_and_scenes/billboards_blending_in_3d. It will illustrate most features mentioned in this post clearly :)

New features:

1. Existing 3D sorting algorithms ([cgeref id=sort3D], [cgeref id=sort3DOrigin], [cgeref id=sort3DGround]) now order by projecting shape point on the camera direction (instead of measuring the distance from shape point to camera position).

2. By default [cgeref id=TCastleBillboard] adjusts to camera direction, not vector from camera to billboard origin. (One can get old behavior by setting [cgeref id=TCastleBillboard.MatchCameraDirection] = false.)

   The end result of these 2 points is that if you use [cgeref id=TCastleScene] with [cgeref id=TCastleBillboard], it will have perfect sorting even for extremely thin layers, like when loaded from Spine. Out of the box, i.e. default blending sorting and default billboard behavior are good.

   This works as long as [cgeref id=TCastleBillboard.AxisOfRotation] is zero or equal to camera up. The sorting is 100% reliably correct in this case.

3. Moreover, if [cgeref id=TCastleBillboard.AxisOfRotation] is +Y (default), then all you have to do is just to flip [cgeref id=TCastleViewport.BlendingSort] to [cgeref id=sort3DVerticalBillboards]. [cgeref id=sort3DVerticalBillboards] is a new option that sorts perfectly for billboards rotated around Y.

   Blending in manual was extended to mention the new option.

4. The [cgeref id=TCastleBillboard] transformation algorithm was also improved. It is now faster, and it is applied without any delay (you will never observe artifacts caused by potential-1-frame-delay between changing camera and transforming billboards to account for it).

   The [cgeref id=TCastleBillboard] transformation also accounts now for billboards instantiated many times. Whether you use TCastleTransformReference or Multiple viewports to display one world, all instances of the billboard will be now correctly oriented toward the camera.

5. In both new blending methods and billboard algorithms, I'm proud that I actually achieved a bit of code simplifications and optimizations. It's nice when the better result is also just internally simpler, and the calculations involved are also simpler. E.g. sorting methods no longer transform 8 points of a box (they only transform 1). E.g. billboards get camera vectors in world space without any extra calculation (and these camera vectors correspond to current viewport).