Specialize terrain nodes

[Zylann]

Currently, there are two terrain types:

• VoxelTerrain: paged terrain where all blocks are 16x16x16 no matter the distance. Supports both blocky and smooth voxels, although some special cases required to introduce a bit of complexity.

• VoxelLodTerrain: paged terrain managing multiple layers of blocks of varying levels of detail. It works only with smooth voxels, and blocky ones would not only add complexity but also not look good anyways.

Given the use cases for each type, I am tempted to change them:

• VoxelTerrain => VoxelTerrainBlocky: drop support for smooth voxels, specialize into blocky ones, eliminate special cases.

• VoxelLodTerrain => VoxelTerrainSmooth: keep supporting smooth only, and do it well.

This also has the advantage that it's easier to expose settings in the inspector, as they can be wildly different between blocky and smooth. One drawback is, VoxelLodTerrain currently doesn't support edition, mostly because I need to find a strategy to replicate edits on all LODs.

---

Edit: this may be part of a larger refactoring to come in the future. Main points being:

• Introduction of VoxelServer so we can efficiently organize resource usage with many voxel nodes, not just one
• Eventual specialization of the two current terrain nodes to simplify their use
• Possible introduction of new nodes
• Customizability of streaming behavior (paged or not, lodded or not) and exposition of network-friendly APIs
• Renamings

(How these requirments will be implemented may vary from the final design)

[TokisanGames]

Sounds like a good idea. They can have simpler, more focused code, and the LOD terrain is already 4-10x faster than the other.

Though, yes adding/removing terrain and infinite paging need to make their way into LOD before cutting smooth out of VoxelTerrain.

I don't really understand the code, but as a user it seems the map is built completely 8 times (or several times), one LOD at a time, starting at 0,0,0.

It would seem to me that one could edit the data source*, mark the block as dirty and the mesher would just remesh the data on whatever LOD is appropriate. No need to replicate the change to all LODs, instead LODs regenerate exactly as they are built now.

*How to "edit the data source?" I'm not sure how it's acquired, but...

If every time a block is remeshed, it's loaded from the stream, then perhaps there could be a stream-cacher which retains the edits and gives those if present, instead of querying the stream.

If the stream is not queried again and the data in VoxelBuffer is sufficient to remesh the block, then the voxelbuffer can be edited and the block marked dirty, forcing a remesh.

It seems to me that editing the mesh itself is the worst way to go, as we have to edit 8 different ones per block. You've already built algorithms that mesh 8 LODs based on source data. So it seems most efficient just to edit the data and remesh.

Also I think the mesher should mesh only one LOD per block. Say the first thing built is LOD0, 2 blocks around the player. Then LOD1, 3-4 blocks around. LOD2, 5-6 blocks around.... This would eliminate the current delay before LOD0 is built, and might speed up the whole process up to 8x more as only one LOD is built per block instead of 8.

What do you think about these ideas?

[Zylann]

I don't really understand the code, but as a user it seems the map is built completely 8 times (or several times), one LOD at a time, starting at 0,0,0.

This is due to the technique currently in use, which is a classic Octree that cannot subdivide a node if it does not fully have the data it needs. And conversely, 8 children cannot join if their parent data is unavailable. So in order to transition, it has to go through all intermediate layers. If it doesn't, you run the risk of opening a crack in the terrain (which happened a lot in my early experiments). Loading the world for the first time sounds like a bad case for this logic, but it makes sense when you move around (and in fact, that's what happens if you zoom very fast on Google Maps :D) I think it can be improved with a specialized implementation (current is generic), but I'm focused on saving at the moment. We could open an issue to discuss this.

It would seem to me that one could edit the data source*, mark the block as dirty and the mesher would just remesh the data on whatever LOD is appropriate. No need to replicate the change to all LODs, instead LODs regenerate exactly as they are built now.

The situation with LOD is that it currently works just like texture mipmaps, and loads them progressively. There really is as many sets of data as LODs, except each one spans a larger world distance. If you run an edit at a given LOD, data from other affected LODs must be updated too and then remeshed. Some of them will be already in memory, others won't. For those, I thought I could remember edits, and apply them as soon as blocks of that specific LOD are loaded (as if they were "part" of the generation of those never-explored LODs). Other strategies involve downscaling, which requires two levels to be loaded, and it works only downwards, so edition would only work within the constraints of LOD0 (so not suitable for landscaping in editor or large-scale explosions). Upscaling is even worse because a large-scale edit would not look as good from up close. I tried that in 2D and it even created seams. In any case, the more we can fit in memory, the faster this process will be, and that's where compression may help. So many things to do! We could open an issue about this as well.

I think I will likely do this refactoring on terrain nodes once I come up with something mergeable from the persistence branch.

[afonsolage]

Also, if I'm not mistaken, there are some extra work done by smooth terrain which isn't needed on blocky.

For instance, on this block of code, at voxel_terrain.cpp(763), both channels (type and sdf/isolevel) are copied into the buffer, which is gonna be sent to MeshUpdateProcessor, but the blocky mersher only uses type_channel, so the sdf/isolevel channel is useless here.

unsigned int channels_mask = (1 << VoxelBuffer::CHANNEL_TYPE) | (1 << VoxelBuffer::CHANNEL_ISOLEVEL);
_map->get_buffer_copy(_map->block_to_voxel(block_pos) - Vector3i(padding), **nbuffer, channels_mask);

By specializing VoxelTerrainBlocky it'll be able to copy only the needed channel.

[Zylann]

@afonsolage the copy is avoided by not having an SDF channel in the first place. If you don't want smooth terrain, there won't be one, so it won't be copied. By default, channels you don't touch will have a UNIFORM compression, which means their data will be defined by a single default value rather than a full memory block.

[afonsolage]

I see, indeed I misinterpreted the VoxelBuffer::fill_area function, which is called by VoxelMap::get_buffer_copy when get_block return a null block. After reading it more carefully and reading your explanation, it relays on default value and doesn't touch the channel. Amazing :+1:

[Zylann]

Another thing I've been thinking about, was to replace the terrain term with paged volume, as there could be fixed volume for non-paged voxel nodes... although it might sound less appealing.

[TokisanGames]

The end result is a mesh(instance), so maybe that is the final word.

Is paging something that can be turned on and off for a finite mesh, or will it be a separate object? Maybe the paged one inherits the non paged one.

Maybe VoxelBlockyMesh, VoxelSmoothMesh, and if needed Voxel(Blocky|Smooth)PagedMesh.

[Zylann]

Paging is not something that can easily be turned off, it's two entirely different systems. I would also not make paged inherit non-paged, but they might both have a common base.

[Zylann]

Update: don't wait too much for this one, I'm still figuring out design adjustments for those terrain nodes types, it may take a while. Some parts are embarrassingly similar between the two so they might at least have a common base. I'm also considering to have a concept of "voxel configuration" so that the same meshing stack can be used for different kinds of nodes (i.e paged terrain, fixed size debris, as some games may not necessarily have only one voxel node) and meshing can be centralized instead of being in every terrain node (VoxelServer might become a reality in the future), or at least introduce a job system because I'm worried about thread starvation.