Apply terrain Generation only to a masked region of terrain (for multi-biome / multi type terrains)

Is your feature request related to a problem? Please describe. The terrains built-in generator is phenomenal at adding a natural looking landscape that is difficult, time consuming and often subpar to hand paint. The terrain generator you've included can be used to create hills, marshes, valleys and mountains but it affects the entire terrain all at once. What if the terrain has a valley, with long meandering hills, with a harsh swampy relief and mountains in the corner? Currently you can only generate one of those three and the rest you have to hand sculpt once more. What if we could preliminarily select / paint a mask a region of a terrain where we would like the generation to take place and the generator could be told to only affect those parts of the terrain?

Describe the solution you'd like Add a new tool called "Mask Terrain Region" to the tool bar 1) Select that tool, and start painting on the terrain. What is being painted is essentially a mask (highlighted in bright color) The green mask could be made only visible when you chose the mask painting tool. If any other tool is selected, no mask is visible.

2) Press Generate and click on a checkbox that only applies to the masked region. Usually the generator preview window entirely displays what the new terrain shape will look like. Perhaps if this option is selected, the plugin can display the current height map being used and merges on top of it the new terrain you're generating over the masked region.

3) Play around with generation functions. 4) Press apply, the generated shape is only applied to the masked part of the terrain.

Additional context This method of only painting on the masked region is borrowed from Blender's sculpting tool. You can paint a mask on a 3D shape, and only the geometry of that mask, will react to any further sculpting.

Now this may produce a seam between the generated terrains but smoothing the seams /transitions between them should be trivial and can be done by hand.

Thoughts:

The tool has no effect on its own, or any of the other tools. If it only serves the generator (which is a separate tool), it seems like poor discoverability for other people. It could be moved to the generator window somehow (then you'd paint a mask there instead, though it's more work), or other painting tools could have "mask only" checkboxes as well, but I don't like this because it is feature creep and wasn't the original need.
Painting directly on the terrain supposes that there is a terrain texture to store that data. But it's editor only for a very specific use case. Excluding individual resources from exporting in Godot is not great (rather than just being in the asset, it has to be input by file path in every export preset and also has to be kept in sync manually if the asset gets moved...). At the very least, this texture would not be saved.
The generator remains a global tool. Even if you paint a mask, it will still run on the whole terrain, just getting interpolated by the mask, and actually writing into the heightmap and normalmap. It will only happen to write the same values in masked out regions. Maybe you already knew that, but I'm mentionning it just in case.
Using that tool requires to visualize the mask. For doing that, either the shader of the terrain has to be temporarily overriden with the lookdev shader (showing it black and red for masked regions), or the editor should use a decal to project it on the entire terrain.
That's only one mask. Maybe you also assumed that: once you're done working with a region and work on another, the mask you painted earlier will be lost as you erase and paint elsewhere. There could be options to save/load but that's feature creep again and it would be affected by the exporting issue too. It's probably not an issue to have one mask only? (like a selection).
Adding an extra tool is making the toolbar even larger. I'm thinking perhaps we should move it to the side at this point, because on smaller monitors space is lacking severely, and I dont like the underline Godot adds under them.
The generator is getting more and more features, the window is getting taller and I don't know what the threshold for "stop" is going to be. In other issues I mentionned the fact that if this continues, it would have to be rethought entirely into something more modular. But currently I don't have time for large reworks like this.

The terrains built-in generator is phenomenal at adding a natural looking landscape that is difficult, time consuming and often subpar to hand paint

The brush tools have the same operators though, so everything the generator does, brushes can. Maybe the only one missing is a noise brush, which can be added quite quickly, check this out:

https://zylannprods.fr/dl/godot/hterrain/noise_brush_prototype.mp4 (here hardcoded to a variant of ridged noise, currently not available in the generator but is a simple tweak to perlin noise that "looks" eroded, without actually using costly erosion)

This is different from just picking a brush shape, because noise is applied in world space so results are continuous and unique. The only issue is how to customize that noise; I'm not comfortable with the idea of adding many more controls into the brush panel, ideally I could plug in FastNoiseLite which can be edited in the inspector, however Godot's editing behavior switches out the entire terrain edition just in order to edit a FastNoiseLite resource (so can't customize noise while keeping the terrain editor active) and it's also not possible to pop a second inspector in a window without a lot of work replicating every property (would need https://github.com/godotengine/godot-proposals/issues/7010#issuecomment-1575105505). Something even crazier would be an ability to directly "paint the generator" as a tool. But that's even more work than the mask because multiple passes are involved, and would likely be very slow given the limitations of Godot. Or the noise brush could take "some" of its parameters from the generator settings, not sure. Would you still prefer painting a mask and switching to another window to apply the existing generator tool?

The tool has no effect on its own, or any of the other tools. If it only serves the generator (which is a separate tool), it seems like poor discoverability for other people. It could be moved to the generator window somehow (then you'd paint a mask there instead, though it's more work), or other painting tools could have "mask only" checkboxes as well, but I don't like this because it is feature creep and wasn't the original need.

Borrowing from the way sculpting tools handle it in blender, when you're in mask painting mode, you see the mask on top of the mesh you're working with. It's partially transparent but it's not in a separate window.

https://github.com/Zylann/godot_heightmap_plugin/assets/52581279/f62d447f-52e3-4486-ba60-eb2d575df2d2

Just like it wouldn't make sense to have the terrain painting be done in a separate window and is instead done directly in the editor, masking should also be done in the editor. Pair that with the fact that if the mask is to be painted within the generator panel, you probably will be painting a mask around some existing landmarks like buildings, props, trees, structures.

So then the renderer within would have to also draw the rest of the game word. The renderer camera is also quite limited, and does not allow to pan the camera. If you're masking out a small segment near a prop or building, there's no way to focus on that.

https://youtu.be/Dg0MZFZcjqI

Painting directly on the terrain supposes that there is a terrain texture to store that data. But it's editor only for a very specific use case. Excluding individual resources from exporting in Godot is not great (rather than just being in the asset, it has to be input by file path in every export preset and also has to be kept in sync manually if the asset gets moved...). At the very least, this texture would not be saved.

Yep, that's what I figured it would be like. This mask is really a temporary resource. It's there to do one job and be discarded a the end. Maybe it can be made persistent until Godot closes so you can try reusing the same mask between multiple generations. When you hit generate, the tool takes the original terrain height map, then uses this mask to interpolate the new terrain heightmap and final result gets baked into the current terrain height map.

The generator remains a global tool. Even if you paint a mask, it will still run on the whole terrain, just getting interpolated by the mask, and actually writing into the heightmap and normalmap. It will only happen to write the same values in masked out regions. Maybe you already knew that, but I'm mentionning it just in case.

Yeah that sounds appropriate to me. I don't see a problem with this.

Using that tool requires to visualize the mask. For doing that, either the shader of the terrain has to be temporarily overriden with the lookdev shader (showing it black and red for masked regions), or the editor should use a decal to project it on the entire terrain.

Maybe while you have the masking tool equipped, the mask texture gets additively (or another blending method) blended with the rest of the terrain texture. It's only temporary, right by the end of the shader before the total color data enters the color output of the fragment shader. I assume you have a separate shader for editing and for in-game rendering?

Adding an extra tool is making the toolbar even larger. I'm thinking perhaps we should move it to the side at this point, because on smaller monitors space is lacking severely, and I dont like the underline Godot adds under them.

I meeeean... we're just talking about a itty bitty paint brush icon for the tool bar. Is it really that much more? It goes away when you switch from the terrain node one way or another. Anything regarding masking tools you could place within the paint panel at the bottom. Like Clear mask or invert mask And that is alrady placed in the tools sub panels where there's still plenty of real estate.

Just like it wouldn't make sense to have the terrain painting be done in a separate window and is instead done directly in the editor, masking should also be done in the editor

The reason I brough this up is because with the proposed use case, masking would have no purpose outside masking the generator tool. I was also thinking of 2D painting on a separate tab instead of the 3D preview. Despite that, I do agree it's better to have it directly on the terrain. So I'm thinking of this like a "smooth selection" tool of an image editor, and it might get an icon like that instead of a shovel. In that sense, it could be used by more tools in the future, but doesn't have to be right now.

Maybe while you have the masking tool equipped, the mask texture gets additively (or another blending method) blended with the rest of the terrain texture. It's only temporary, right by the end of the shader before the total color data enters the color output of the fragment shader. I assume you have a separate shader for editing and for in-game rendering?

Code has to be written to do it, temporary or not. "Appending code at the end of the shader" would work, but it is not practical. There are many different shaders, not just one. They also don't have multiple versions for editor and game (which would further multiply the amount to maintain), and when users create their own shaders I can't force them to do so either. This is why the two options I mentionned were either the lookdev shader, or a giant decal, because they don't require to make any change to existing shaders. The decal approach would end up projecting on props though, I dont know if there is a way to limit that.

I meeeean... we're just talking about a itty bitty paint brush icon for the tool bar. Is it really that much more? It goes away when you switch from the terrain node one way or another.

You have a large screen, you don't realize what I've seen on other user's screens xD It's already taking more space than the Controls toolbar, which was a problem when it was added during Godot 4 development and they shrunk it a bit since. It might go away selecting something else but it could still break the editor's layout when selected. I would like to add the noise brush too eventually.

That said, I did a quick test of putting the toolbar on the left. It looked good, sadly Godot's left-side container pushes the bottom container slightly to the right, and there is nothing I can do against that... so I thought of more layout changes to optimize space, but that might be for another time.

The reason I brough this up is because with the proposed use case, masking would have no purpose outside masking the generator tool. I was also thinking of 2D painting on a separate tab instead of the 3D preview. Despite that, I do agree it's better to have it directly on the terrain. So I'm thinking of this like a "smooth selection" tool of an image editor, and it might get an icon like that instead of a shovel. In that sense, it could be used by more tools in the future, but doesn't have to be right now.

Yeah one use case is painting a mask and then filling that masked region with a choice of foliage.

because they don't require to make any change to existing shaders. The decal approach would end up projecting on props though, I dont know if there is a way to limit that.

Yeah that's what I figured the decal would do.

I think that more or less covers the general idea of the feature. Also in the Youtube video I posted (video was over 10mb in size for github) I mentioned that a camera panning within the editor (maybe even WASD controls like you got in Godot editor ) might be a necessity in order to be able to assess the masked terrain that's not strictly in the middle. Panning should really be enough I think. Don't forget a return to center option then.

Zylann / godot_heightmap_plugin

Apply terrain Generation only to a masked region of terrain (for multi-biome / multi type terrains) #397