Wind Animation on Vegetation

[lijianwen13]

Summary:

This proposal outlines how to add an Environment gem to provide wind settings and wind animation implementation for vegetation in EnhancedPBR materialtype in O3DE.

What is the relevance of this feature?

Wind animation for vegetation is a kind of important environment effects for natural scenes. Vegetation in O3DE now has always been mainly static. Although we could use PhysX components to provide force region to simulate the wind animation, but the performance is relatively poor for large-area plants. We need a GPU-based implementation to animate the wind effect.

To solve the problem that our scenes can have thousands of different vegetations, we push the envelope further by making vegetation react to the global wind source, and we bend not only the plant but also the leaves, in detail, with all computations procedurally and efficiently done on the vertex shader.

Feature design description:

This design is mainly divided into two parts: Environment gem and wind animation implementation in EnhancedPBR materialtype.

The gem provide necessary global wind settings, and the response happens in the material. An example is given in the following diagram, where we show the relationship. We obtain parameters from the gem, and send them to the SceneSrg. When rendering, material could get all parameters from the SceneSrg.

Design of the Environment gem

The gem is placed on the Level entity. There is a component named HWEnvironment Settings, as the picture shows:

Now the wind settings in the component include wind switch, wind vector, different wind modes, etc. We send these parameters into SceneSrg, thus material shaders could get them. The following diagram shows the editor of this gem.There are two modes provided by the component:

• Static Wind: The wind vector is a constant value, as the picture shows:

• Breezy Wind: The wind vector could be changed following the time change by a sine function. And the breezy settings could affect the value of the sine function. In the breezy state, the meadow wave effect can be turned on. The meadow wave effect is only for large areas of grass by sampling from a wave texture. At this time, a additional wave texture needs to be set, as the picture shows:

Here are parameters descriptions in the component:

Parameter	Description
Enable Wind	Click the button to enable the wind.
Wind Vector	Set the direction and strength of the wind.
Wind Mode	Set the wind mode, including static and breezy modes.
Breezy Frequency (1/sec)	Set the frequency of the breezy sine function, the unit is 1/second.
Breezy Life Time (min)	Set the duration of the breezy sine function in minutes, if set to "-1.0", there is no limit.
Breezy Strength (m/s)	Set the strength of the breezy sine function in meters per second.
Meadow Wave Settings	Enable Meadow Wave: Click the button to turn on the effect of generating meadow wave based on plant positions and the wave texture. Wave Texture Scale: Set the scaling factor for wave texture sample values. Wave Texture: Specify the sample texture for the grass waves.

Now the gem only supports global wind settings, but could extend to add more environment settings. Any other component which responses to this could get the wind related parameters. We hope that other components that need to use the wind settings can uniformly obtain them from here.

Besides the wind setting, we wish that any other global common environment settings could be included in this gem.

Design of the wind animation for vegetation in EnhancedPBR materialtype

Wind animation is basically a kind of vertex animation implemented in vertex shader.

There are two types for wind animation: Main Bending and Detail Bending. The Main Bending is an overall bending of the vegetation which refers to the overall offset of all vertices on the mesh controlled by the wind vector and object mass. The Detail Bending is a localized bending for trunks or leaves controlled by wind vector and another detail sources, like noise texture or vertex color attribute.

We generate the main bending by using the xy components of the wind vector, which gives us the wind direction and its strength, using the vegetation mesh height as a scale to apply a directional vertex deformation. Note that care must be taken to limit the amount of deformation. Otherwise, the results will not look believable.

Main Bending panel in the material editor is shown as following picture:

The Object Mass can affect the overall bending range, the greater the mass, the smaller the overall bending range. In addition, when opening Perpendicular Bending, the force in the perpendicular direction can also be added to make the plants bend along the vertical direction of the wind direction, enriching the diversity of bending.

To achieve the meadow wave effect, we need to enable the Enable Meadow Wave in the wind settings of HWEnvironment Setting component and enable the Meadow Wave Bending in the Wind Animation material property. We only need to perform this operation on the vegetation that we want to have the meadow wave animation. If not desired, just turn off Meadow Wave Bending.

For detail bending, we approach things in a similar fashion, but in this case only wind strength is taken into account. The detail bending of vegetation refers to the fine adjustment of the bending of vegetation branches or leaves through Detail Source. There are two types of detail source, Noise Texture and Vertex Color.

The parameter panels are shown by the following pictures when switching Detail Source:

When the detail source is noise texture, the bending of each vertex in the model is controlled by a certain sampling point in the texture. The sampling point is related to the spatial position of the vertex in the model and changes with time, so it can produce the effect of the vegetation swinging over time. The noise texture needs to be created and specified by the user. The gem has built-in commonly used noise textures by default, such as HWEnvironment\Assets\Textures\NoiseTextures\perlinNoise_sum.tif, as shown in the following figure:

When the detail source is vertex color, the bending of each vertex in the model is controlled by the vertex color attribute and the control function. The control function is built-in and cannot be changed temporarily. Vertex colors are stored in attributes of each vertex. Artists paint one RGB color per-vertex using a common DCC tool. This color gives us extra information about the detail bending determining the phase and amplitude scaling when sampling from the control function. Taking a leaf as an example, the red channel is used for the stiffness of leaves' edges, the green channel for per-leaf phase variation, and the blue channel for the overall stiffness of the leaves. The red, green and blue components of the vertex color in the leaf are displayed in turn from left to right, as the following figure shows:

Different parameter settings occur in the case of two different detail sources. Also, there are several common parameters both for main bending and detail bending, like common bending frequency and scale. The diagram below shows all the parameters and corresponding descriptions in the wind animation material property:

Parameter	Description
Enable	Click the button to enable wind animation.
Main Bending	Use main bending to simulate animation effects.
Object Mass	Set the object mass, only applied to main bending.
Perpendicular Bending	Animate with perpendicular bending, in addition to the main bending.
Perpendicular Force	Set the magnitude of the perpendicular bending force.
Meadow Wave Bending	Use meadow waves to animate the effect, in addition to the main bending.
Meadow Wave Shift	Set the sample offset for the wave texture.
Detail Bending	Use detail bending to simulate animation effects.
Breezy Effect	Use the breezy wind to affect detail bending animations.
Detail Source	Indicate how to obtain detail input. There are two input sources, namely noise texture and vertex color.
Noise Texture	Specify to obtain the noise value from a gray-scale texture.
Noise Texture Scale	Set the scaling factor for the sample value from the noise texture.
Start Height	Set the start height of the detail bending, vertices under the start height would not be animated.
Branch Amplitude	Set the up and down bending range of the branches or the entire leaf trunk.
Edge Amplitude	Set the up and down bending range of the leaf edge.
Inverse Blue Channel	Inverse the green channel value of vertex color, some special assets may need this operation.
Common Bending Frequency	Sets the common bending frequency, both main bending and detail bending are affected.
Common Bending Scale	Set the common bending scale, both the main bending and the detail bending are affected.

Technical design description:

The HWEnvironment gem is just developed by following the gem framework in O3DE. The UML class diagram of the gem is shown in the following picture:

classDiagram
    class HWEnvironmentComponent {
        +Reflect(AZ::ReflectContext*)$ void
    }

    class HWEnvironmentRequestBus {
        <<Interface>>
        +GetHWEnvironmentParams()*
        +SetHWEnvironmentParams()*
    }
    class HWEnvironmentFeatureProcessorInterface {
        <<Interface>>
        +GetSettingsInterface(AZ::EntityId)* HWEnvironmentSettingsInterface
        +GetOrCreateSettingsInterface(AZ::EntityId)* HWEnvironmentSettingsInterface
        +OnSettinigsChanged()* void
    }
    class HWEnvironmentSettingsInterface {
        <<Interface>>
        +GetHWEnvironmentParams()*
        +SetHWEnvironmentParams()*
        +OnConfigurationChanged()* void
    }

    class EditorHWEnvironmentComponent {
        -HWEnvironmentComponentController m_controller
        -bool m_boolForCheckButton
        +Reflect(AZ::ReflectContext*)$ void
        +OnConfigurationChanged() u32
    }
    class HWEnvironmentComponentConfig {
        +m_HWEnvironmentParams
        +bool m_cfgCheckBox
        +bool m_randomCheckBox
        +AZ::Data::Asset m_breezyTextureAsset
        +Reflect(AZ::ReflectContext*)$ void
        +CopySettingsFrom(HWEnvironmentSettingsInterface*) void
        +CopySettingsTo(HWEnvironmentSettingsInterface*) void
        +UpdateEnableFlags() void
        +WindModeIsStatic() bool
        +WindModeIsBreezy() bool
        +CheckBoxIsCfgAndRandom() bool
    }

    class HWEnvironmentComponentController {
        -HWEnvironmentSettingsInterface* m_ttSettingsInterface
        -HWEnvironmentComponentConfig m_configuration
        -AZ::Data::Asset m_breezyTextureAssetForEditor
        -AZ::EntityId m_entityId
        +GetHWEnvironmentParams()
        +SetHWEnvironmentParams()
        +Reflect(AZ::ReflectContext*)$ void
        +SetConfiguration(const HWEnvironmentComponentConfig&) void
        +GetConfiguration() HWEnvironmentComponentConfig
        +OnAssetReady(AZ::Data::Asset) void
        +OnAssetReloaded(AZ::Data::Asset) void
        +OnAssetError(AZ::Data::Asset) void
        +LoadImage(AZ::Data::Asset&) void
        +UpdateWithAsset(AZ::Data::Asset) void
        +IsAssetValid(AZ::Data::Asset&) void
        +OnBreezyTextureAssetChanged() void
        +OnConfigurationChanged() void
    }

    class HWEnvironmentFeatureProcessor {
        -PrepareSceneSrgEvent m_updateSceneSrgHandler
        -WindParam m_windParam
        -AZ::Data::Instance<AZ::RPI::Buffer> m_windParamBuffer
        -bool m_deviceBufferNeedsUpdate
        -bool m_isWriteable
        -EntityId m_entityId
        -AZStd::unique_ptr<HWEnvironmentSettings> m_settings
        -AZ::Data::Instance<AZ::RPI::Image> m_defaultBreezyTexture
        +Reflect(AZ::ReflectContext*)$ void
        +OnBeginPrepareRender() void
        +OnEndPrepareRender() void
        +GetSettingsInterface(AZ::EntityId)* HWEnvironmentSettingsInterface
        +GetOrCreateSettingsInterface(AZ::EntityId)* HWEnvironmentSettingsInterface
        +OnSettinigsChanged()* void
        +PrepareBuffers() void
        +UpdateSceneSrg() void
        +LoadDefaultBreezyTexture() void
    }

    class HWEnvironmentSettings {
        -HWEnvironmentFeatureProcessor* m_featureProcessor
        -m_HWEnvironmentParams
        +GetHWEnvironmentParams()
        +SetHWEnvironmentParams()
        +OnConfigurationChanged() void

    }

    class WindParam {
        +AZStd::array<float, 3> m_windVector
        +float m_breezyFrequency
        +float m_breezyLifeTime
        +float m_breezyStrength
        +float m_breezyTextureScale
    }

    HWEnvironmentRequestBus <|.. HWEnvironmentComponentController : Realization
    HWEnvironmentSettingsInterface <|.. HWEnvironmentSettings : Realization
    HWEnvironmentComponentController *-- EditorHWEnvironmentComponent : Composition
    HWEnvironmentSettingsInterface <.. HWEnvironmentComponentConfig : Dependency
    HWEnvironmentSettingsInterface <-- HWEnvironmentComponentController : Association
    HWEnvironmentComponentConfig *-- HWEnvironmentComponentController : Composition
    HWEnvironmentSettingsInterface <.. HWEnvironmentFeatureProcessor : Dependency
    HWEnvironmentFeatureProcessorInterface <|.. HWEnvironmentFeatureProcessor : Realization
    HWEnvironmentSettings <-- HWEnvironmentFeatureProcessor : Association
    WindParam *-- HWEnvironmentFeatureProcessor : Composition
    HWEnvironmentFeatureProcessor <-- HWEnvironmentSettings : Association   

The most important function is UpdateSceneSrg which is responsible for updating SceneSrg by using values from the editor component if necessary. We have added a new file SceneSrg.azsli to receive the updates from HWEnvironmentFeatureProcessor.

void HWEnvironmentFeatureProcessor::UpdateSceneSrg(AZ::RPI::ShaderResourceGroup* sceneSrg)
{
    if (!m_settings)
    {
        return;
    }

    sceneSrg->SetConstant(m_enabledWindIndex, m_settings->GetEnabledWind());
    sceneSrg->SetConstant(m_enabledBreezyIndex, m_settings->GetWindMode() == static_cast<uint32_t>(WindMode::BREEZY));
    sceneSrg->SetConstant(m_enabledMeadowWaveIndex, m_settings->GetEnabledMeadowWave());
    sceneSrg->SetBufferView(m_windParamBufferIndex, m_windParamBuffer->GetBufferView());
    sceneSrg->SetImage(
        m_meadowWaveTextureIndex, (m_settings->GetMeadowWaveTexture().get() ? m_settings->GetMeadowWaveTexture() : m_defaultMeadowWaveTexture));

    AZ::Render::RayTracingFeatureProcessor* rayTracingFeatureProcessor =
        GetParentScene()->GetFeatureProcessor<AZ::Render::RayTracingFeatureProcessor>();
    rayTracingFeatureProcessor->UpdateRayTracingEnvSrgs(m_settings->GetEnabledWind(), m_settings->GetWindVector());
}

About the shader, we would add a special COLOR input as a optional value. m_optional prefix gets set automatically by the system at runtime only if it's available.

// ---------- Vertex Shader ----------

struct VSInput
{
    // Base fields (required by the template azsli file)...
    float3 m_position : POSITION;
    float3 m_normal : NORMAL;
    float4 m_tangent : TANGENT;
    float3 m_bitangent : BITANGENT;

    // Extended fields (only referenced in this azsl file)...
    float2 m_uv0 : UV0;
    float2 m_uv1 : UV1;
    // Add the color attribute as a optional value
    float4 m_optional_vertexColor : COLOR;
};

In the vertex shader entry function, we add a snippet to call ApplyWindAnimation function to change the world position of this vertex.

VSOutput EnhancedPbr_ForwardPassVS(VSInput IN)
{
    VSOutput OUT;

    ...

    float4x4 objectToWorld = GetObjectToWorld();
    float4 worldPosition = mul(objectToWorld, float4(IN.m_position, 1.0));

    // Add the wind animation snippet
    if (o_enableWindAnimation) {
        float3 vPos = worldPosition.xyz;
        if (o_vertexColor_isBound) {
            ApplyWindAnimation(vPos, IN.m_normal, IN.m_optional_vertexColor, objectToWorld);
        } else {
            ApplyWindAnimation(vPos, IN.m_normal, float4(0.0, 0.0, 0.0, 1.0), objectToWorld);
        }
        worldPosition.xyz = vPos;
    }

    ...
}

In function ApplyWindAnimation, the detail implementations are just following the explanations in the feature design description.

What are the advantages of the feature?

• It is possible to achieve realistic wind animation for vegetation in enormous natural scene. Each plant instance can has its own animation state with the same global wind vector.
• Any other gem which responses to this component could get the wind related settings. As I know, the water system developed by our team now makes use of the wind settings to provide effect of the water wave, which will be contributed to the community too.
• The presented wind animation technique is implemented in a general way, so it is possible to apply wind forces even to non-vegetation objects, such as cloth, the only difference is that no main bending is used for these cases.

What are the disadvantages of the feature?

• It is a little difficult to bring up a very good effect because there are so many combinations from all parameters provided. It's possible that we don't need some parameters.
• Adding vertex deformation will have some minimal performance impact.

How will this be implemented or integrated into the O3DE environment?

We need to merge a new gem into O3DE with modifications of CMake lists. Besides, some updates about EnhancedPBR materialtype should be merged.

Are there any alternatives to this feature?

• Maybe it is more better to create a new materialtype to implement wind animation. Now the SSS effect for vegetation in O3DE, especially leaves, is also not quite as desirable. Our team currently try to implement a new SSS algorithm for two sided leaves rendering. When that is done, we could combine both the wind animation and SSS for leaves into a new vegetation materialtype.
• If the material canvas is fully available, it is an option to move wind animation implementation into material canvas as a common canvas node.
• Now the gem only supports global wind settings, but could extend to add more environment settings.

How will users learn this feature?

Once this proposal is passed, documents will be provided for users and developers to learn this feature. This feature can be added in USER GUIDE to let them know how to add the component and PBR Materials to let them know how to adjust the wind animation parameters in material editor.

Are there any open questions?

• The animation methods introduced here is a little old. There may be some better techniques now.
• Is it appropriate to place this component editor in the Level editor, since we may need to set several different wind sources with different world positions?
• How to add single character interactive response for single plant? How about for multiple plants simultaneously? How about adding multiple character interactions in a meadow?
• Is there any possibility to introduce SpeedTree to provide a complete and detailed vegetation animation system?

[adiblev]

This is a very important RFC that will make natural scenery 'act alive' as it is expected in real life. Small observations / reservation - this feature should apply to all PBR shader as it is specifically related to the vertex shader. In the game industry it is common to have a dedicate vegetation shader because it assumes faster pixel shader based on optimization methods, otherwise the price of alpha blend / pixel coverage and regular shader operation over large vegetation area is quite high. I suggest avoiding to specific shader as suggested (EnhancedPBR) and apply it to all generic PBR vertex shader with the understanding / assumption that a possible dedicated vegetation PBR shader might be created soon.

The application of the different vertex color channels as different attributes and wave length is very important (Ala CryTek GPU Gems 3 if I'm not mistaken) and should be well documented when the feature is done.

All in all - very important addition

[moudgils]

Based on this addition

    // Add the color attribute as a optional value
    float4 m_optional_vertexColor : COLOR;

I think this feature can benefit from adding support for optional vertex streams. I wonder how easy it would be to add shader option support within the vertex layout structure. It would mean that based on the shader variant we would need to provide different vertex data. Tagging @amzn-tommy @galibzon.

[moudgils]

Actually why dont we just create a new material type with optimized shader for vegetation as part of this work instead of adding this support to enhancedpbr? I would imagine that you would not want to apply enhanced pbr to trees/vegetation as it can significantly impact gpu perf. So we could create something like BasePBRWithWind material type?

[lijianwen13]

This is a very important RFC that will make natural scenery 'act alive' as it is expected in real life. Small observations / reservation - this feature should apply to all PBR shader as it is specifically related to the vertex shader. In the game industry it is common to have a dedicate vegetation shader because it assumes faster pixel shader based on optimization methods, otherwise the price of alpha blend / pixel coverage and regular shader operation over large vegetation area is quite high. I suggest avoiding to specific shader as suggested (EnhancedPBR) and apply it to all generic PBR vertex shader with the understanding / assumption that a possible dedicated vegetation PBR shader might be created soon.

The application of the different vertex color channels as different attributes and wave length is very important (Ala CryTek GPU Gems 3 if I'm not mistaken) and should be well documented when the feature is done.

All in all - very important addition

Thanks for your reply. I am not sure where to apply it, since the possible vegetation PBR shader is being developed now. As moudgils said, he suggested to create a new material type like BasePBRWithWind. The application of the different vertex color channels as different attributes is actually introduced from CryTek GPU Gems 3. I will add more technique details about the application into this RFC as soon as possible.

[lijianwen13]

Based on this addition

    // Add the color attribute as a optional value
    float4 m_optional_vertexColor : COLOR;

I think this feature can benefit from adding support for optional vertex streams. I wonder how easy it would be to add shader option support within the vertex layout structure. It would mean that based on the shader variant we would need to provide different vertex data. Tagging @amzn-tommy @galibzon.

Thanks for your reply. This feature actually benefit from the optional vertex streams. There is a soft naming convention that associates this with o_vertexColor_isBound, which will be set to true whenever m_optional_vertexColor is available. m_optional_vertexColor would be processed correctly in ShaderVariantAssetBuilder. If there is a vertex color attribute in the vertex streams of the mesh, m_optional_vertexColor would be set true, otherwise false. By the way, I am also expecting a more elegant way to deal with this.

[lijianwen13]

Actually why dont we just create a new material type with optimized shader for vegetation as part of this work instead of adding this support to enhancedpbr? I would imagine that you would not want to apply enhanced pbr to trees/vegetation as it can significantly impact gpu perf. So we could create something like BasePBRWithWind material type?

Actually, I am not sure where to apply it now. Maybe a new material type like BasePBRWithWind is a better way compared with applying it to all generic PBR vertex shader as adiblev suggested. If do that, features in BasePBRWithWind would only include wind animation at current time, since other features like adding a faster pixel shader and better SSS for two sided foliage haven't completely done. Is a material type like this acceptable for the community?

[mbalfour-amzn]

HWEnvironment Settings Component:

• Consider renaming to "Rendering Wind Settings" or something to make it clear that this is only creating visual wind and not physics wind.
• It would be awesome if the wind settings could be tied to volumes in the world instead of being global so that the wind direction and force could change in different areas. Imagine windbreaks, large fans, or just different sides of a mountain.
• The "Wind Mode" maybe shouldn't be a part of this component. If the Wind Vector is exposed to the behavior context, people could write whatever variance function they want using ScriptCanvas / Lua / etc. If it stays, maybe use less friendly but more technically accurate names like "constant", "sine wave", etc, since "breezy" could mean a lot of different things.

Gem name - consider "Wind Effects" instead of "Environment"? Or are there plans to put other environmental controls in this gem as well?

[lijianwen13]

HWEnvironment Settings Component:

• Consider renaming to "Rendering Wind Settings" or something to make it clear that this is only creating visual wind and not physics wind.
• It would be awesome if the wind settings could be tied to volumes in the world instead of being global so that the wind direction and force could change in different areas. Imagine windbreaks, large fans, or just different sides of a mountain.
• The "Wind Mode" maybe shouldn't be a part of this component. If the Wind Vector is exposed to the behavior context, people could write whatever variance function they want using ScriptCanvas / Lua / etc. If it stays, maybe use less friendly but more technically accurate names like "constant", "sine wave", etc, since "breezy" could mean a lot of different things.

Gem name - consider "Wind Effects" instead of "Environment"? Or are there plans to put other environmental controls in this gem as well?

Thanks for the helpful suggestions. We would like to change the components as you said, as they are necessary to improve the usability for users. As for the gem name, we indeed want to put other environmental controls in our internal version of the engine, such as volume fog/cloud and so on. By the way, if developers in community who have added a new environment effect, I think it should be a chance to add more controls in this gem.

[galibzon]

Approved.

[moudgils]

Since this RFC is accepted please open a PR and move this RFC to this folder - https://github.com/o3de/sig-graphics-audio/tree/main/rfcs where we will track all the new RFCs. Thanks.