Move Blend Shape Modifiers to MeshInstance

kayomn commented 3 years ago

Describe the project you are working on

A 3D game that uses blend shapes for character customization.

Describe the problem or limitation you are having in your project

Blend shapes modifiers are attached to the Mesh resource type and not the MeshInstance node type, meaning that meshes that are modified by blend shapes affect all others that share the same mesh resource.

Currently, the recommended best practice is to duplicate the mesh, but this rarely ever works without a series of deformation bugs occurring on the original mesh and bloats memory with duplicate meshes that don't vary in any way except for their blend shape modifiers - which are calculated on-the-fly anyway.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

By moving blend shape modifiers to the MeshInstance, modifiers become local to that instance of the mesh, instead of global to all instances. Meshes continue to function as before, but blend shapes only apply to specific users of the mesh rather than all of them.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

All modifier values in the inspector for blend shapes are moved from the Mesh resource type to the MeshInstance node type. Modifiers are applied in the scene or programmatically by getting and setting the properties of the MeshInstance blend shapes.

If this enhancement will not be used often, can it be worked around with a few lines of script?

There is no work-around for this limitation that I know of beyond duplicating mesh resources at runtime, which is wasteful and error-prone.

Support for the old blend shape approach may be phased out by keeping the old modifier logic on the Mesh resource and simply adding instance-based modifiers. Then, in the next breaking release, they may be removed.

Is there a reason why this should be core and not an add-on in the asset library?

Instance-based blend shape modifiers are the standard way that other 3D engines tackle this same issue, and not just due to ad populum. By making blend shape modifiers local to the renderer / instance of the mesh, the creation of needless duplicate resources is avoided and assets become far more reusable, which is invaluable in a game that uses blend shapes for any of the following:

Character appearance modification.
Mouth animation.
Discrete animations that do not need a dedicated bone.

Calinou commented 3 years ago

For reference, this was discussed in https://github.com/godotengine/godot/issues/37854.

smix8 commented 3 years ago

I second this proposal change and more.

I am getting PTSD if someone pings me and mentions Godot Blendshapes. Well, every improvement is welcome and I see a few issues with how blendshapes are currently implemented which is a pity as Blendshapes are such a core feature for 3D games and really cumbersome at the moment in both Godot 3.x and 4.x. Nearly half my opened issues have blendshapes as a topic.

Might as well list them here again so they can be considered if someone should start on proposed rework and blendshapes receive a new home/format.

Current Issues

Blendshapes create artifacts and VisualServer crashes with runtime changes, seems dirty flags are missing / not set correctly on resource/skeleton changes and blendvalues not always initialize with correct interpolation values, see issue #37854
Blendshapes can't be shared among multiple actors without huge overhead in editor filesize and runtime memory, see below.
(File)format is inefficient / unoptimized with many near zero values that could be omitted in file and compensated at runtime
Editor Inspector handling is poor with Blendshapes. See issue #35934

So here is my wishlist if anyone feels brave enough to tackle Blendshapes:

Implementation Wishlist:

Blendshape offset data are their own, separate saved resource files (this proposal?). I cannot stress enough how very important this is for game patching. Currently a small changed blendshape (e.g. an eye or lip tweak) requires not only a redundant copy of arraymesh data but also redunant copies of all the other blendshapes of the same mesh resource. A typical highres 3D character with a 150 mb mesh has easily a filesize of 500-1000 mb with current blendshape implementation creating patchsizes that are unacceptable cause more than half the patchdata is for giggles.
Blendshape resources can be loaded, applied, stacked and removed smoothly at runtime (threadsave) .. without destroying mesh, stuttering the game for seconds (due to duplicating entire meshes) or crashing the Visualserver.
Blendshape resources can be reused on any meshinstance with meshresource with matching vertices count Cause I don't always have the same shaped basemesh resource but I want to reuse the identical blendshape offset data.
Blendshape resources can be shared among multiple meshinstances without duplicate() as offset values are always the same for interpolation, no need to waste memory/filesize with redunant data.
Blendshape resources save default to binary format The huge filesize and long editor and game loadtimes created by the many small numbers of blendshapes saved as e.g. .tres textfile are a beginner trap and also very annoying on larger editor (auto)imports.
3D importer has optimization options to stop unchanged blendshape values from saving to file. or nearly unchanged, like all those very small 0.001 values that are invisible to spot and basically the same as the basemesh value.
Blendshape data limiter for Vertexgroups/Surfaces e.g. a way so a mini blendshape that changes 12 vertices on a single surface doesn't resave/duplicate nearly entire mesh data. Fill omitted groups/surface with zero interpolation values from the basemesh at runtime.
Editor Inspector shows all found Blendshapes with sliders with multiple Meshinstances selected Currently a single missing blendshape disables all blendshape sliders. Very unfun to adjust shared blendshapes on a large set of characters that have a few additonal, unshared custom blendshapes. see #35934

kayomn commented 3 years ago

I second this proposal change and more.

I am getting PTSD if someone pings me and mentions Godot Blendshapes. Well, every improvement is welcome and I see a few issues with how blendshapes are currently implemented which is a pity as Blendshapes are such a core feature for 3D games and really cumbersome at the moment in both Godot 3.x and 4.x. Nearly half my opened issues have blendshapes as a topic.

Might as well list them here again so they can be considered if someone should start on proposed rework and blendshapes receive a new home/format.

Current Issues

* Blendshapes create artifacts and VisualServer crashes with runtime changes, seems dirty flags are missing / not set correctly on resource/skeleton changes and blendvalues not always initialize with correct interpolation values, see issue #37854

* Blendshapes can't be shared among multiple actors without huge overhead in editor filesize and runtime memory, see below.

* (File)format is inefficient / unoptimized with many near zero values that could be omitted in file and compensated at runtime

* Editor Inspector handling is poor with Blendshapes. See issue #35934

So here is my wishlist if anyone feels brave enough to tackle Blendshapes:

Implementation Wishlist:

* **Blendshape offset data are their own, separate saved resource files (this proposal?)**.
  I cannot stress enough how very important this is for game patching. Currently a small changed blendshape (e.g. an eye or lip tweak) requires not only a redundant copy of arraymesh data but also redunant copies of all the other blendshapes of the same mesh resource. A typical highres 3D character with a 150 mb mesh has easily a filesize of 500-1000 mb with current blendshape implementation creating patchsizes that are unacceptable cause more than half the patchdata is for giggles.

* **Blendshape resources can be loaded, applied, stacked and removed smoothly at runtime (threadsave)**
  .. without destroying mesh, stuttering the game for seconds (due to duplicating entire meshes) or crashing the Visualserver.

* **Blendshape resources can be reused on any meshinstance with meshresource with matching vertices count**
  Cause I don't always have the same shaped basemesh resource but I want to reuse the identical blendshape offset data.

* **Blendshape resources can be shared among multiple meshinstances without duplicate()**
  as offset values are always the same for interpolation, no need to waste memory/filesize with redunant data.

* **Blendshape resources save default to binary format**
  The huge filesize and long editor and game loadtimes created by the many small numbers of blendshapes saved as e.g. .tres textfile are a beginner trap and also very annoying on larger editor (auto)imports.

* **3D importer has optimization options to stop unchanged blendshape values from saving to file**.
  or nearly unchanged, like all those very small 0.001 values that are invisible to spot and basically the same as the basemesh value.

* **Blendshape data limiter for Vertexgroups/Surfaces**
  e.g. a way so a mini blendshape that changes 12 vertices on a single surface doesn't resave/duplicate nearly entire mesh data. Fill omitted groups/surface with zero interpolation values from the basemesh at runtime.

* **Editor Inspector shows all found Blendshapes with sliders with multiple Meshinstances selected**
  Currently a single missing blendshape disables all blendshape sliders. Very unfun to adjust shared blendshapes on a large set of characters that have a few additonal, unshared custom blendshapes. see #35934

I agree with most of this but have a few things I want to raise in contest:

I think that as per convention with the rest of the engine, the file should default to a textual resource file by default and use binary instead if set in the project preferences or explicitly saved as such. Realistically, the file contents would look something like this:

[gd_resource type="MeshBlendShapes" format=2]

[resource]
blend1 = 0.1
blend2 = 0.02
blend_name_foo = 0.987
blend3 = 0.23

A per-vertex group and / or per surface blend shape data limiter is beyond the scope of this proposal I think.
When you say optimization options for a file, are you referring to the blend shape resource file? I'm unsure why this would need to be an option to begin with; can you elaborate?

smix8 commented 3 years ago

I see what format you had in mind from your example, I think I misunderstood parts of your proposal.

I meant the actual storage for the blendshape data as part of the mesh resource and on the visual server not just the interpolation values so my part about blendshape fileformat changes/concerns might be outside the scope of your proposal.

If this is the case I am curious what is the point of moving just the interpolation values to a new modifier/resource. The mesh resource still needs to be kept unique to cause no Visualserver glitches and the large PoolByteArray numbers for each blendshape are currently all part of the mesh resource. So this proposal would keep the old status quo on this? I don't see how this would then be able to solve half the mentioned issues, especially with memory bloat from blendshape mesh duplicates with large amounts of redundant data.

When you say optimization options for a file, are you referring to the blend shape resource file? I'm unsure why this would need to be an option to begin with; can you elaborate?

Currently Arraymeshes store gigantic PoolByteArray values for each Blendshape and Surface both in file and on Visualserver. A large amount of these values are so small or unused, they don't affect the final mesh visible with Godot's 3D scale and could all be replaced by default basemesh values at runtime and removed from the saved file to reduce filesize of Arraymesh resources with blendshapes considerably.

Regarding fileformat convention, Godot 3D assets with blendshapes require 500%+ more storage saved as text due to all the small PoolByteArray numbers. That is why text formats for 3D assets become rapidly pointless for practical use except for special debug cases. They bloat the diskspace and take long to process/load. Godot also doesn't save 2D image files as raw text for good reasons so why do it for even more complex 3D asset formats? Binary is the default export option for 3D in most DDC tools for a reason.

kayomn commented 3 years ago

Currently Arraymeshes store gigantic PoolByteArray values for each Blendshape and Surface both in file and on Visualserver. A large amount of these values are so small or unused, they don't affect the final mesh visible with Godot's 3D scale and could all be replaced by default basemesh values at runtime and removed from the saved file to reduce filesize of Arraymesh resources with blendshapes considerably.

I see, I was under the impression that the interpolation modifiers were applied at render-time.

lukostello commented 2 years ago

@Kayomn

I see, I was under the impression that the interpolation modifiers were applied at render-time.

the problem with doing it that way is if the blendshape doesn't change then there isn't any need to make that same computation again. Storing it in the mesh resource allows us to save the results of that computation which is why I'm suggesting we move the blendshape values to the mesh resource instead. See proposal #3477

godotengine / godot-proposals