Figura avatars do not work correctly with physics mod ragdolls & other physics features

[NobleDraconian]

Currently, as of figura v0.1.0-rc.14+1.19.2, Figura does not correctly render the ragdolls from physics mod. Instead of the avatar ragdolling, it does the default entity death animation of tilting sideways.

What should happen:

https://github.com/Moonlight-MC-Temp/Figura/assets/26460940/9e23414f-255f-4725-a7d0-a6e782f6e739

What actually happens:

https://github.com/Moonlight-MC-Temp/Figura/assets/26460940/f1ab60ff-39b1-44f1-9758-7a5dd2304766

After speaking with the physics mod developer, it was found out that it is completely possible to make figura avatars work with physics mod, including its ragdolls. Essentially, figura has to get the vertices of the character & send them to physics mod for rendering every frame. E.g. while ragdoll is active, figura sends mesh info (verts etc) to physics mod.

Here's some of the code snippets the mod author sent:

package net.diebuddies.compat;

import java.util.List;

import org.joml.Matrix3f;
import org.joml.Matrix4f;
import org.joml.Vector2f;
import org.joml.Vector3f;
import org.joml.Vector4f;

import com.mojang.blaze3d.vertex.PoseStack;

import net.diebuddies.config.ConfigMobs;
import net.diebuddies.opengl.TextureHelper;
import net.diebuddies.physics.Mesh;
import net.diebuddies.physics.PhysicsEntity;
import net.diebuddies.physics.PhysicsMod;
import net.diebuddies.physics.StarterClient;
import net.diebuddies.physics.settings.mobs.MobPhysicsType;
import net.minecraft.client.Minecraft;
import net.minecraft.client.renderer.entity.EntityRenderer;
import net.minecraft.client.renderer.entity.layers.RenderLayer;
import net.minecraft.util.Mth;
import net.minecraft.world.entity.Entity;
import software.bernie.geckolib.cache.object.GeoCube;
import software.bernie.geckolib.cache.object.GeoQuad;
import software.bernie.geckolib.cache.object.GeoVertex;

public class GeckoLib {

    public static void createParticlesFromCuboids(PoseStack local, GeoCube cube, Entity entity, EntityRenderer renderer, RenderLayer feature,
            int overlay, float red, float green, float blue) {
        Matrix4f localM = local.last().pose();
        Matrix3f localNM = local.last().normal();

        PhysicsMod mod = PhysicsMod.getInstance(entity.getCommandSenderWorld());

        int textureID = TextureHelper.getLoadedTextures();

        float partialTicks = Minecraft.getInstance().getFrameTime();
        double px = Mth.lerp(partialTicks, entity.xo, entity.getX());
        double py = Mth.lerp(partialTicks, entity.yo, entity.getY());
        double pz = Mth.lerp(partialTicks, entity.zo, entity.getZ());

        Vector4f[] minMax = new Vector4f[6];
        Vector3f min = new Vector3f(Float.MAX_VALUE);
        Vector3f max = new Vector3f(-Float.MAX_VALUE);

        Vector3f tmpNormal = new Vector3f();
        Vector4f tmpPos = new Vector4f();
        for (int i = 0; i < minMax.length; i++) minMax[i] = new org.joml.Vector4f();
        MobPhysicsType type = ConfigMobs.getMobSetting(entity).getType();

        Vector3f tmp = new Vector3f();

        for (int i = 0; i < cube.quads().length; i++) {
            GeoQuad quad = cube.quads()[i];

            float minU = 1.0f, maxU = 0.0f;
            float minV = 1.0f, maxV = 0.0f;

            for (GeoVertex vertex : quad.vertices()) {
                if (vertex.texU() < minU) minU = vertex.texU();
                if (vertex.texV() < minV) minV = vertex.texV();
                if (vertex.texU() > maxU) maxU = vertex.texU();
                if (vertex.texV() > maxV) maxV = vertex.texV();

                tmp.set(vertex.position().x(), vertex.position().y(), vertex.position().z());
                min.min(tmp);
                max.max(tmp);
            }

            minMax[i].set(minU, maxU, minV, maxV);
        }

        int[] remap = new int[] { 3, 0, 2, 1, 4, 5 };

        float volume = (Math.abs(max.x - min.x)) * (Math.abs(max.y - min.y)) * (Math.abs(max.z - min.z));
        boolean isBlocky = type == MobPhysicsType.BLOCKY || type == MobPhysicsType.RAGDOLL || type == MobPhysicsType.RAGDOLL_BREAK || type == MobPhysicsType.RAGDOLL_BREAK_BLOOD;

        // this is the proper solution but crashes with the convex physx solver since the convex mesh would be coplanar
//          if (volume <= 0.0001 && !isBlocky) continue;
        // so just skip small volumes
        boolean noVolume = false;

        if (volume <= 0.0001) {
            if (isBlocky) {
                noVolume = true;
            } else {
                return;
            }
        }

        boolean mirror = cube.mirror();

        List<Mesh> meshes = PhysicsMod.brokenBlocksLittle.get((int) (net.diebuddies.math.Math.random() * PhysicsMod.brokenBlocksLittle.size()));

        if (volume <= 0.04 || isBlocky) meshes = PhysicsMod.brokenBlock;

        for (Mesh mesh : meshes) {
            PhysicsEntity particle = new PhysicsEntity(PhysicsEntity.Type.MOB, entity.getType());
            particle.feature = feature;
            particle.noVolume = noVolume;
            particle.models.get(0).textureID = textureID;
            Mesh clone = new Mesh();
            particle.models.get(0).mesh = clone;
            particle.getTransformation().translation(px, py, pz);
            particle.getOldTransformation().set(particle.getTransformation());
            int count = 0;
            Vector3f offset = new Vector3f();

            for (int i = 0; i < mesh.indices.size(); i++) {
                int index = mesh.indices.getInt(i);
                byte sideIndex = mesh.sides.getByte(index);

                Vector3f position = mesh.positions.get(index);
                Vector2f uv = mesh.uvs.get(index);
                Vector3f normal = mesh.normals.get(index);

                float r = red, g = green, b = blue;

                if (sideIndex == -1) {
                    if (type == MobPhysicsType.FRACTURED_BLOOD) {
                        r = 0.6f;
                        g = 0.0f;
                        b = 0.0f;
                    }

                    sideIndex = 0;
                }

                tmpNormal.set((mirror) ? (float) -normal.x : (float) normal.x, (float) normal.y, (float) normal.z);
                localNM.transform(tmpNormal);

                org.joml.Vector4f minMaxUVs = minMax[remap[sideIndex]];

                boolean lmirror = (mirror && (sideIndex == 0 || sideIndex == 2)) || (!mirror && (sideIndex == 1 || sideIndex == 3));

                tmpPos.set(
                        (float) net.diebuddies.math.Math.remap(position.x + mesh.offset.x, lmirror ? 0.5 : -0.5, lmirror ? -0.5 : 0.5, min.x, max.x),
                        (float) net.diebuddies.math.Math.remap(position.y + mesh.offset.y, -0.5, 0.5, min.y, max.y),
                        (float) net.diebuddies.math.Math.remap(position.z + mesh.offset.z, -0.5, 0.5, min.z, max.z), 1.0f);

                localM.transform(tmpPos);

                clone.indices.add(count);
                offset.add(tmpPos.x(), tmpPos.y(), tmpPos.z());
                count++;
                Vector3f posR = new Vector3f(tmpPos.x(), tmpPos.y(), tmpPos.z());
                clone.positions.add(posR);

                if (sideIndex == 4 || sideIndex == 5) {
                    clone.uvs.add(new Vector2f(
                            net.diebuddies.math.Math.remap((float) uv.x, 0.0f, 1.0f, minMaxUVs.x, minMaxUVs.y),
                            net.diebuddies.math.Math.remap((float) uv.y, 0.0f, 1.0f, minMaxUVs.z, minMaxUVs.w)));
                } else if (sideIndex == 0 || sideIndex == 2) {
                    clone.uvs.add(new Vector2f(
                            net.diebuddies.math.Math.remap((float) uv.x, 0.0f, 1.0f, minMaxUVs.x, minMaxUVs.y),
                            net.diebuddies.math.Math.remap((float) uv.y, 1.0f, 0.0f, minMaxUVs.z, minMaxUVs.w)));
                } else {
                    clone.uvs.add(new Vector2f(
                            net.diebuddies.math.Math.remap((float) uv.x, 1.0f, 0.0f, minMaxUVs.x, minMaxUVs.y),
                            net.diebuddies.math.Math.remap((float) uv.y, 1.0f, 0.0f, minMaxUVs.z, minMaxUVs.w)));
                }

                clone.normals.add(new org.joml.Vector3f(tmpNormal.x(), tmpNormal.y(), tmpNormal.z()));
                clone.addColor(r, g, b);
            }

            if (StarterClient.iris || StarterClient.optifabric) {
                clone.calculatePBRData(false);
            }

            offset.div(clone.positions.size());

            for (Vector3f position : clone.positions) {
                position.sub(offset);
            }

            clone.offset = offset;

            mod.blockifiedEntity.add(particle);
        }
    }

}

@Pseudo
@Mixin(GeoRenderer.class)
public interface MixinIGeoRenderer {

    @Inject(at = @At("HEAD"), method = "renderCube", remap = false)
    default void renderCube(PoseStack stack, GeoCube cube, VertexConsumer bufferIn, int packedLightIn,
            int packedOverlayIn, float red, float green, float blue, float alpha, CallbackInfo info) {
        if (PhysicsMod.getCurrentInstance() != null && PhysicsMod.getCurrentInstance().blockify) {
            GeckoLib.createParticlesFromCuboids(stack, cube,
                    PhysicsMod.getCurrentInstance().cubifyEntity, PhysicsMod.getCurrentInstance().cubifyEntityRenderer, PhysicsMod.getCurrentInstance().blockifyFeature,
                    packedOverlayIn, red, green, blue);
        }
    }

}

EntityRenderer entityRenderer = Minecraft.getInstance().getEntityRenderDispatcher().getRenderer(entity);
DummyMultiBufferSource source = new DummyMultiBufferSource();

try {
    renderer.render(entity, 0.0f, Minecraft.getInstance().getFrameTime(), stack, source, 0);
} catch (Exception e) {
    System.err.println("error rendering " + entity.getClass());
    e.printStackTrace();
} finally {
    if (source.lastLayer != null) source.lastLayer.clearRenderState();
}

To read the full conversation, go here: https://discord.com/channels/231062298008092673/882927654007881778/1104313532826267702

Fixing this incompatibility would expand what we can do with figura avatars! We could for example have bunny ears that move / flop while the player walks, etc.

[TheBunnyMan123]

This seems like it's better suited for an addon or fork imo

[Manuel-3]

This seems like it's better suited for an addon or fork imo

No harm if it's in the base Figura mod in my opinon.

[TheBunnyMan123]

This seems like it's better suited for an addon or fork imo

No harm if it's in the base Figura mod in my opinon.

I do agree there is no harm, I just feel it is better suited for an addon/fork

[Manuel-3]

This seems like it's better suited for an addon or fork imo

No harm if it's in the base Figura mod in my opinon.

I do agree there is no harm, I just feel it is better suited for an addon/fork

Well, what I was trying to say is, if its in an addon, it will only work for the people who have that addon installed. So imagine someone just installing Figura and Physics mod, and it just doesnt work. Then they will go ask about it, and get told to install an addon. Could have just been in base Figura. Also, it will only work for people who have that addon installed, so for multiplayer everyone will have to do that.

And it's not like this is some exotic feature that doesnt feel right in base Figura, it's literally a mod compat. Why on earth would you put a mod compat in an external addon, that makes no sense.

[TheBunnyMan123]

This seems like it's better suited for an addon or fork imo

No harm if it's in the base Figura mod in my opinon.

I do agree there is no harm, I just feel it is better suited for an addon/fork

Well, what I was trying to say is, if its in an addon, it will only work for the people who have that addon installed. So imagine someone just installing Figura and Physics mod, and it just doesnt work. Then they will go ask about it, and get told to install an addon. Could have just been in base Figura. Also, it will only work for people who have that addon installed, so for multiplayer everyone will have to do that.

And it's not like this is some exotic feature that doesnt feel right in base Figura, it's literally a mod compat. Why on earth would you put a mod compat in an external addon, that makes no sense.

Oh I didn't even think of that :P I agree it should probably be in base figura if it is made

[Djspac3]

Wouldn't this be VERY Inefficient Especially for your model you used Wouldn't something like a collision set of body parts be better? Like that got to be over 100 Complexity so that's more then like 400 vertices being sent PER FRAME (I might be getting this wrong) Even then still very bad for proformance

Please don't fully take me seriously I know BASICALY nothing about what it would have to do or how just saying som ideas

[NobleDraconian]

That's not how that works, the default blocky body is used for collisions.