Provide dedicated skin and hair shaders for creating realistic human characters

[GeorgeS2019]

Describe the project you are working on

An early concept of the feasibility of using Godot for creating realistic character

Describe the problem or limitation you are having in your project

As we can expect more users would be interested in creating 3D games with a more realistic style, it'd be better if we can provide dedicated shaders to render human skin and hairs.

===> This is not my proposal but simply bringing old (2019) suggestion here

Describe the feature / enhancement and how it helps to overcome the problem or limitation

As I described in another issue What it takes to create a photorealistic character, there are many obstacles that prevent one from creating a realistic human character in Godot and the lack of such dedicated shaders is one of them.

Moreover, a human skin material within an environment with a GI probe would either look completely dry or soaked in oil, depending on how it's shadowed, thus making it practically impossible to achieve even a remotely realistic result as is described in https://github.com/godotengine/godot/issues/19561.

And if we can accept what is discussed in that issue, it's apparently an expected behavior and we would need a dedicated skin shader to fix such a problem, so I think this would give this issue a bit more of importance than it would otherwise.

Currently, both Unity and Unreal provide their users such dedicated shaders to create a character with decent-looking skin and hairs by default. In comparison, we give our users a generic spatial material with which they can spend much time to make a plastic-looking skin that looks completely broken when it's shadowed, and they wouldn't find a clear remedy for the problem unless they are well versed in writing custom shaders (that is, provided https://github.com/godotengine/godot/issues/19561 can be actually solved by writing a shader, without any engine modification).

Eventually, we may be able to provide our users something like this, so they can produce photorealistic results.

But in the meanwhile, I think even a much more primitive shader that can at least mitigate such a problem like https://github.com/godotengine/godot/issues/19561 would help those who try to create games with a more realistic style with Godot for now.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

If this enhancement will not be used often, can it be worked around with a few lines of script?

No

Is there a reason why this should be core and not an add-on in the asset library?

This is add-on to create realistic human skin and hairs (revised based on feedback)

One of the main feedback Juan received from his presence at Game Developers Conference (GDC), on March 20-24 2023 was that there is a strong demand for improving Godot4 rendering.

I think this is a strong justification for why such photo-realistic rendering needs to go into ONE of its Future Core Features.

---

Further references:

Photo Realistic Graphics in Godot? Yes!

• Godot Realistic Graphics

• Photorealism in Godot 4 !

• Photorealistic demo made in a few hours in Godot 4

• Godot Photorealistic Graphics

• Realistic human skin practice

[Zireael07]

Is there a reason why this should be core and not an add-on in the asset library? This is add-on to create realistic human skin and hairs

If you proposed an addon, then this is not the place for this.

If this is supposed to go into core, then this is not sufficient explanation why this should be core

[AThousandShips]

Adding to the argument that this is addon territory rather than core, beyond the aim that Godot should be lean, is that it places the burden of responsibility to not be caught in biases about human diversity on the development of Godot, which is a major and important thing that requires a lot of work

[AThousandShips]

What do you mean by "core human realistic rendering"?

[smix8]

This can basically be solved with a documentation page and some shader code examples like Godot documentation already has for some other shader stuff that can not be solved with just basic StandardMaterial3D.

A lot that goes into a proper "realistic" look is texture and mesh model work and good light setup, the shader is only one part of that.

[Zireael07]

If you want a reference model and texture, this is further proof this is addon-territory, see for instance Calinou's Sponza scene

[AThousandShips]

I believe that Godot should provide the base but not the specifics for high level functionality like specific photorealism, Godot should be flexible enough to allow users to do this, while not adding the complexity of providing the specific implementation, in other words work towards making Godot better for photorealism on one hand, and adding tools or bundled resources for doing specific things on the other

The former liberates and creates room for what you want to accomplish, but the later risks restricting and forcing you to make your own anyway if it doesn't fit

Providing the specific resources puts a lot of additional work on the development team

[AThousandShips]

(Also consider the fact that all the examples of photorealistic humans you have provided yourself are of light skin)

[AThousandShips]

It has to be inclusive, anything else is unacceptable, and my concern here is the harm that can be caused if Godot makes specific decisions in creating resources for realistic rendering of humans based on these biases, but agreed the ability of the rendering system to accomplish this is also important, but as this is about dedicated shaders for this the risk of that bias should be the focus imo

[smix8]

I though this topic was about hair and skin shader, which has nothing at all to do with your inclusivity concerns.

The things you are concerned about is what kind of body shape the character meshes have or what albedo colors the used textures have, which again, has nothing at all to do with how to make a proper hair or skin shader.

[AThousandShips]

Maybe I misunderstood this but this proposal sounded like a way to create realistic human skin shaders out of the box, and MakeHuman was given as an example which seems to be providing that feature, i.e. creating from scratch this feature, that's where the concerns are

[AThousandShips]

tl;dr; Entering any situation involving the representation, depiction, etc. of humans and not considering how biases and diversity of the people involved and the ability to do this correctly is setting oneself up to fail

---

Also I'm not convinced that these aspects of diversity doesn't apply even just with the basic aspects of the parameters of plain shaders without making any input from the engine on albedo, I am not informed enough but I don't think it can be dismissed either, does the setup and specifics of such a base shader not impact the ability of it to render the full diversity of human skin in any way?

To be clear, I have no deeper knowledge on this, but I am well aware of the issues of correct representation of diverse skin across the gaming and rendering landscape (beyond just the lack of diversity of choice in way too many games) and my concerns are that I am not alone in lacking this in-depth knowledge and that how it can affect the implementation of such a feature, especially if the very fact of it is not considered, like how this kind of biases is why photographic film and even digital cameras are less able to faithfully capture darker skin tones

To be clear, I'm not saying this has to be something that necessarily affects the shaders, and if it doesn't then that's perfect! But if it does and no attention is paid to it that is an issue, for example is there no chance what so ever that a shader tested primarily with white skin as a reference won't be less able to correctly represent darker skin? Because that's the kind of bias that this is about, and that the process must be conscious of this

I feel that any time representation of humans, be that rendering humans or speaking about humans, these things must be taken into account, because they so often aren't, especially in these fields

[YuriSizov]

Thanks for showing the interest in this proposal, everyone, but I would like to remind you that this is a discussion platform, not a chat. Please avoid posting several messages in quick succession, and use the edit button instead. Every post you make creates a notification for everyone subscribed to the repository (or the issue), and several disjointed comments are harder to read than a single concise one.

I would also urge you to take a bit of time before replying, so your answer doesn't require one or two more addendums in the first place, and so the other party doesn't feel hurried by your immediate response to argue back. If you want to have a more live discussion on the matter, I invite you to the Godot contributors chat at https://chat.godotengine.org.

[mrjustaguy]

My 2 cents on this

1) We don't need a dedicated Skin Shader, Base Material works just fine and can fairly easily get the right values for proper skin, and this can be documented for those that don't get proper Texture Maps for skin bundled with the humans, such as is the case with MakeHuman mentioned above 2) The Main issue that's causing the skin to look wrong is a problem inherent with raster, and that's reflections, they look wrong in most instances. This cannot be simply fixed with a dedicated shader, only mitigated by artists dealing with environment lighting. A true solution would be to do Ray Tracing, as that's the only way to get proper reflections 3) Only a Hair Shader makes sense, something along the lines of TressFX https://gpuopen.com/tressfx/ as hair does need to be simulated and card hair is typically horrible and quite obvious 4) I'd understand a Dedicated Human Node in the editor, for things like Character creators and for having a Node that handles realistic motion, however both don't have license compatible bases to work off and are massive undertakings each in their own right, and is something likely to start out as an addon if ever

[Zireael07]

@mrjustaguy I never could get skin materials working passably well so it would need more documentation indeed

[yosoyfreeman]

Just to get some context about what i'm going to say, the hair technology is an specific example in the talk "A case against fidelity" which i aligns with what Godot is trying to achieve. So i'm not going to talk about hair. Source: https://www.youtube.com/watch?v=ylV5R-Z-ZGg

Now, for skin, and talking from an artist perspective, not a rendering programmer one, i think skin shaders do not exist at all.

Strictly speaking, a skin shader is just a pbr texture who represents skin. This is easy to do.

What people usually refer to as "Skin shaders" is not an "Skin" shader, is a "Human surface" shader. Which is basically a common pbr shader with lots of overlays for things that are NOT sking but you can see trough skin: The flesh, the veins, the bones even with extreme light.

So, where we stop adding layers of detail? What is human enough? Why should we make it photorealistic in fist place if Godot openly says it does not try to achieve that?

Even with ray tracing you can not simulate a totally realistic human just with materials. You would need to simulate the organic structure inside the body or you will be simulating a skin bag.

But lets say we make that. We implement a shader that is just a bunch of layers in which put detail textures. You can't make that generic in an easy way, that's why Metahuman exists in first place. If its white guy skin texture, that is what is going to be. Black skin, or any other type of skin is not a variation of white skin. There is not even a consensus of how real black people should be illuminated in a movie. And this is race... Skin appearance change even with body fat.

And of course, forget about any artistic interpretation of what skin is. Don't even try to use it for your creatures in a fantasy game.

Long story short, i don't think is a good idea. I don't even think is a good idea in unreal or unity. I never used them.

Skin shaders are just pbr materials with lot of layers. They just happens to be extremely difficult to emulate. But that is where the artist and the tech artist have to do the work.

I said it years ago. Godot should be treating creating official 3D demos as a priority. Let's show people how is made instead of keep telling people it is technically possible.

Unity started doing its demos to say "It can be done, but in the unity way". Lets do that. Lets document everything.

We should making a collective effort so one of the best communities ever existed for a game engine can do what the blender team do. It would serve the purpose of finding limitations in Godot itself.

[yosoyfreeman]

I think it can be done with standard PBR.

You can use visual shading to create the layers required for your needs in relatively intuitive way.

Here is a super good example: An incredible looking skin shader made with common pbr workflow inside unity optimized even for mobile devices. Edit: Forgot the link, LOL https://www.youtube.com/watch?v=B0_hvPWq90s You can take a look at the properties and see some of the layers and configuration needed for a setup like that.

But i still think official demos about this topic would help everyone.

Edit 2: Here is super simple setup in blender using the equivalent of a pbr shader.

https://www.youtube.com/watch?v=e7z3PhdCTY8

[and-rad]

I'm gonna shamelessly plug one of the projects I'm working on where I explore Godot's capabilities for rendering digital humans:

https://github.com/RoboPoets/digital_human

So far I've only gotten to the eyes, but the current implementation is close to on par with Unreal's and Unity's eye shaders. I agree that this is nothing that needs to be added to core and instead should be done via plugins. Hair might turn out to be an exception if the goal is to create a strand-based system. This is definitely something I want to look into and it could require core modifications, but I'm not in a position to tell yet.

Until then I think the best this proposal can hope for is a discussion about whether an additional demo project for this topic should be created and if so, what it should contain.

[and-rad]

Don't overhype MetaHuman. The system is neither the best nor the most efficient at what it does. The polycount is way higher than it needs to be and the number of joints is just completely insane. Whenever people go on about how good Metahumans are, I like to point at this tweet from one of the animators working on TLOU2.

The ideal demo project will be like MetaHuman

What exactly do you mean here? MetaHuman is not an Unreal Engine project, it is a software that generates humanoid characters that you can import into Unreal Engine. MetaHuman doesn't really have anything to do with the quality of the shaders used. You can slap a realistic skin shader on a cube and it's going to look just as good.

The Facial Blendshapes will be based on ArKit 52 blendshape Industry standards and nothing ELSE.

This is waaay too dogmatic. Not everyone needs this kind of fidelity and some are going to need even more. This insistence of yours on the One True Way™ to do these things is just not going to work within the greater Godot ecosphere.

Also, I'd like to point out that this proposal (written by you) is only about shaders and rendering, not about animation features.

[and-rad]

That is certainly a more reasonable scope. There's two things I like to clarify though.

Yes, the The polycount is way higher, so as that of reallusion.

Maybe I'm reading this wrong, but are you saying that both MetaHumans and Reallusion characters have a similar polycount? Because that's not correct. CC4 characters have about 20K polys total, while a MetaHuman at LOD0 has 30k polys in the face (not the head, the face) alone. Which just goes to show again that you really don't need these enormous polycounts to get good realism.

Yes, we defer animation outside this discussion. The animation I suggested is less a challenge than the realistic skin.

Not even close. Realistic skin rendering is going to be way easier to achieve. It's basically already a solved problem with Godot's current rendering capabilities, including stuff like wrinkle maps and blood flow. Someone just has to write a good, tweakable shader. Creating animations on the other hand, even just as few as 50 blend shapes for the face, is going to be so much more work.

The way I see it, this is a rough hierarchy from easiest to hardest:

1. Skin
2. Eyes
3. Hair
4. Animation

Eyes include some complex math, but there are enough papers out there to get by. Hair is technologically challenging if you want to go beyond cards, and resources are sparse once you go that way. Animation is artistically challenging because the smallest mistake can easily land you deep in the uncanny valley. It also takes so much time and you need skilled modelers to create the shapes, and a well-thought-out workflow to be able to reuse them across different character models.

EDIT: I have no idea why OP decided to delete all of their comments in this discussion, but it now looks like a bunch of weirdos talking to themselves in here. Honestly, since they removed themselves from the conversation, I'm inclined to ask that this proposal be closed. It feels like we're mostly in agreement anyway.

[yosoyfreeman]

Yeap. Yesterday i spend a considerable ammount of time trying to provide an answer with some value, but OP posted and deleted or edited things so many time that i desisted after an hour. Including posting messages "Only for x user that i will delete soon cause godot admins does not allow feedback" and i surrendered after that sentence. Please close this.

[AThousandShips]

Doesn't explain removing comments from a dialog, the comment was about not spamming with new comments, the reason your comments were deleted before was that they were moved to the proposal body to make it more navigable

All you did was remove all the context of our comments and break any kind of logic to an ongoing discussion

[GeorgeS2019]

@AThousandShips

I have checked and-rad https://github.com/RoboPoets/digital_human

This is exactly one part of what we need for realistic human characters.

I hope @and-rad continues beyond the eye to skin and hair.

Just the eye alone is already a BIG step for the Godot community

The proposal has led us to identify the digital_human repos and generated discussions. To me, this alone is a success that I think will benefit the community.

[yosoyfreeman]

I'm sorry but I'm unable to keep with this conversation. Now there is no context, all our answers won't help anyone in the future reading this. You just marked this as closed and eleven minutes later continued with the conversation.

I just will say that @and-rad work is a demonstration of current Godot capabilities and he/she even said does not find any that needed to be added as core.

That magnificent work is awesome, yes, but releasing it or not is not our choice but the choice of the artist who made them. It may be a plugin. It may be a paid asset. It may be a demo. Or it may be nothing cause he or she wants to keep their work. I feel this is more asking about game assets that is a Godot feature request.

I'm not trying to be rude, I'm just unable to understand what we are even discussing at this point.

[Zireael07]

@yosoyfreeman there's nothing left to discuss here since OP deleted their comments thus rendering this discussion/conversation a mess

[emberlightstudios]

As someone who is working on a character creator for Godot (see here: https://github.com/NitroxNova/Humanizer) I think there are much bigger problems than photorealistic shaders.

We are trying to create a skin shader and running into all kinds of issues mostly stemming from the fact that the skinning pipeline is a black box.

1) For staticbodies (e.g. humanoid statues) we cannot bake a posed mesh to generate a trimesh collider. That data exists only on the gpu and is inaccessible on the cpu side.

2) We are having a hell of a time generating procedural skin materials due to UV seams.
2a) Sure we could use triplanar or 3d noise, but you can't get that data back off the gpu onto a texture which we need to do because 2b) the rest pose or un-skinned vertex positions are not exposed in the vertex shader so we cannot use a shader on an animated human. The skin details do not stay fixed in UV space as the human moves if you use vertex positions.

So our current attempt is to somehow map UV texels to world space positions (basically UV re-wrapping) and then either use a compute shader or due some hacky workaround where we generate the texture map on a quad and take a picture of it with a secondary viewport.

Would like to functionality added in these areas before anything about photorealistic skin shaders.

[GeorgeS2019]

We had Godot 4 makehuman before, deleted due to commercial reason Photorealistic skin is the key problem

[emberlightstudios]

Our plugin is makehuman for godot and it's not deleted.

You can't have photorealism if you can't write a working skin shader due to the skinning pipeline.

[GeorgeS2019]

The deleted is Godot4 c# version of gdscript Godot 3 old version of https://github.com/vikram3/Godot4CharacterCreator

[GeorgeS2019]

The skin shader provided by makehuman is no way close to CC3

[emberlightstudios]

I know about Go-Makehuman-dot. Ours is a much more advanced than that and I'm telling you there are fundamental roadblocks in Godot shaders and the skinning pipeline that have to be addressed before we can make progress here unless we can find some very tedious workarounds.

I don't care about makehuman's skin shader. It's completely irrelevant. It's not compatible with godot anyway. But the MPFB shader is not terrible.

[GeorgeS2019]

fundamental roadblocks in Godot shaders and the skinning pipeline that have to be addressed

https://github.com/RoboPoets/digital_human/issues/3#issuecomment-2084986101

I am not the expert If U know how to articulate the missing parts, please list them and we iterate through to get other more experts to involve Thx for pointing out

[GeorgeS2019]

For trimesh, I heard someone spoke before. Join V-Sekai discord

Speak to iyamma

[emberlightstudios]

I already did in my original post. The main problem is the lack of accessibility to data from the gpu. We need to either be able to implement a fragment shader and grab the resulting texture maps (and preferably skinned vertex coordinates) from the gpu, or we need to have access to un-skinned vertex positions in the vertex shader, preferably the former, or both.

[GeorgeS2019]

https://youtu.be/ft--lVFpx4E?si=yGxtPq9NPVmSmBNM

[emberlightstudios]

There is no body. There are probably no seams. If you only want headless people maybe that can work for you. We also have face tracking animations. Not a big deal.

[Calinou]

1. For staticbodies (e.g. humanoid statues) we cannot bake a posed mesh to generate a trimesh collider. That data exists only on the gpu and is inaccessible on the cpu side.

For animations, there's an open PR: https://github.com/godotengine/godot/pull/85018

For blend shapes, this is available in 4.3 thanks to https://github.com/godotengine/godot/pull/76725. You can bake a mesh from a blendshape mix, then generate a trimesh collision shape based on the baked mesh.

[emberlightstudios]

1. For staticbodies (e.g. humanoid statues) we cannot bake a posed mesh to generate a trimesh collider. That data exists only on the gpu and is inaccessible on the cpu side.

For animations, there's an open PR: godotengine/godot#85018

For blend shapes, this is available in 4.3 thanks to godotengine/godot#76725. You can bake a mesh from a blendshape mix, then generate a trimesh collision shape based on the baked mesh.

Good to know. For us, we are using makehuman target files, so we already have the mesh data for blendshapes. We just need the posed skeleton data. Although this idea of humanoid statues is a relatively minor feature which could be easily exported from blender instead. This feature wouldn't be enough to help us with the skin shader problem.