How to render objects using phong shader instead of depth map?

[davidfant]

Moving question from https://github.com/Unity-Technologies/com.unity.perception/issues/571 to here!

Regarding your followup questions:

Do you want all objects in the scene to be rendered with this material? Or just the body? Only the body. This goes a bit hand in hand with my other question here https://github.com/Unity-Technologies/com.unity.perception/issues/571, and I'll try follow your steps to only render the body that way. What I'm unsure about is how to make it render like the image in the original issue (rather than e.g. a depth map)

@csun

[csun]

Thanks for moving this issue. I just want to clarify a couple things so that I can provide the best possible answers to your question.

It sounds like you already have a depth labeler set up, and we've discussed how to get that to ignore clothing in your other issue. When you set up perception to output a depth map, it can do so in addition to outputting "standard" color images of the scene.

In your question, you're asking about rendering using a phong shader instead of a depth map. These two don't serve the same purpose - you will not be able to extract depth information from an image rendered using phong shading.

My current interpretation is that you want your scene to render two images. The first image is a black and white phong shaded version of a person with no clothes and no hair. The second image is a depth map of the same person, also with no clothes and hair. Is this the correct interpretation? If not, could you please provide a list of output images you're hoping to generate, as well as whether clothes and hair should be ignored in each one?

[davidfant]

Really appreciate the help!

This is roughly what I'm after:

1. RGB images (which I'm getting already). This should be RGB, with clothes, etc
2. A labeler that outputs these kinds of gray 3D models as a new type of ground truth. This should be purely the human body , without hair and clothes

I ultimately want to have images that have a mapping somewhat like this:

I don't really need the depth map. Rather, I'm trying to create a new kind of ground truth labeler that outputs this 3D shape of the human body. Does that make sense?

For context, what I'm trying to do is train a network to predict body shape from an RGB image.

[csun]

Thanks for the explanation, this makes everything a lot more clear to me!

The "best" way to do this is probably to write a custom labeler in perception that ignores hair and clothes (using the layer system we discussed in your other issue) and then renders all objects with the simple gray material you're describing. Unfortunately, I'm not the most knowledgeable on how to write a custom labeler like that. If you opt for this you may want to read up on the perception labeler docs, or dig into the labeler code yourself.

The "easiest" way to do this is probably to actually do this in two passes with no labelers, just outputting the RGB image. In your human generation config, set it so that humans only refresh every two frames. Then, write a custom MonoBehavior and stick it on your base prefab (which you set in the human generation config). Finally, write some code in that MonoBehavior that does nothing for the first frame the object is alive, then deletes the hair and clothing objects in the second frame and sets the body SkinnedMeshRenderer to use your custom gray material. To find the hair and clothes for deletion, you can do a mix of accessing the SingleHumanGenerationAssetRefs and searching for children by name. The hair child object should always have the same name, so it should be pretty easy to destroy using transform.Find().

Please let me know if you need any clarification on either of those options! Best of luck.