Is there any convenient way that I can color my own generated character model?

[Marianoeli]

Thanks for your apealling work,sir.The character model being generated has beyond my preliminary imagination,better than other models I 've used before.It certainly will be way more better if I am capable of coloring it. So I wanna ask that is there any convenient way that I can color my own generated character model or get the model texture and apply to it?I would be very grateful if you could answer.Thanks

[Yuanhong200727]

I met the same question.Can I color the exporting model according to the raw photo or picture?

[YuliangXiu]

I updated the infer.py#L532 with query_color from original image, but this could only provide front-side texture, as for the back-side colors, I set normal color.

PIFu[1] and PaMIR[2] uses another network to predict back-side colors from front-side colors, but always lead to blurry and unrealistic back-side textures. I am thinking about using the depth2img feature in Stable Diffusion[3] to inpaint the full colors. But this is still under construction, will let you know once finished. I would leave this issue open until the texture issue is solved completely.

[1] Saito, Shunsuke, et al. "Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019. [2] Zheng, Zerong, et al. "Pamir: Parametric model-conditioned implicit representation for image-based human reconstruction." IEEE transactions on pattern analysis and machine intelligence 44.6 (2021): 3170-3184. [3] Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. https://huggingface.co/stabilityai/stable-diffusion-2-depth

[Yuanhong200727]

Thank you so much,sir. But when I ran the new demo in infer.py,an error occured when rendering the final mesh: The error occured both in my own photo and the initial photo given.

[YuliangXiu]

Thank you so much,sir. But when I ran the new demo in infer.py,an error occured when rendering the final mesh: The error occured both in my own photo and the initial photo given.

I have updated the code with torch.tensor wrapper. The query_color is almost the same as ICON's query_color.

[YuliangXiu]

@Marianoeli @Yuanhong200727

I have a try on Blender's add-on Dream-Textures under depth2img mode, and it gives me some promising textures.

Front Prompt: asian man dressed in red blazer, blue jeans and brown boots Back Prompt: asian man dressed in red blazer, blue jeans and brown boots, and back to the camera

Front	Back

Apparently, the 1) face/hands 2) front+back consistency could be improved further, but I think this is a promising way to synthesize the textures you want.

[Yuanhong200727]

Thank you so much,sir!I will have a try!

[YuliangXiu]

Here is a highly related paper which shares the same insight but with much more promising results

https://texturepaper.github.io/TEXTurePaper/

[Yuanhong200727]

Here is a highly related paper which shares the same insight but with much more promising results

https://texturepaper.github.io/TEXTurePaper/

Awesome work！Thanks for your sharing

[YuliangXiu]

@Marianoeli @Yuanhong200727

Please check https://github.com/YuliangXiu/ECON#texture to texturify the ECON reconstruction with full texture.

[YuliangXiu]

@Marianoeli @Yuanhong200727

We recently released a new work, TeCH. Given a single image, TeCH could produce a full textured avatar with both intricate geometric details and consistent high-quality texture.

Homepage: https://huangyangyi.github.io/TeCH/ Code: https://github.com/huangyangyi/TeCH