search

MrForExample / ComfyUI-3D-Pack

An extensive node suite that enables ComfyUI to process 3D inputs (Mesh & UV Texture, etc) using cutting edge algorithms (3DGS, NeRF, etc.)
MIT License
2.19k stars 219 forks source link

Where can I find models of T2I V2 and V3? #311

Open jakechai opened 4 days ago

jakechai commented 4 days ago

Where can I find models of CRM T2I V2 and V3? missing

MrForExample commented 3 days ago

Haha, you are the first to noticed :) that's a PBR textures generation model I'm researching and training myself, and haven't finish it yet, but almost, I may release it after a month or two along with research paper for the whole development process.

Given a 3D mesh and a reference image, it can generate you a 3 texture maps (albedo, metalness, roughness), To show you a few testing results:

https://github.com/user-attachments/assets/82ca0abe-0ac6-4c32-ae09-b0419baa6bd7

https://github.com/user-attachments/assets/057d90d8-3f59-410e-92a6-621dd59a1b07

https://github.com/user-attachments/assets/7cb6a3d8-e500-4c6a-839c-69f41b839b4e

Cheers, have a good day 👍

jakechai commented 3 days ago

Great! I can't wait to try it. I have some thoughts about your texture projection process. We don't need to make 2nd mesh and texture refinement in 3D software if you could make all these happen.

  1. Can you add an orthographic camera switch for Mesh Orbit Renderer and Explicit Target Color Projection? In some cases, orthographic projection is much better than perspective one. That is a 90 FOV orthographic camera with a Target Distance of 1.0 and a orbit radius of 2.0 based on a 1.0 unit cube, which is the CRM algorithm (the following image). This is based on my experiences on the UE texture projection project. crm camera

https://github.com/user-attachments/assets/28311bdb-ea41-4904-a202-452a598fcddb

  1. Is it possible to bake all the PBR textures such as Albedo, Roughness, and Normal, (Metallic, AO, Cavity, etc.) at the same time? All of them can be converted from the Albedo texture using different methods. One of my favorites is Materialize. materialize

  2. Because we need mesh optimization before the texture projection process. Can you bring Mesh Anything the powerful mesh optimization algorithm into the 3D Pack world? This will make the 3D pack a more complete and practical toolkit for mesh reconstruction.

GeorgeSte commented 2 days ago

Those examples looks awesome! Iv been wondering about the CRM T2I V3 :D Plus the requirements needed. Im guessing its a normal map and single image of the subject? or MV images and MV normals? saw the example workflow having a image bath loaders. Keen to try it out. Also fantastic points from Jake, would be wonderful to have orthographic camera switch for Mesh Orbit Renderer and Explicit Target Color Projection. Had issues where the top of a mesh wouldn't have color projected on its only possible to run images with 0 angle as far as I know.

Exiting development on this all!

MrForExample commented 1 day ago

Hi @jakechai

  1. I'll see if I can add that orthographic camera switch for Mesh Orbit Renderer and Explicit Target Color Projection
  2. Right now, the Mesh class can only support single UV and single Albedo, single Metallic (Blue channel) & Roughness (Green channel) map, so as right now, Explicit Target Color Projection node support those three types of maps, I do have plan to extend the Mesh class's functionality more in the future
  3. I have tried Mesh Anything multiple times before, so it's my colleagues, for both v1 and v2, but we all think it doesn't work as robust as advertised, most meshes are kind of unusable, so that is on hold as for now.

Hi @GeorgeSte thanks for the kind words, it uses 6 orthogonal MV normals to condition the shape, and one front view images to condition the appearance, and generate 6X3 MV views for Albedo, Metallic and Roughness respectively