Model Inference Output Issue | Base Color not Showing

[UsamaShami11]

Hi, kudos for the amazing work, and the results shown in paper looks promising! However, I'm facing issues while generating/reproducing the exact output as highlighted in the results. I have created a docker image, and uploaded the provided controlnet weights, as suggested by you.

Now, when I run the inference, the generated 3D model lacks color. Seems like the PBR materials are not generating properly, especially the Albedo output (3rd image in gif), which must include the base color. Here is the 'a brown basketball' evaluation.gif.

I am using this command (same as guided), so I don't know where the issue lies. Maybe the trained model is not working properly, or there is some parameter related issue. python launch.py --config configs/dreammat.yaml --test --gradio --gpu 0 system.prompt_processor.prompt="A brown basketball" system.geometry.shape_init=mesh:load/shapes/objs/basketball.obj system.geometry.shape_init_params=0.6 data.blender_generate=true

I'm aiming to achieve these results:

Please help me resolve the issue, as I am doing research in the domain of 3D and really impressed by your work. I am looking forward to getting a quick response from your end, so I can incorporate your work in my pipeline accordingly. Thanks!

[zzzyuqing]

It seems that the values of albedo, roughness, and metallic are still at their initial settings. Could you please check if the material representation is indeed being optimized?

[UsamaShami11]

I appreciate your prompt response! Basically, after setting up the repo, I just ran the DreamMat inference code from .cmd/run_examples.sh on the provided Objects (3D meshes) located in the ./load/shapes/objs. Ideally, the code shall work because I tried my best to ensure all dependencies/packages are installed. Can you please confirm from the Inference Command I'm using, if the issue can be probed from there? Or let me know any .py file, where I may dig further.

One more thing, does it have something to do with the system.geometry.shape_init_params=0.6 or any other parameters being passed in the inference command (which I pasted earlier)? Actually, the commands in cmd/run_examples.sh are for training and not testing, and I myself had to figure out what the inference commands are. So, I'm not sure if the command I used, will generate the desired results. Any guidance in this perspective would be beneficial regarding the issue observed.

[zzzyuqing]

Previously, someone encountered a similar problem when using Docker images, and he mentioned that pulling the image again solved the issue. Although I haven't personally experienced this, you might want to try setting up the environment directly by installing packages in requirements.txt as a possible solution.

Indeed, the cmd/run_examples.sh script is used to generate PBR materials. You can run it directly, and this issue should be unrelated to the parameter settings.

[UsamaShami11]

Previously, someone encountered a similar problem when using Docker images, and he mentioned that pulling the image again solved the issue. Although I haven't personally experienced this, you might want to try setting up the environment directly by installing packages in requirements.txt as a possible solution.

I switched to building a Dockerfile since the Conda environment was not being setup, due to the issues caused by the dependencies which are based on other GitHub repos e.g.

git+https://github.com/KAIR-BAIR/nerfacc.git@v0.5.2
git+https://github.com/NVlabs/tiny-cuda-nn.git#subdirectory=bindings/torch
git+https://github.com/NVlabs/nvdiffrast.git
git+https://github.com/ashawkey/envlight.git

The Dockerfile had some issues too, but they were less compared to Conda environment. I can try 2 things if the issue is related to Environment Setup

1. Try setting up Anaconda Environment once again (and ask for specific issues observed during setup)
2. Try pulling the Dockerfile you hosted on Docker Hub, since I built the Docker Image from the Dockerfile you provided, currently.

Indeed, the cmd/run_examples.sh script is used to generate PBR materials. You can run it directly, and this issue should be unrelated to the parameter settings.

About this, if you check the file, by opening the following URL: https://github.com/zzzyuqing/DreamMat/blob/bd33e9122d9f7c05734b5f419af2cd0d621b9f05/threestudio_dreammat/cmd/run_examples.sh

The --train param is passed, which will not initiate the Inference (Testing) rather the full fledge Training (proof below). It must be --test to run the Inference (like I showed earlier)

Last, I don't think it is an Environment related issue rather a Command/Code related issue, where the model is getting some default parameters, instead of the actual ones (as you stated). And no matter you start Training or Testing, both commands run the code, which means my environment has been set up properly.

Anyways, please keep this issue open, till I experiment further keeping in view your suggestions. I'll update you accordingly. Thanks!

[zzzyuqing]

The --train parameter is used for score distillation sampling to train the material representation network, while the --test parameter is solely for rendering some visual results from the trained material representation network.

Could you please specify what exact problems you encountered when installing other GitHub repositories in the Conda environment? One potential solution could be to try cloning the repositories and installing them.

[UsamaShami11]

I apologize for getting back to you a bit late, as I missed your message and was a bit busy. Mainly while setting up the Conda environment, the GitHub repos I mentioned in my previous comment (which are actually being installed by running pip install -r requirements.txt) were creating issues. I even tried to install them separately, but there were some issues. I'll try to set up the Conda environment again today, and update the exact issue/error here.

Moreover, my main concern is regarding reproducing the exact inference results, as discussed earlier, and which I feel is due to some initialized value being passed (as stated by you earlier) or some other code-related issue (and NOT environment issue).

Can you please comment if the Inference for both the steps, i.e. PBR Material Generation and the output generation (3D textured model) is being done in 1 step via the command below, OR it requires running multiple commands?

python launch.py --config configs/dreammat.yaml --test --gradio --gpu 0  system.prompt_processor.prompt="A brown basketball" system.geometry.shape_init=mesh:load/shapes/objs/basketball.obj system.geometry.shape_init_params=0.6 data.blender_generate=true

[zzzyuqing]

I apologize for getting back to you a bit late, as I missed your message and was a bit busy. Mainly while setting up the Conda environment, the GitHub repos I mentioned in my previous comment (which are actually being installed by running pip install -r requirements.txt) were creating issues. I even tried to install them separately, but there were some issues. I'll try to set up the Conda environment again today, and update the exact issue/error here.

Moreover, my main concern is regarding reproducing the exact inference results, as discussed earlier, and which I feel is due to some initialized value being passed (as stated by you earlier) or some other code-related issue (and NOT environment issue).

Can you please comment if the Inference for both the steps, i.e. PBR Material Generation and the output generation (3D textured model) is being done in 1 step via the command below, OR it requires running multiple commands?

python launch.py --config configs/dreammat.yaml --test --gradio --gpu 0  system.prompt_processor.prompt="A brown basketball" system.geometry.shape_init=mesh:load/shapes/objs/basketball.obj system.geometry.shape_init_params=0.6 data.blender_generate=true

No，you should run this command for PBR material generation and output generation.

python launch.py --config configs/dreammat.yaml --train --gradio --gpu 0  system.prompt_processor.prompt="A brown basketball" system.geometry.shape_init=mesh:load/shapes/objs/basketball.obj trainer.max_steps=3000 system.geometry.shape_init_params=0.6 data.blender_generate=true

[UsamaShami11]

No，you should run this command for PBR material generation and output generation.

python launch.py --config configs/dreammat.yaml --train --gradio --gpu 0  system.prompt_processor.prompt="A brown basketball" system.geometry.shape_init=mesh:load/shapes/objs/basketball.obj trainer.max_steps=3000 system.geometry.shape_init_params=0.6 data.blender_generate=true

Okay, actually the flag --train was a bit confusing, and it was taking a bit longer to execute, which is why I thought it is training the model instead of inference. I'll try it and get back to you then. Thanks!

Just let me know in the meantime, that on average, how much time it should take to generate the PBR materials for 1 example, on a RTX 3090 (24 GB) GPU?

[zzzyuqing]

The training process, which pertains to the SDS procedure, indeed takes some time. Pre-rendering with Blender typically requires around 15 minutes, but this duration may vary depending on your CPU performance. When the pre-rendering is done, the generation process should be completed within 20 minutes on a RTX 3090 GPU.

[UsamaShami11]

Alright, thanks for the clarification! Right now, the command you shared, is working. I'll update you here, once outputs are generated. Moreover, I'll try to set up Conda environment as well, and let you know regarding exact issues.

[UsamaShami11]

Hi again, I still haven't been able to set up the Conda environment since I was busy, so please keep the issue open till then. Moreover, I was able to generate the outputs for the provided examples, as shown in the paper.

Right now, I have 3 small questions, in which I need your assistance.

1. During inference, I'm using the command provided by you. Although the inference proceeds smoothly and generate results, there is a small warning/error shown in between, which highlights that the ControlNet weights file shall be a .safetensors file, but the provided weights file in Part 3 of section Preparation for inference are diffusion_pytorch_model.bin, hence it gives the File Not Found Error (image attached)

_Do you think it will take less time, if the diffusion_pytorch_model.safetensors is provided, or inference time will stay the same?_

2. Whenever I use a different prompt for the same object (to generate its variation) while keeping the data.blender_generate flag set to false, it throws an error, as the pre_render directory doesn't have all the required folders. Then I have to either copy the pre-render directory from the older output folder of the same object, to the new one OR infer it by setting the flag to true. What is the benefit of this flag disabling feature, if we can't even generate different variants of the same object by setting it as False?

3. I am generating textures for my custom objects. I really love your model, however, some of the results aren't up to the mark. I want to know what is the intuition behind setting the system.geometry.shape_init_params value to 0.6, 0.7 or 0.8, since it's different for each prompts in run_examples.sh? Moreover, please tell the same for system.guidance.cond_scale being changed from default value. My question is that what effect these params have on the output, and how do you set it for each example?

[zzzyuqing]

1. The storage format differs. When training your own ControlNet, you can use the .safetensors format for storage, but this does not affect usage.
2. The setting of data.blender_generate was designed for my convenience in tuning parameters, such as different iteration counts. If you want different text prompts all in the same folder without having to re-render, you can name the result folders after the mesh names, which eliminates the need for manual copying.
3. In most cases, you can directly use the default parameters: system.geometry.shape_init_params=1.0 and system.guidance.cond_scale=1.05. The parameter shape_init_params scales the size of the input mesh, while cond_scale is the weight in csd.

[UsamaShami11]

1. The storage format differs. When training your own ControlNet, you can use the .safetensors format for storage, but this does not affect usage.

Alright!

2. The setting of data.blender_generate was designed for my convenience in tuning parameters, such as different iteration counts. If you want different text prompts all in the same folder without having to re-render, you can name the result folders after the mesh names, which eliminates the need for manual copying.

Thanks for the tip! Will try to explore it further, and test it!

3. In most cases, you can directly use the default parameters: system.geometry.shape_init_params=1.0 and system.guidance.cond_scale=1.05. The parameter shape_init_params scales the size of the input mesh, while cond_scale is the weight in csd.

Thanks a lot for the insight, I was wondering today why the mesh size of output changed 😅, so it makes sense now! Will try to experiment with those as well!

[UsamaShami11]

Hi again, Yuqing, I have queries from 2 perspectives, one is regarding outputs, and the other is regarding error regarding Conda env creation.

1. Regarding the outputs, I have been able to generate textures for some custom objects:
   • Kindly tell if the generated texture isn't up to the mark, shall we experiment with cond_scale ?
   • Regarding the system.geometry.shape_init_params, I tried all the values 0.7, 0.8 and even 1.0. It turned out that 0.7 or maybe 0.8 are the best options, as 1.0 resized the Mesh to quite large size and the PBR material didn't align with the Mesh.
     • Does the DreamMat generate its own mesh in addition to Texture, since the Output Mesh is a bit low-poly and Altering the Shape parameters, generated objects of varying sizes, as verified in blender?
     • Also, does it alter UV maps, because I tried wrapping the generating PBR materials over the original mesh, but it didn't fit well

• Is the code modular, in case someone wants to replace the underlying model. Since the Stable Diffusion, doesn't create that many accurate textures, for specific prompts, would it be possible to replace it with a better option, or give image at input (along with text) for conditioning the texture generation?

2. Shall I create a different issue for environment setup related issues (to avoid cluttering) or ask here? As stated in https://github.com/zzzyuqing/DreamMat/issues/20#issuecomment-2293143851, the main issues arise during installation of these repos. Moreover, there are some differences even in the order of installation of setup, is you see the Dockerfile and instructions for Conda env setup. I have tried to resolve some issues, but even installing each and every single repo causes issues of its own, which didn't occur much in case of Dockerfile (although I had to modify it as well)

I would love to hear your opinion on the above-mentioned questions, thanks!

[zzzyuqing]

About the quality of the generated textures not being "up to the mark," I'm not entirely sure what specific issues you're encountering. While I cannot guarantee that adjusting cond_scale will definitely improve the quality, since DreamMat isn't a perfect material generation scheme, I would be very interested in seeing some failure cases. These could provide valuable directions for future improvements.

The scaling adjustments we make are applied on a normalized mesh. The determination of scale primarily depends on the shape of the object to ensure it occupies a reasonable proportion of the overall scene. I appreciate your feedback on the parameters, and I agree that 0.7-0.8 seems suitable for most models.

We do not alter the topology of the input model; we only change the model's scale. For models without UVs, we use xatlas to generate UV mapping.

The underlying model can be replaced in the config file. For image-guided texture generation, please look forward to our work at SIGGRAPH Asia.

Regarding the environment setup issues, feel free to create a new issue for that. You might also find solutions in the closed issues if others have encountered similar problems.

Thank you for your questions, and please do not hesitate to contact me if you have more.

[UsamaShami11]

About the quality of the generated textures not being "up to the mark," I'm not entirely sure what specific issues you're encountering. While I cannot guarantee that adjusting cond_scale will definitely improve the quality, since DreamMat isn't a perfect material generation scheme, I would be very interested in seeing some failure cases. These could provide valuable directions for future improvements.

I agree that it isn't perfect, but it still is one of the best available model. I'll definitely share the failure cases. Mostly either the Generated Texture isn't up to the mark as the mesh is also a bit of low polygon (maybe due to normalization) and in some cases, it takes even more memory than 24 GB (for inference) e.g. cube/box shapes, giving Error: CUDA Out of Memory. Some other cases include swapping the front and back side textures, like on a clock, it generated face on the back side of it. Moreover, mostly it requires a lot of prompt engineering, but in some cases a simple text prompt suffices.

The scaling adjustments we make are applied on a normalized mesh. The determination of scale primarily depends on the shape of the object to ensure it occupies a reasonable proportion of the overall scene. I appreciate your feedback on the parameters, and I agree that 0.7-0.8 seems suitable for most models. We do not alter the topology of the input model; we only change the model's scale. For models without UVs, we use xatlas to generate UV mapping.

Thanks for the clarification. And yes, I was considering to apply the generated PBR material to the original mesh, but it doesn't fit properly on that, which is why I asked. I would like to know more about xatlas, are you using it in DreamMat for objects with no UV, or it's a separate study/experiment of yours?

The underlying model can be replaced in the config file. For image-guided texture generation, please look forward to our work at SIGGRAPH Asia.

I will check it for exploration and let you know for any queries. Upon some research, I believe it's an upcoming event, so is there any arXiV or pre-release of that work available?

=============================================

Regarding the environment setup issues, feel free to create a new issue for that. You might also find solutions in the closed issues if others have encountered similar problems.

About the Environment Issues, thankfully, after a tedious effort, I managed to set it up at last, successfully. There were some CUDA/GCC version incompatibility issues at my server/machine side, but I feel some further instructions may be added, in the provided guidelines for DreamMat too. I'll let you know, after documenting them, if I found any major change. And yes, I checked all the issues too, they did help to some extent, initially.

Thanks once again for the detailed answers to my queries!