search

astra-vision / MaterialPalette

[CVPR 2024] Official repository of "Material Palette: Extraction of Materials from a Single Real-world Image"
MIT License
233 stars 10 forks source link

Is the "displacement" useful? #14

Open zhang0227 opened 2 weeks ago

zhang0227 commented 2 weeks ago

I found multiple instances of “displacement” in the code. This will cause an error in the code. Is this part useful or not?

wonjunior commented 2 weeks ago

Hi, this was part of an extension I was working on but it didn't make it to the paper. The idea was to output an additional displacement map and supervise that branch to improve the material quality. The current implementation does not require that key do you can discard it!

wonjunior commented 2 weeks ago

Can you let me know which file and line threw the error? Thanks!

zhang0227 commented 2 weeks ago

Can you let me know which file and line threw the error? Thanks! You can search “displacement” in these files to get which line. I add a switch solve the error. capture/callbacks/visualize.py capture/render/main.py capture/source/loss.py

zhang0227 commented 2 weeks ago

I have some problem when finetuning the network for a further 100 epochs on AmbientCG and TexSD, when compute loss (capture/source/loss.py line105) if self.use_target: for k, w, loss_fn in self.weights: loss[f'tgt_{k}'] = self.pl_reg_weight*loss_fn(y_tgt[k], x_tgt[k]) the 'x_tgt' keys is dict_keys(['dir', 'name', 'path', 'input']), but it needed dict_keys(['albedo', 'roughness', 'normals']) .Is the dataset class StableDiffusion is wrong version or I need to preprocess the TexSD before training start?

wonjunior commented 2 weeks ago

Indeed, TexSD only contains the texture images. To train the fine-tuning stage, you first need to output pseudo-labels using the checkpoint from the pertaining stage. Then you must load the labels in capture/data/target.py.

For your information, if you are interested in using our published weights, you can do so by downloading this release.