search

giuvecchio / matfuse-sd

MatFuse: Controllable Material Generation with Diffusion Models (CVPR2024)
https://gvecchio.com/matfuse/
MIT License
31 stars 2 forks source link

Lowering VRam #1

Closed Night1099 closed 4 months ago

Night1099 commented 5 months ago

Hello gluvecchio! Your work looks very exciting and I want train my dataset with it.

The only huddle I see is the inference vram requirement, is there a new feature on the way to help this, or a way to use lower precision, thanks

Night1099 commented 5 months ago

I just now saw your mat forger project, is there plans to release how to train that type of model or further optimize it?

giuvecchio commented 5 months ago

Hi! About vram requirements, you can run the model at half precision and prune unnecessary parts of it. There's an awesome repository that does it for you: https://github.com/lopho/stable-diffusion-prune Regarding MatForger, it can be trained with this same codebase just switching the autoencoder architecture to the VAE. You can also finetune it directly in diffusers: https://huggingface.co/docs/diffusers/tutorials/basic_training