Changing Diffusion Model

AndreyVLD commented 3 months ago

How can I change the core diffusion model: stabilityai/stable-diffusion-xl-base-1.0 to some other model?

gjnave commented 3 months ago

did you figure this out yet?

AndreyVLD commented 3 months ago

I think I have changed the core model to something else by following their Hugging Face repository (https://huggingface.co/yisol/IDM-VTON/tree/main) and making a new hugging face repo with the following changes:

swapping the tokenizers, schedulers, encoders, unet and vae to fit the new core model. These were found in the new model's Hugging Face repo.
in the model_index.json I have switched the line "_name_or_path": "stability/stable-diffusion-xl-base-1.0", to the repository of the new model. In their code, when loading the models instead of specifying the base path to be yisol/IDM-VTON I have changed it to my own Hugging Face repository.

The output of the model is changed a little bit in terms of generation so it seems that the switch was successful but I am not entirely sure.

osmanyz commented 1 month ago

@AndreyVLD if you don't mind, could you please share the repository that you used for these changes?

yisol / IDM-VTON

Changing Diffusion Model #91