search

gligen / GLIGEN

Open-Set Grounded Text-to-Image Generation
MIT License
1.91k stars 145 forks source link

Convert checkpoint to Diffusers checkpoint #49

Open tuanh123789 opened 10 months ago

tuanh123789 commented 10 months ago

Great Work! I see that with checkpoint text box and inpainting text checkpoint can use in Diffusers library and I want to use other checkpoint with Diffusers too. How can I convert your original checkpoint to Diffusers checkpoint. Thank you !

k15201363625 commented 9 months ago

+1

tuanh123789 commented 9 months ago

+1

https://github.com/huggingface/diffusers/blob/main/scripts/convert_gligen_to_diffusers.py I have implement this and summit it on diffusers. You can check this

zengjie617789 commented 9 months ago

@tuanh123789 hello, I wonder that the netework of GLIGEN is different from the diffusers and how to convert it successfully? As far as i know, it added new parameters in that, after converted, how to use them in diffusers?

GLIGEN: The way GLIGEN introduces new parameters is by adding a gated self-attention layer between the self-attention and cross-attention layers within the encoder. The details of the gated self-attention are depicted in the left part of the following figure.

tuanh123789 commented 9 months ago

@tuanh123789 hello, I wonder that the netework of GLIGEN is different from the diffusers and how to convert it successfully? As far as i know, it added new parameters in that, after converted, how to use them in diffusers?

GLIGEN: The way GLIGEN introduces new parameters is by adding a gated self-attention layer between the self-attention and cross-attention layers within the encoder. The details of the gated self-attention are depicted in the left part of the following figure.

You can check this: https://huggingface.co/anhnct/Gligen_Text_Image

zengjie617789 commented 9 months ago

Thanks yout shared link and helped me a lot. Furthermore, is it the diffusers deployment of GLIGEN? Is each model same from the officel model which is based on SD1.4? And is it possible to change the unet or vae to another when i load that? I will be appreciate if any help provided.

tuanh123789 commented 9 months ago

Thanks yout shared link and helped me a lot. Furthermore, is it the diffusers deployment of GLIGEN? Is each model same from the officel model which is based on SD1.4? And is it possible to change the unet or vae to another when i load that? I will be appreciate if any help provided.

Yes,it is the version using in diffusers of GLIGEN. The weights of model is converted from offical checkpoint by me :D :D :D