search

openai / glide-text2im

GLIDE: a diffusion-based text-conditional image synthesis model
MIT License
3.53k stars 500 forks source link

Question about the CLIP model #37

Open Vanint opened 2 years ago

Vanint commented 2 years ago

Hi, thanks for your great work!

I found that you release several checkpoints, including CLIP ( "clip/image-enc": "https://openaipublic.blob.core.windows.net/diffusion/dec-2021/clip_image_enc.pt", "clip/text-enc": "https://openaipublic.blob.core.windows.net/diffusion/dec-2021/clip_text_enc.pt").

Are these checkpoints trained with noised images, or are they public CLIP models?

Best wishes,

woctezuma commented 2 years ago

See; https://github.com/openai/glide-text2im/blob/main/model-card.md#datasets

Datasets

GLIDE (filtered) was trained on a filtered version of a dataset comprised of several hundred million text-image pairs collected from the internet. We constructed a set of filters intended to remove all images of people, violent objects, and some and hate symbols (see Appendix F of the paper for details). The size of the dataset after filtering was approximately 67M text-image pairs.

Our noised CLIP model which was trained on the dataset described above, augmented with a filtered version of the dataset used to train the original CLIP models. The total size of this augmented dataset is approximately 137M pairs.

Moreover, the original CLIP is from January 2021, while the checkpoints are all from December 2021 (according to the URL), so newly trained (a priori). If the checkpoints were the old CLIP ones, I believe the URL would have been older, because there would be no need to re-host the models at a new location. It is still possible that the new URL is a redirection, but I find it unlikely for the aforementioned reasons.

https://github.com/openai/glide-text2im/blob/69b530740eb6cef69442d6180579ef5ba9ef063e/glide_text2im/download.py#L10-L17

Vanint commented 2 years ago

See; https://github.com/openai/glide-text2im/blob/main/model-card.md#datasets

Datasets

GLIDE (filtered) was trained on a filtered version of a dataset comprised of several hundred million text-image pairs collected from the internet. We constructed a set of filters intended to remove all images of people, violent objects, and some and hate symbols (see Appendix F of the paper for details). The size of the dataset after filtering was approximately 67M text-image pairs. Our noised CLIP model which was trained on the dataset described above, augmented with a filtered version of the dataset used to train the original CLIP models. The total size of this augmented dataset is approximately 137M pairs.

Moreover, the original CLIP is from January 2021, while the checkpoints are all from December 2021 (according to the URL), so newly trained (a priori). If the checkpoints were the old CLIP ones, I believe the URL would have been older, because there would be no need to re-host the models at a new URL. It is still possible that the new URL is a redirection, but I find it unlikely for the aforementioned reasons.

https://github.com/openai/glide-text2im/blob/69b530740eb6cef69442d6180579ef5ba9ef063e/glide_text2im/download.py#L10-L17

I agree with you. Thanks~