ljleb
commented
1 year ago FWIW, this is what chatGPT returned me:
image_path = r"C:\path\to\image.png"
import torch
from PIL import Image
from torchvision.transforms.functional import resize
from transformers import CLIPProcessor, CLIPModel
import numpy as np
image = Image.open(image_path)
image = resize(image, size=(512, 512))
image = np.array(image)
# convert PIL image to numpy array and normalize
image = [torch.tensor(image).permute(2, 0, 1).float() / 255.0]
# shape: (1, C, H, W)
processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")
text = ["image"]
input_dict = processor(text=text, images=image, return_tensors="pt")
# pass the image through the model
model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
outputs = model(**input_dict)
# print the embeddings
print(outputs.vision_model_output.last_hidden_state.shape)But this script returns torch.Size([1, 257, 1024]). The embed has shape 1024 instead of 768, so I'm not sure this is going to work.
missionfloyd
Ehplodor
ClashSAN
PladsElsker
Is there an existing issue for this?
What would your feature do ?
As CLIP models map both images and text into a common latent space, I was wondering: what if, instead of encoding text for conditioning during sampling, we used an image?
I don't know much about machine learning and transformers, but I tried asking chatGPT to see if this would be possible. With what it replied with, I'm now certain that either 1) it doesn't know or 2) I don't know how to ask.
I know that SD checkpoints only include a text encoder, and no image encoder. I'm still curious, are there any model that could be used to map an image to an ok-ish clip space? (i.e. something that resembles the text->clip encoder of the SD1.5 ckpt for example) How hard would it be to train a model to do this if no such model exists?
Proposed workflow
Image embeddingcheckboxAdditional information
No response