Open BIGJUN777 opened 2 years ago
shall we need to normalize the CLIP embedding before inputting it to the model?
I downloaded several images from laion2B-en-aesthetic and tried to use the CLIP model (ViT-L/14) to get the clip embedding and take it as input to the NSFW detector. However, the results were different from those shown on laion2B-en-aesthetic.
Yes you need to normalize the embeddings in input
On Thu, Aug 25, 2022, 09:48 BIGJUN777 @.***> wrote:
I downloaded several images from laion2B-en-aesthetic https://huggingface.co/datasets/laion/laion2B-en-aesthetic and tried to use the CLIP model (ViT-L/14) to get the clip embedding and take it as input to the NSFW detector. However, the results were different from those shown on laion2B-en-aesthetic https://huggingface.co/datasets/laion/laion2B-en-aesthetic.
— Reply to this email directly, view it on GitHub https://github.com/LAION-AI/CLIP-based-NSFW-Detector/issues/4#issuecomment-1226901990, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAR437UBH2TZIBVBUJKO5TLV24QMLANCNFSM57RNSXEA . You are receiving this because you are subscribed to this thread.Message ID: @.***>
Yes, I tried to leverage the normalization from improved-aesthetic-predictor.
def normalized(self, a, axis=-1, order=2):
l2 = np.atleast_1d(np.linalg.norm(a, order, axis))
l2[l2 == 0] = 1
return a / np.expand_dims(l2, axis)
def __call__(self, clip_embs):
if isinstance(clip_embs, torch.Tensor):
clip_embs = self.normalized(clip_embs.cpu().numpy())
return self.model.predict_on_batch(clip_embs)
But the results still could not match your results shown on laion2B-en-aesthetic. I checked the img_embs in your provided dataset, and their data type is float16. I tried to use fp16 inference to get the float16 embeddings, but the results were still not correct. Did I miss something? Thanks.
with autocast(enabled=fp16_model):
clip_embs = extractor.encode_image(data['image'].cuda())
I constructed the CLIP model as following:
class ClipExtractor(nn.Module):
def __init__(self, model_name="ViT-L/14", jit=False):
super().__init__()
self.model, self.transform = clip.load(model_name, device='cpu', jit=self.jit)
@torch.no_grad()
def encode_image(self, images):
with torch.no_grad():
images_embeddings = self.model.encode_image(self.transform(images))
return images_embeddings
@rom1504 @christophschuhmann Any ideas on my problems? Thanks.
Hi there,
When using this model, I took safe images as inputs but got opposite results. The code I constructed is almost the same as yours.
I encountered several warnings when inferring. My running environment: autokeras==1.0.19 and tensorflow==2.9.1. Did I miss something?![image](https://user-images.githubusercontent.com/35102882/186561955-3e1c71fb-e83c-499a-b99d-9860830cdb0d.png)