dbl001
commented
2 years ago Was the image corrupted during download?
Open dbl001 opened 2 years ago
dbl001
commented
2 years ago Was the image corrupted during download?
MaartenGr
commented
2 years ago That indeed does sound familiar. I can remember having trouble with some of the images when starting out developing the packages but I cannot find the code for fixing this. I believe you are indeed correct that some of the images are either corrupted or in a different format.
I would have to do some digging but it indeed seems that this might not be the best dataset to be used unless some images are removed...
dbl001
commented
2 years ago Not an elegant solution ... but it was quick:
# Prepare images
batch_size = 1
nr_iterations = int(np.ceil(len(images) / batch_size))
# Embed images per batch
embeddings = []
for i in tqdm(range(nr_iterations)):
start_index = i * batch_size
end_index = (i * batch_size) + batch_size
images_to_embed = [Image.open(filepath) for filepath in images[start_index:end_index]]
try:
img_emb = self.embedding_model.encode(images_to_embed, show_progress_bar=False)
embeddings.extend(img_emb.tolist())
# Close images
for image in images_to_embed:
image.close()
except Exception:
print("Skipping: %s", images[start_index:end_index])
# Close images
for image in images_to_embed:
image.close()
continue
return np.array(embeddings)Is this what you get?
MaartenGr
commented
2 years ago I just tried out the following code in a Kaggle session and I had no issues:
import os
import glob
import zipfile
from tqdm import tqdm
from sentence_transformers import util
from concept import ConceptModel
# 25k images from Unsplash
img_folder = 'photos/'
if not os.path.exists(img_folder) or len(os.listdir(img_folder)) == 0:
os.makedirs(img_folder, exist_ok=True)
photo_filename = 'unsplash-25k-photos.zip'
if not os.path.exists(photo_filename): # Download dataset if does not exist
util.http_get('http://sbert.net/datasets/' + photo_filename, photo_filename)
# Extract all images
with zipfile.ZipFile(photo_filename, 'r') as zf:
for member in tqdm(zf.infolist(), desc='Extracting'):
zf.extract(member, img_folder)
img_names = list(glob.glob('photos/*.jpg'))
# Train model
concept_model = ConceptModel()
concepts = concept_model.fit_transform(img_names)It might mean that something indeed went wrong when trying to load the images in your environment.
Is this what you get?
Concept uses UMAP, which is stocastisch by natures, which means that every run you will get different results. So comparing outputs is not easily done unless you set a random_state in UMAP which might hurt performance.
dbl001
commented
2 years ago Its something in the inner workings of SentenceTransformer('clip-ViT-B-32'). Possibly an issue with tensorflow_macos/tensorflow_metal and torch. Investigating ...
dbl001
commented
2 years ago It appears that the images which cause the exceptions int the normailze() methods have 4 channels (not 3).
numpy 1.20.3 transformers 4.11.3 sentence_transformers 2.1.0
Are you using the same versions?
these defaults in class CLIPFeatureExtractor(FeatureExtractionMixin, ImageFeatureExtractionMixin): look odd.
do_normalize (:obj:`bool`, `optional`, defaults to :obj:`True`):
Whether or not to normalize the input with :obj:`image_mean` and :obj:`image_std`.
image_mean (:obj:`List[int]`, defaults to :obj:`[0.485, 0.456, 0.406]`):
The sequence of means for each channel, to be used when normalizing images.
image_std (:obj:`List[int]`, defaults to :obj:`[0.229, 0.224, 0.225]`):
The sequence of standard deviations for each channel, to be used when normalizing images.
...
self.do_normalize = do_normalize
self.image_mean = image_mean if image_mean is not None else [0.48145466, 0.4578275, 0.40821073]
self.image_std = image_std if image_std is not None else [0.26862954, 0.26130258, 0.27577711]
But why doesn't your system generate these exceptions?
dbl001
commented
2 years ago MaartenGr
commented
2 years ago But why doesn't your system generate these exceptions?
Indeed, this is quite strange as it is working perfectly fine for me in a Kaggle session with the following packages:
transformers==4.15.0
tokenizers==0.10.3
sentence-transformers==1.2.0
numpy==1.20.3Hopefully, the issue you posted on the CLIP page gives a bit more insight.
Running a Concept example on OS S Monterey 12.3.1 ...Transformers/Image_utils #143: return (image - mean) / std
image is (4,224,224) mean is (3,) std is (3,)
Here's the code:
The exception is in the normalize() function ... I believe in the 9th Pil image: