search

simonw / llm-clip

Generate embeddings for images and text using CLIP with LLM
Apache License 2.0
58 stars 2 forks source link

Ability to run alternative models #3

Open simonw opened 11 months ago

simonw commented 11 months ago

The model ID is hard-coded here: https://github.com/simonw/llm-clip/blob/ffa91b341230cccd39bdef3d67e2ec7a8ade205b/llm_clip.py#L20-L23

It would be neat if you could register and use alternative models.

Relevant: https://carleton.ca/xlab/2023/archaeclip-or-building-a-visual-search-engine-for-archaeology/ - about fine-tuning a custom CLIP model for use with this plugin.

ekansa commented 11 months ago

Perhaps we can pass an environment variable (?) to be able to specify alternative models, as:

def embed_batch(self, items):
    # Embeds a mix of text strings and binary images
    model_to_use = os.getenv('LLM_CLIP_MODEL_PATH', 'clip-ViT-B-32')
    if self._model is None:
        self._model = SentenceTransformer(model_to_use)