search

simon987 / sist2

Lightning-fast file system indexer and search tool
GNU General Public License v3.0
845 stars 55 forks source link

support CLIP embedding for images #424

Closed ghost closed 12 months ago

ghost commented 12 months ago

Which SIST2 component is your Feature Request related to? index

Is your feature request related to a problem? Please describe. The CLIP image embedding can be used for finding images that looks (or conceptually) similar to user given text or image. This will bring a lot of interesting use cases.

What would you like to see happen? https://github.com/photoprism/photoprism/issues/1287

Additional context embeddings can be stored into hnsqlite (or #425) instead of elasticsearch.

simon987 commented 12 months ago

Hi, CLIP embeddings are already supported in the master branch (not yet officially released)

You can create a script in the User Script tab using the CLIP template and add it in the job to enable it.

It supports text prompts and KNN ("find more images like this" button)

curtwagner1984 commented 11 months ago

Hi, CLIP embeddings are already supported in the master branch (not yet officially released)

You can create a script in the User Script tab using the CLIP template and add it in the job to enable it.

It supports text prompts and KNN ("find more images like this" button)

Can you please elaborate on how to do this, where to start?

simon987 commented 11 months ago

Can you please elaborate on how to do this, where to start?

If you are using sist2-admin,

  1. Go to the User Scripts tab
  2. Enter the name of the new script (ex. "clip") and select the CLIP template
  3. Go to the job where you want to enable CLIP embeddings, select the created script in the list
  4. re-index