search

dotnet-smartcomponents / smartcomponents

Experimental, end-to-end AI features for .NET apps
622 stars 54 forks source link

Encode images #55

Closed Giorgi closed 2 weeks ago

Giorgi commented 1 month ago

Are there any plans to add support for encoding images and other types of data (audio, video, etc). Similar to SentenceTransformers Image Search

Onnx already supports working with images:

https://onnxruntime.ai/docs/tutorials/csharp/resnet50_csharp.html

SteveSandersonMS commented 2 weeks ago

Can you clarify what this has to do with smart components? Which smart component would work with images and in what way?

Giorgi commented 2 weeks ago

The Local Embeddings one. I want to generate vector embeddings from images and run semantic search on the generated embeddings.

SteveSandersonMS commented 2 weeks ago

I see, thanks. You'd need to run a different model on Onnx (e.g., CLIP). There's an example of doing this at https://bartbroere.eu/2023/07/29/openai-clip-csharp-onnx/.

Once you have used that to get an embedding vector, you can do semantic search either by calling LocalEmbedder.FindClosest (for in-memory search) or by calling the "search" API of whatever external vector DB you may be using.