Feature request: Add connector for FAISS

[HQarroum]

Use case

We want to add support for a FAISS index for a very low-cost, non-production setup. This would be a new middleware that acts as a storage connector taking embeddings from other middlewares in a pipeline and stores them in a FAISS index in a given storage.

Solution/User Experience

It would be possible to use a S3 bucket as a mean of low-cost storage. The FAISS storage connector would be based on a Lambda compute with a reserved concurrency of 1, loading and writing back the index on the S3 bucket.

Alternative solutions

No response

[HQarroum]

Would it be possible to list the following items to be able to proceed with a prototype ?

• A minimum viable deployment of a lambda function packaged with FAISS (is a lambda layer required? can we use faiss-cpu ?) that creates an index and inserts vector embeddings in the index ? The vector embeddings can be hard-coded in the example.
• A list of the properties that you would like to see in the FAISS storage connector construct ? For example, the type of the index (e.g L2), the number of dimensions, etc.
• Would you like the middleware to create its own S3 storage internally, or would you like to alternatively specify a bespoke user-provided S3 bucket to the construct ?