ijyliu
commented
5 months ago Here's what I think would be a good set of tests.
Testing Indexes
Write a reference query ("Does Apple sell my information to third-party advertisers?" for example; use something like the below code from the RAG folder to format your query text!) and try it out for the different index types (FLAT, IVF_FLAT, etc.) for a fixed metric (L2 I guess): https://milvus.io/docs/index.md. For each index type, get the query time. Also, get the similarity score of the most similar vector returned. We can compare this score with that obtained from the exhaustive search FLAT, to see how much accuracy is lost when using IVF, etc. ANNS (approximate nearest neighbor search) methods.
def get_mixedbread_of_query(model, query: str):
'''
Returns mixedbread embedding for an input text. Text is appropriately formatted to be a query.
Parameters:
- model: embedding model
- query: str: The query to be transformed.
'''
transformed_query = f'Represent this sentence for searching relevant passages: {query}'
return model.encode(transformed_query)Be sure to reset the index to whatever it is right now (L2, IVF_FLAT, 128 I think) once you are done.
Testing Queries
For a fixed metric and index, compare the retrieval time for a variety of queries ("What am I allowed to post on Facebook?", "How can I delete my Reddit account?", etc.)
FayeL6
Find ways to do performance testing in Milvus like
EXPLAIN ANALYZE. Try to get timing, query plans, etc.