ChromaDB loading is slow

[ivanpetrushev]

Chroma.from_documents(...) appears to be very slow

Step 3: Creating Chroma database... loading 939 documents... Time elapsed: 957.11 s

• embeddings model: amazon.titan-embed-text-v2:0

Lets measure Titan v2 vs Titan v1

[ivanpetrushev]

Titan v2:

Step 3: Creating Chroma database... loading 100 documents...
Time elapsed: 48.55 s

Titan v1:

Step 3: Creating Chroma database... loading 100 documents...
Time elapsed: 41.55 s

Both versions perform roughly the same.

[ivanpetrushev]

Generating embeddings "manually" via BedrockEmbeddings (not via Chroma), is taking roughly the same. Code:

def generate_embeddings(docs_split):
    print("Step 3: Generating embeddings...")
    time_start = time.time()
    embeddings = get_embeddings()
    for doc in docs_split:
        result = embeddings.embed_documents([doc.page_content])
    time_end = time.time()
    print(f"Time elapsed: {time_end - time_start:.2f} s")

Titan v2:

Step 3: Generating embeddings...
Time elapsed: 42.32 s

Titan v1:

Step 3: Generating embeddings...
Time elapsed: 51.13 s

Maybe we can parallelize this job?

[ivanpetrushev]

Running in parallel seems to have decent time reduction. Tried with batch_size = 10, = 20 and it looks with batch_size = 10 we get x10 reduction: Titan v2:

Step 3: Generating embeddings...
Time elapsed: 5.43 s

With batch_size = 20 there is no further increase, we may hit diminishing returns there.