Issue in remove_duplicates

[cheungdaven]

embedding_model: BAAI/bge-m3

Constructing Vector DB index
Traceback (most recent call last):
  File "/fsx/lumos/test_agrag.py", line 176, in <module>
    ag_rag_single()
  File "/fsx/lumos/test_agrag.py", line 81, in ag_rag_single
    agrag.initialize_rag_pipeline()
  File "/fsx/github/autogluon-rag/src/agrag/agrag.py", line 554, in initialize_rag_pipeline
    self.batched_processing()
  File "/fsx/github/autogluon-rag/src/agrag/agrag.py", line 504, in batched_processing
    self.construct_vector_db(embeddings)
  File "/fsx/github/autogluon-rag/src/agrag/agrag.py", line 329, in construct_vector_db
    self.vector_db_module.construct_vector_database(embeddings)
  File "/fsx/github/autogluon-rag/src/agrag/modules/vector_db/vector_database.py", line 103, in construct_vector_database
    vectors, indices_to_keep = remove_duplicates(vectors, self.similarity_threshold, self.similarity_fn)
  File "/fsx/github/autogluon-rag/src/agrag/modules/vector_db/utils.py", line 69, in remove_duplicates
    embeddings_array = np.array([emb.numpy().flatten() for emb in embeddings])
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (89,) + inhomogeneous part.

[shreyash2106]

Hey @cheungdaven, you need to use cls pooling for bge-m3 model. You can set this in the config:

embedding:
  embedding_model: BAAI/bge-m3
  pooling_strategy: cls

[cheungdaven]

ack