run-llama / llama_index

LlamaIndex is a data framework for your LLM applications
https://docs.llamaindex.ai
MIT License
35.38k stars 4.98k forks source link

[Question]: Add new documents in an existing collection in Qdrant Vector Store #13653

Closed harshachopra507 closed 2 weeks ago

harshachopra507 commented 3 months ago

Question Validation

Question

How can I add new documents in an existing collection in Qdrant Vector Store? The existing collection already contains chunk embedding for few documents.

dosubot[bot] commented 3 months ago

To add new documents to an existing collection in Qdrant Vector Store, you can use the add method of the QdrantVectorStore class. Here’s a step-by-step guide:

  1. Create a Qdrant client: Initialize the Qdrant client to interact with the Qdrant instance.
  2. Initialize the QdrantVectorStore: Create an instance of QdrantVectorStore with the existing collection name.
  3. Create a list of TextNode objects: Each TextNode should contain the text, unique ID, relationships, metadata, and embeddings.
  4. Add the nodes to the Qdrant collection: Use the add method of the QdrantVectorStore instance to add the new nodes.

Here is an example code snippet:

from llama_index.legacy.schema import TextNode, NodeRelationship, RelatedNodeInfo
from llama_index.legacy.vector_stores import QdrantVectorStore
import qdrant_client

# Step 1: Create a Qdrant client
client = qdrant_client.QdrantClient(":memory:")

# Step 2: Initialize the QdrantVectorStore with the existing collection name
qdrant_vector_store = QdrantVectorStore(collection_name="existing_collection", client=client)

# Step 3: Create a list of TextNode objects with embeddings
new_node_embeddings = [
    TextNode(
        text="new document text 1",
        id_="new-doc-id-1",
        relationships={NodeRelationship.SOURCE: RelatedNodeInfo(node_id="source-id-1")},
        metadata={
            "author": "Author Name",
            "theme": "Theme",
        },
        embedding=[0.5, 0.5],  # Example embedding
    ),
    TextNode(
        text="new document text 2",
        id_="new-doc-id-2",
        relationships={NodeRelationship.SOURCE: RelatedNodeInfo(node_id="source-id-2")},
        metadata={
            "author": "Another Author",
            "theme": "Another Theme",
        },
        embedding=[0.2, 0.8],  # Example embedding
    ),
]

# Step 4: Add the new nodes to the Qdrant collection
qdrant_vector_store.add(new_node_embeddings)

This will add the new documents to the existing collection, and if the collection is not already initialized, it will be created with the appropriate vector size.

#### Details **Sources** - [llama-index-legacy/llama_index/legacy/vector_stores/qdrant.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/llama_index/legacy/vector_stores/qdrant.py) - [llama-index-integrations/vector_stores/llama-index-vector-stores-qdrant/llama_index/vector_stores/qdrant/base.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/vector_stores/llama-index-vector-stores-qdrant/llama_index/vector_stores/qdrant/base.py) - [llama-index-legacy/tests/vector_stores/test_qdrant.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/tests/vector_stores/test_qdrant.py) - [docs/docs/examples/vector_stores/qdrant_hybrid.ipynb](https://github.com/run-llama/llama_index/blob/main/docs/docs/examples/vector_stores/qdrant_hybrid.ipynb) - [llama-index-integrations/vector_stores/llama-index-vector-stores-qdrant/tests/test_vector_stores_qdrant.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/vector_stores/llama-index-vector-stores-qdrant/tests/test_vector_stores_qdrant.py)
**About Dosu** This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.