[Bug]: Metadata length is longer than chunk size (1024). Consider increasing the chunk size or decreasing the size of your metadata to avoid this.

[loyal812]

Bug Description

Currently, I am following this article steps to implement llama index with mongodb. The only difference is that I'm using my own CSV data. https://www.mongodb.com/developer/products/atlas/rag-with-polm-stack-llamaindex-openai-mongodb/

However, when I run the Python code, this error occurs. "Metadata length (1527) is longer than chunk size (1024). Consider increasing the chunk size or decreasing the size of your metadata to avoid this."

This is my code.

                for document in documents_list:
                temp_str = ""
                # Create a copy of the dictionary before modifying it
                new_document = document.copy()
                for key, value in document.items():
                    temp_str = temp_str + f"{key}: {value} "
                    new_document["contents"] = temp_str

                # Replace the original document with the modified copy
                document.update(new_document)

                # Create a Document object with the text and excluded metadata for llm and embedding models
                llama_document = Document(
                    text=document["contents"],
                    metadata=document,
                    excluded_llm_metadata_keys=["contents"],
                    excluded_embed_metadata_keys=["contents"],
                    metadata_template="{key}=>{value}",
                    text_template="Metadata: {metadata_str}\n-----\nContent: {content}",
                    )

                llama_documents.append(llama_document)

            parser = SentenceSplitter()
            nodes = parser.get_nodes_from_documents(llama_documents)

            for node in nodes:
                node_embedding = embed_model.get_text_embedding(
                    node.get_content(metadata_mode="all")
                )
                node.embedding = node_embedding

I'm not sure how I can solve this problem. Is there any good solution to solve this problem?

I look forward to your help from Llama Index support.

Thank you regards.

Version

0.10.22

Steps to Reproduce

It's my code.

           # Create an instance of the MongoDB class
            mongo_context = MongoDB(collection_id)

            # Parse the JSON data into a list of dictionaries
            documents_list = json.loads(excel_data)

            embed_model = OpenAIEmbedding(model="text-embedding-3-small", dimensions=1536)
            llm = OpenAI()

            Settings.llm = llm
            Settings.embed_model = embed_model

            llama_documents = []

            for document in documents_list:
                temp_str = ""
                # Create a copy of the dictionary before modifying it
                new_document = document.copy()
                for key, value in document.items():
                    temp_str = temp_str + f"{key}: {value} "
                    new_document["contents"] = temp_str

                # Replace the original document with the modified copy
                document.update(new_document)

                # Create a Document object with the text and excluded metadata for llm and embedding models
                llama_document = Document(
                    text=document["contents"],
                    metadata=document,
                    excluded_llm_metadata_keys=["contents"],
                    excluded_embed_metadata_keys=["contents"],
                    metadata_template="{key}=>{value}",
                    text_template="Metadata: {metadata_str}\n-----\nContent: {content}",
                    )

                llama_documents.append(llama_document)

            parser = SentenceSplitter()
            nodes = parser.get_nodes_from_documents(llama_documents)

            for node in nodes:
                node_embedding = embed_model.get_text_embedding(
                    node.get_content(metadata_mode="all")
                )
                node.embedding = node_embedding
            # Insert JSON array into the MongoDB collection

            result = mongo_context.insert_embedding_data(nodes)

Relevant Logs/Tracbacks

"Metadata length (1527) is longer than chunk size (1024). Consider increasing the chunk size or decreasing the size of your metadata to avoid this."

[dosubot[bot]]

Hey @loyal812, good to see you around here again! Ran into a bit of a pickle, I see? Let's see what we can do about that.

Thank you for providing the detailed information. The error you're encountering is likely due to the size of the metadata you're trying to index or store alongside your vectors in MongoDB. If you're using MongoDB's Atlas Vector Search or a similar feature, it's important to ensure that individual pieces of metadata associated with each vector do not exceed the chunk size limit (in this case, 1024 bytes).

To resolve this error, you can consider the following approaches:

1. Reduce Metadata Size: Before indexing your data, reduce the size of the metadata for each vector to ensure it's below the chunk size limit. This might involve truncating strings, removing less relevant fields, or compressing the data in some way.

2. Split Metadata: If possible, split the metadata into smaller chunks that can be indexed separately and still associated with the correct vector. This approach might require additional logic to reassemble the metadata when querying.

3. Custom Indexing Logic: Implement custom logic to handle large metadata sizes more gracefully. This could involve creating a separate collection for metadata that exceeds the chunk size limit and linking it back to the main vector data through references.

4. Consult MongoDB Documentation: Since MongoDB's capabilities and limitations can change, and features like Atlas Vector Search might have specific requirements or settings, reviewing the latest MongoDB documentation related to vector search and metadata indexing is advisable. There might be configuration options or best practices for handling large metadata.

5. Contact MongoDB Support: If you're using MongoDB Atlas and continue to face this issue, reaching out to MongoDB support can provide more tailored advice. They might offer insights specific to Atlas Vector Search indexes or suggest configuration changes to accommodate larger metadata.

In addition, the default chunk size for metadata in LlamaIndex is 512, and it is configurable. You can change it globally or locally as shown in the customization documentation:

# Global settings
from llama_index.core import Settings

Settings.chunk_size = 512

# Local settings
from llama_index.core.node_parser import SentenceSplitter

index = VectorStoreIndex.from_documents(
    documents, transformations=[SentenceSplitter(chunk_size=512)]
)

Please try these suggestions and let me know if you need further assistance.

Sources

• A single term is larger than the allowed chunk size.
• llama-index-integrations/vector_stores/llama-index-vector-stores-mongodb/llama_index/vector_stores/mongodb/base.py
• docs/docs/getting_started/customization.md

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.

[loyal812]

@dosu-bot

Please explain more detail about second solution. Split Metadata: If possible, split the metadata into smaller chunks that can be indexed separately and still associated with the correct vector. This approach might require additional logic to reassemble the metadata when querying.

[dosubot[bot]]

To resolve the error "Metadata length (1527) is longer than chunk size (1024)" while implementing LlamaIndex with MongoDB using your own CSV data, you can indeed split the metadata into smaller chunks and reassemble it when querying. Here's how you can do it:

1. Splitting Metadata for Insertion: You need to modify your insertion method to check the size of the metadata. If the metadata exceeds the MongoDB document size limit (in this case, a chunk size of 1024), you should split the metadata into smaller chunks.

2. Storing Metadata Chunks: Each chunk can be stored as a separate document or within a single document as an array of chunks. Make sure each chunk is identifiable and can be reassembled (e.g., by adding an index or a sequence number to each chunk).

3. Reassembling Metadata on Query: When querying, you need to retrieve all chunks related to the metadata and reassemble them in the correct order to reconstruct the original metadata.

Here's an example implementation based on the provided context:

Step 1: Modify the Insert Method to Split Metadata

First, you need to modify the method responsible for inserting documents to check and split the metadata if necessary. This example assumes you have a method for insertion and will modify it to handle metadata splitting:

def insert_document(self, document):
    metadata = document.get(self._metadata_key, {})
    metadata_str = str(metadata)  # Convert metadata to string for length check
    max_chunk_size = 1024  # Define the maximum chunk size
    if len(metadata_str) > max_chunk_size:
        # Split metadata into chunks
        metadata_chunks = [metadata_str[i:i+max_chunk_size] for i in range(0, len(metadata_str), max_chunk_size)]
        document[self._metadata_key] = metadata_chunks  # Replace metadata with chunks
    else:
        document[self._metadata_key] = [metadata_str]  # Store metadata as a single chunk for consistency
    # Proceed with document insertion...

Step 2: Store Metadata Chunks

The modified insertion method will store the metadata in chunks. Each chunk is stored as an element in an array under the metadata key.

Step 3: Reassemble Metadata on Query

When querying, you'll need to reassemble the metadata chunks. Here's an example of how you might modify a query method to reassemble metadata:

def query_document(self, query):
    # Perform the query to retrieve documents
    documents = self._collection.find(query)
    for document in documents:
        # Reassemble metadata if it's stored in chunks
        if isinstance(document.get(self._metadata_key), list):
            metadata_chunks = document[self._metadata_key]
            metadata_str = ''.join(metadata_chunks)
            document[self._metadata_key] = eval(metadata_str)  # Convert string back to dictionary
    return documents

Note: The eval function is used here to convert the string representation of the metadata back into a dictionary. However, using eval can be dangerous if the string is not well-controlled, as it can execute arbitrary code. Ensure the metadata content is safe or consider using a safer method of serialization/deserialization, such as json.loads and json.dumps, if applicable.

This approach allows you to work around the limitation by splitting large metadata into manageable chunks and reassembling them when needed, ensuring that the metadata's integrity is maintained throughout the process.

As for the maximum size of metadata that can be handled by the LlamaIndex framework without causing errors, I wasn't able to find an answer within the repository. The answer may be available elsewhere or I could have missed it. If you have any further questions or need more clarification, feel free to ask.

Sources

• llama-index-integrations/vector_stores/llama-index-vector-stores-mongodb/llama_index/vector_stores/mongodb/base.py

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.

[loyal812]

@dosu-bot

Can you tell me the maximum chunk size?

Setting the chunk size to 2048 works fine, but how much can I increase it?

[dosubot[bot]]

Based on the information available in the LlamaIndex repository, it seems that the chunk size is managed by the node_parser attribute of the _Settings dataclass. The chunk_size property getter and setter within this class allow for the configuration of the chunk size. Here is the relevant code snippet:

@property
def chunk_size(self) -> int:
    """Get the chunk size."""
    if hasattr(self.node_parser, "chunk_size"):
        return self.node_parser.chunk_size
    else:
        raise ValueError("Configured node parser does not have chunk size.")

@chunk_size.setter
def chunk_size(self, chunk_size: int) -> None:
    """Set the chunk size."""
    if hasattr(self.node_parser, "chunk_size"):
        self.node_parser.chunk_size = chunk_size
    else:
        raise ValueError("Configured node parser does not have chunk size.")

(Source: llama-index-core/llama_index/core/settings.py)

However, the repository does not provide explicit information on the maximum chunk size that can be set without causing performance issues or errors. It's possible that the maximum chunk size is dependent on the specific node_parser being used, as well as the resources available on your system (such as memory).

If you're experiencing issues with larger chunk sizes, it might be worth investigating whether these are due to limitations within the node_parser or resource constraints on your system. If you have any additional information about the node_parser you're using or the specific errors you're encountering with larger chunk sizes, that could help in providing a more precise answer.

Sources

• llama-index-core/llama_index/core/constants.py
• llama-index-core/llama_index/core/settings.py

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.

[loyal812]

It was helpful in gaining insight into chunk size.

https://www.llamaindex.ai/blog/evaluating-the-ideal-chunk-size-for-a-rag-system-using-llamaindex-6207e5d3fec5