[Question]: How to avoid reparsing with llama parse on app relaunch using LlamaIndex + Qdrant?

[alouiadel]

Question Validation

• [X] I have searched both the documentation and discord for an answer.

Question

I'm using LlamaIndex, Qdrant, and llama parse to:

• Parse PDFs
• Embed using OpenAI models
• Store embeddings in Qdrant On relaunch, I want to:
• Skip reparsing with llama parse if the data is already in Qdrant, and reuse the qdrant embeddings I checked the docs but couldn't find a clear way to Achieve this with the current stack I tried saving nodes as a pickle, and reloading them, but It's messy and complicated Is there a cleaner solution you suggest

[dosubot[bot]]

To skip reparsing PDFs and reuse Qdrant embeddings on app relaunch using LlamaIndex, Qdrant, and OpenAI models, you can follow these steps:

1. Save the Index to Disk: After creating the index, save it to disk so that it can be reloaded later without reparsing the PDFs.
2. Load the Index from Disk: On app relaunch, load the saved index from disk.
3. Reuse Qdrant Embeddings: Ensure that the embeddings are stored in Qdrant and can be accessed using the Qdrant API.

Here is an example of how you can achieve this:

Saving the Index to Disk

from llama_index import SimpleDirectoryReader, GPTSimpleVectorIndex

# Load documents and create the index
documents = SimpleDirectoryReader('path/to/pdf_directory').load_data()
index = GPTSimpleVectorIndex.from_documents(documents)

# Save the index to disk
index.save_to_disk('path/to/saved_index.json')

Loading the Index from Disk

from llama_index import GPTSimpleVectorIndex

# Load the index from disk
index = GPTSimpleVectorIndex.load_from_disk('path/to/saved_index.json')

# Now you can use the index to query without reparsing the PDFs
response = index.query("Your query here")
print(response)

Reusing Qdrant Embeddings

Ensure that your Qdrant embeddings are stored and accessible. You can use the Qdrant API to manage and retrieve embeddings as needed.

from qdrant_client import QdrantClient

# Initialize Qdrant client
client = QdrantClient(api_key='your_qdrant_api_key', url='your_qdrant_url')

# Example of retrieving embeddings from Qdrant
# This assumes you have already stored your embeddings in Qdrant
embeddings = client.retrieve_embeddings(collection_name='your_collection_name', ids=['embedding_id1', 'embedding_id2'])

# Use the retrieved embeddings as needed

If you do not have the index.json file but have stored your embeddings in Qdrant, you can recreate the index using the embeddings stored in Qdrant. This is similar to functionality available in Langchain:

from llama_index import GPTSimpleVectorIndex
from qdrant_client import QdrantClient

# Initialize Qdrant client
client = QdrantClient(api_key='your_qdrant_api_key', url='your_qdrant_url')

# Retrieve embeddings from Qdrant
embeddings = client.retrieve_embeddings(collection_name='your_collection_name', ids=['embedding_id1', 'embedding_id2'])

# Recreate the index using the retrieved embeddings
index = GPTSimpleVectorIndex.from_embeddings(embeddings)

# Now you can use the index to query
response = index.query("Your query here")
print(response)

By following these steps, you can skip reparsing PDFs and reuse Qdrant embeddings on app relaunch ^[1][2].

To continue talking to Dosu, mention @dosu.

[alouiadel]

any help here ? @logan-markewich