nlmatics / llmsherpa

Developer APIs to Accelerate LLM Projects
https://www.nlmatics.com
MIT License
1.15k stars 113 forks source link

Nodes and llama idnex #50

Open drewskidang opened 5 months ago

drewskidang commented 5 months ago

when using the llama index example from llama_index.readers.schema.base import Document from llama_index import VectorStoreIndex

index = VectorStoreIndex([]) for chunk in doc.chunks(): index.insert(Document(text=chunk.to_context_text(), extra_info={}))

are the chunks converted to nodes??? I'm tyring to use pinecone but it requires documents or nodes for ingestion but this happens after the index is created

ansukla commented 5 months ago

This code is creating a llamaindex document with the chunk's text. I am not sure how llamaindex handles it internally to pass it on to pinecone.