Closed madawei2699 closed 12 months ago
I had the same issue. It seems like a lack of documentation or full examples problem. However, I would second a motion that makes this easier and more intuitive by allowing and init-ing a blank persistence directory.
I solved it by first creating the index without the persist_dir argument or storage context. Then I called index.storage_context.persist(persist_dir=persist_directory) and it created the required json files.
Once created I could load it using index = load_index_from_storage(storage_context)
Hi, @madawei2699! I'm Dosu, and I'm here to help the LlamaIndex team manage their backlog. I wanted to let you know that we are marking this issue as stale.
From what I understand, you reported an issue where initializing StorageContext
with a persist directory raises a FileNotFoundError
because the directory does not exist. The user bot403
suggested a workaround by first creating the index without the persist directory argument and then calling index.storage_context.persist(persist_dir=persist_directory)
to create the required JSON files. This solution was well-received by you, norbert-liki
, and wilmerhenao
.
Before we close this issue, we wanted to check if it is still relevant to the latest version of the LlamaIndex repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself or it will be automatically closed in 7 days.
Thank you for your contribution to LlamaIndex!
Found same issue
@bot403 thanks!
@lakinmindfire my current implementation of the workaround
from llama_index.storage.index_store.simple_index_store import DEFAULT_PERSIST_FNAME as INDEX_STORE_FNAME
if (STORAGE_PERSIST_DIR / INDEX_STORE_FNAME).exists():
storage_context = StorageContext.from_defaults(
persist_dir=STORAGE_PERSIST_DIR
)
index = load_index_from_storage(storage_context=storage_context)
else:
if not STORAGE_PERSIST_DIR.exists():
STORAGE_PERSIST_DIR.mkdir(parents=True)
storage_context = StorageContext.from_defaults()
index = VectorStoreIndex.from_documents(documents,
storage_context=storage_context,
show_progress=True)
storage_context.persist(persist_dir=STORAGE_PERSIST_DIR)
hope that helps
@jscastanoc Where is STORAGE_PERSIST_DIR coming from?
This should be reopened.
Pasting this here so that it might help someone,
TOGETHER_API_KEY = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
from llama_index.llms import OpenAILike
from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext, load_index_from_storage, StorageContext
import os.path
os.makedirs("storage", exist_ok=True)
os.makedirs("input_data", exist_ok=True)
# for openrouter see old file
llm = OpenAILike(
model = "mistralai/Mixtral-8x7B-Instruct-v0.1",
api_base = "https://api.together.xyz/v1",
api_key = TOGETHER_API_KEY,
is_chat_model = True,
is_function_calling_model = True,
temperature = 0.1,
api_version = "v1",
max_tokens = 1164,
)
service_context = ServiceContext.from_defaults(llm=llm, embed_model="local")
# code inside some function
# the input_dir is the dir in which files are stored that we want to read, and the store_dir is the place where we want to persist the embeddings
# what I was missing was adding service_context while loading
def some_function():
# variables declared
if os.path.exists(store_dir):
# load the existing index
storage_context = StorageContext.from_defaults(persist_dir=store_dir)
index = load_index_from_storage(
storage_context,
service_context = service_context,
)
else:
# load the documents and create the index
documents = SimpleDirectoryReader(input_dir).load_data()
index = VectorStoreIndex.from_documents(
documents,
service_context = service_context
)
# store it for later
index.storage_context.persist(persist_dir=store_dir)
# either way we can now query the index
query_engine = index.as_query_engine()
response = query_engine.query(question)
what everyone was missing was passing the service context when reading the data.
hope this helps someone get a sleep, its 6 in the morning and I havent slept today :(
This is my code:
Then it will raise a error, the error log is:
The Python version is 3.10, the
llama-index
is 0.6.9, and thelangchain
is 0.0.154.