dosubot[bot]
commented
9 months ago 🤖
Hi @Pascal-So,
I'm Dosu, an AI bot here to assist you with bugs, answer your questions, and help you get started with contributing to our projects. While we wait for a human maintainer to respond, I'm here to provide some immediate support. Let's get started, shall we?
Thank you for your feature request and the detailed explanation of your use case. I understand the importance of being able to run LlamaIndex completely offline for security reasons and to enhance user trust in the library.
The issue you're experiencing seems to be related to the tiktoken library, which is used by LlamaIndex. The tiktoken library is trying to download a file from the internet, which is causing a network timeout when the computer is not connected to the internet.
To enable completely offline operation, you would need to modify the LlamaIndex code to use local files instead of downloading them from the internet. This would involve downloading all the necessary files once and storing them locally, then modifying the code to use these local files.
Here's a simplified example of how you can modify the get_dataset_info function to use local files:
def get_dataset_info(
local_dir_path: PATH_TYPE,
remote_dir_path: PATH_TYPE,
remote_source_dir_path: PATH_TYPE,
dataset_class: str,
refresh_cache: bool = False,
library_path: str = "library.json",
source_files_path: str = "source_files",
disable_library_cache: bool = False,
) -> Dict:
"""Get dataset info."""
if isinstance(local_dir_path, str):
local_dir_path = Path(local_dir_path)
local_library_path = f"{local_dir_path}/{library_path}"
dataset_id = None
source_files = []
# Check cache first
if not refresh_cache and os.path.exists(local_library_path):
with open(local_library_path) as f:
library = json.load(f)
if dataset_class in library:
dataset_id = library[dataset_class]["id"]
source_files = library[dataset_class].get("source_files", [])
# Fetch up-to-date library from local repo if dataset_id not found
if dataset_id is None:
with open(f"{local_dir_path}/{library_path}") as f:
library_raw_content = f.read()
library = json.loads(library_raw_content)
if dataset_class not in library:
raise ValueError("Loader class name not found in library")
dataset_id = library[dataset_class]["id"]
source_files = _get_source_files_list(
str(local_dir_path), f"/{dataset_id}/{source_files_path}"
)
# create cache dir if needed
local_library_dir = os.path.dirname(local_library_path)
if not disable_library_cache:
if not os.path.exists(local_library_dir):
os.makedirs(local_library_dir)
# Update cache
with open(local_library_path, "w") as f:
f.write(library_raw_content)
if dataset_id is None:
raise ValueError("Dataset class name not found in library")
return {
"dataset_id": dataset_id,
"source_files": source_files,
}Please note that this is a simplified example and you might need to modify other parts of the code as well to make the entire LlamaIndex work offline.
I found a similar solved issue in the LlamaIndex repository: [Feature Request]: Local-model defaults for non-predictor sensitive experimentation. This issue was closed by a pull request: feat(UX): provide all-MiniLM-L6-v2 as local embed_model. This might provide some insights into how to implement offline operation in LlamaIndex.
I hope this helps! If you have any further questions or need more clarification, please don't hesitate to ask.
Sources
- [Feature Request]: Local-model defaults for non-predictor sensitive experimentation
- llama_index/download/dataset.py
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
If you want to continue the conversation, start your reply with @dosu-bot.
gardberg
Pascal-So
ChrisDelClea
logan-markewich
martgra
Feature Description
When using huggingface models and embeddings that have been downloaded in advance, it should be possible to run llama_index on a computer that is not connected to the internet at all, since the usage doesn't depend on openai or any other online services then.
Reason
Right now, even just this import line leads to a network timeout if the computer is not connected to the internet:
(full error message here)
``` File "/app/server.py", line 8, inSince for this application I'm only using the embeddings and no chat, therefore I managed to "fix" this by rewriting the
GlobalsHelper.tokenizermethod to just returnNone, so thattiktoken.get_encoding("gpt2")is no longer called. I also had to useset_global_tokenizer(..)to prevent another network timeout later on.It would be nice if this use case worked out of the box without me having to modify the library.
Value of Feature
Our use case is that we want to search across documents that shouldn't leave our internal network. While I don't believe that the llama_index library will leak any information on purpose, we believe it's best to add additional safety measures, just in case any of the libraries in the dependency tree suddenly hit a bug or receive a malicious update.
Being able to run completely offline is also a feature that would enhance user trust in the library, and as such I believe that this is a feature that you could then prominently advertise in the readme and on the website of the library, once the necessary changes have been made. From what I've seen, it also seems like the required changes shouldn't be too massive.