[Bug]: "Empty Response" by running official documents.

[kiasar]

Bug Description

I'm trying to run your official document code but I get "Empty Response".

I have only one .txt file containing "I have a cat named Max. It is in the park." and here is the version of my code:

import torch
from langchain.llms.base import LLM
from llama_index import SimpleDirectoryReader, LangchainEmbedding, ListIndex
from llama_index import LLMPredictor, ServiceContext
from transformers import pipeline
from typing import Optional, List, Mapping, Any

# set context window size
context_window = 2048
# set number of output tokens
num_output = 256

# store the pipeline/model outisde of the LLM class to avoid memory issues
model_name = "facebook/opt-iml-1.3b"
pipeline = pipeline("text-generation", model=model_name, device="cuda:0", model_kwargs={"torch_dtype": torch.bfloat16})

class CustomLLM(LLM):

    def _call(self, prompt: str, stop: Optional[List[str]] = None, **kwargs) -> str:
        prompt_length = len(prompt)
        response = pipeline(prompt, max_new_tokens=num_output)[0]["generated_text"]

        # only return newly generated tokens
        return response[prompt_length:]

    @property
    def _identifying_params(self) -> Mapping[str, Any]:
        return {"name_of_model": model_name}

    @property
    def _llm_type(self) -> str:
        return "custom"

# define our LLM
llm_predictor = LLMPredictor(llm=CustomLLM())

service_context = ServiceContext.from_defaults(
    llm_predictor=llm_predictor,
    context_window=context_window,
    num_output=num_output
)

# Load the your data
documents = SimpleDirectoryReader('./data2').load_data()
index = ListIndex.from_documents(documents, service_context=service_context)

# Query and print response
query_engine = index.as_query_engine()
response = query_engine.query("What is the name of my cat?")
print(response)

It will return "Empty Response".

P.S. I'm sure that it is able to read the .txt file because this is the output to print(documents)

[Document(id_='b280219b-fcee-4ccd-a1e0-64d6c696133e', embedding=None, metadata={'file_path': 'data2/1.txt', 'creation_date': '2023-11-09', 'last_modified_date': '2023-11-09', 'last_accessed_date': '2023-11-09'}, excluded_embed_metadata_keys=['creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['creation_date', 'last_modified_date', 'last_accessed_date'], relationships={}, hash='e6f967d037f7e35b52c98ab114171f7be1b34515a5354b05204c3338b2808e97', text='I have a cat named Max. It is in the park.', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n')]

Version

085.65

Steps to Reproduce

Just run the code.

Relevant Logs/Tracbacks

No response

[dosubot[bot]]

🤖

Hello @kiasar! I'm Dosu, a friendly bot here to assist you with your LlamaIndex related queries, bug resolutions, and contributions. While you're waiting for a human maintainer, I'll be helping out. I'm currently reviewing the issue you've raised and will provide a comprehensive response shortly. Thanks for your patience!

[dosubot[bot]]

🤖

Thank you for providing the detailed information. Based on the context provided, it seems like you're encountering an "Empty Response" error when running the official document code on LlamaIndex v0.8.65. This could be due to a few reasons:

1. The file you're trying to read is indeed being read, but it might not contain any text that the specific reader (PDFReader, DocxReader, or HWPReader) can extract. This could happen if the file is an image-based PDF or a DOCX file with images only. You can check the content of your .txt file to ensure it contains extractable text.

2. There might be an issue with the encoding of the file. If the file's encoding doesn't match the encoding expected by the reader, the reader might not be able to extract any text, resulting in an "Empty Response". You can check the encoding of your .txt file and ensure it matches the expected encoding.

3. There might be an issue with the specific reader you're using. If there's a bug in the reader, it might not be able to extract text correctly, even if the file is being read and contains extractable text. You can try using a different reader to see if the issue persists.

4. If you're using the HWPReader, it's possible that the file you're trying to read isn't a valid HWP file. The HWPReader checks if the file is a valid HWP file by looking for specific sections in the file. If these sections aren't found, the reader raises an exception. You can check if your .txt file is a valid HWP file.

You can refer to the llama_index/readers/file/docs_reader.py for more information on how the readers work.

In terms of the query_engine.query() function, if it doesn't find a match for the query in the provided documents, it will raise a ValueError with the message "No source nodes passed evaluation." This is different from returning an empty list. You can refer to the llama_index/query_engine/graph_query_engine.py and llama_index/query_engine/retry_source_query_engine.py for more information on how the query_engine.query() function works.

If you're still encountering the issue after checking these possibilities, could you please provide more specific information about the error and the context in which it's occurring? This will help in providing a more specific solution.

Sources

• Query Response Returning NONE (Custom LLM)
• llama_index/readers/file/docs_reader.py
• llama_index/query_engine/graph_query_engine.py
• llama_index/query_engine/retry_source_query_engine.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[logan-markewich]

@kiasar try lowering the context window a bit. "Empty Response" with open-source LLMs is usually because the token counting broke.

Tbh that is a terrible example code, I should really replace that to be more simple.

If you want to use a model from huggingface, we have a few demo notebooks here https://docs.llamaindex.ai/en/stable/module_guides/models/llms.html#open-source-llms

[kiasar]

Thanks, finally it worked with context_window = 350. making it less than 300 and bigger than 400 will both result in errors.

I'll try the link you've provided. Thanks.

[mchoma]

I hit this using llamaindex against SageMaker providing llama2 model. context_window = 350 really helped, but that is unusable. Querying become slow and output is small.