Questions about setting up LLMs

[MatteoRiva95]

Bug Description

I am trying to run this code:

node_parser = SentenceSplitter(chunk_size=1024) nodes = node_parser.get_nodes_from_documents(documents)

taken from:

https://docs.llamaindex.ai/en/latest/examples/low_level/evaluation.html#

but it gives me back this error:

---

ImportError Traceback (most recent call last) Cell In[8], line 1 ----> 1 node_parser = SentenceSplitter(chunk_size=1024) 2 nodes = node_parser.get_nodes_from_documents(documents)

File /usr/local/lib/python3.10/dist-packages/llama_index/core/node_parser/text/sentence.py:87, in SentenceSplitter.init(self, separator, chunk_size, chunk_overlap, tokenizer, paragraph_separator, chunking_tokenizer_fn, secondary_chunking_regex, callback_manager, include_metadata, include_prev_next_rel, id_func) 81 raise ValueError( 82 f"Got a larger chunk overlap ({chunk_overlap}) than chunk size " 83 f"({chunk_size}), should be smaller." 84 ) 85 id_func = id_func or default_id_func ---> 87 callback_manager = callback_manager or CallbackManager([]) 88 self._chunking_tokenizer_fn = ( 89 chunking_tokenizer_fn or split_by_sentence_tokenizer() 90 ) 91 self._tokenizer = tokenizer or get_tokenizer()

File /usr/local/lib/python3.10/dist-packages/llama_index/core/callbacks/base.py:54, in CallbackManager.init(self, handlers) 52 def init(self, handlers: Optional[List[BaseCallbackHandler]] = None): 53 """Initialize the manager with a list of handlers.""" ---> 54 from llama_index.core import global_handler 56 handlers = handlers or [] 58 # add eval handlers based on global defaults

ImportError: cannot import name 'global_handler' from 'llama_index.core' (unknown location)

Version

0.10.10

Steps to Reproduce

from llama_index.core.node_parser import SentenceSplitter node_parser = SentenceSplitter(chunk_size=1024) nodes = node_parser.get_nodes_from_documents(documents)

Relevant Logs/Tracbacks

No response

[dosubot[bot]]

🤖

Hey @MatteoRiva95! Great to see you back, hope you've been having fewer bugs and more fun since we last chatted. What's cooking this time?

Based on the error message you've provided, it seems like the global_handler is not directly importable from llama_index.core because it's not defined in that module. Instead, it's dynamically set in the set_global_handler function in the llama_index/callbacks/global_handlers.py file.

To use the global_handler in your LLM application, you should first call the set_global_handler function with the appropriate eval_mode and eval_params. After this, you can access the global_handler from the llama_index module.

Here's an example of how you can do this:

import llama_index
from llama_index.callbacks.global_handlers import set_global_handler

# Set the global handler
set_global_handler(eval_mode="wandb", param1=value1, param2=value2)

# Now you can use the global handler
handler = llama_index.global_handler

Please replace "wandb", param1=value1, param2=value2 with your actual evaluation mode and parameters.

This behavior is defined in the set_global_handler function in the llama_index/callbacks/global_handlers.py file. The set_global_handler function takes an eval_mode string and any number of additional parameters (eval_params). It then calls the create_global_handler function with these parameters, which returns a handler object based on the eval_mode value. This handler object is then assigned to llama_index.global_handler.

You can find more details in the llama_index/callbacks/global_handlers.py file in the LlamaIndex repository.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Sources

• [Bug]: from llama_index.core import global_handler
• llama-index-legacy/llama_index/legacy/init.py
• llama-index-legacy/llama_index/legacy/callbacks/global_handlers.py

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.

[MatteoRiva95]

@dosu-bot Unfortunately, that was not the right answer. Anyway, I uninstalled llama-index and I pip installed it again.

This time, it gives me back this error:

---

AttributeError Traceback (most recent call last) Cell In[6], line 2 1 node_parser = SentenceSplitter(chunk_size=1024) ----> 2 nodes = node_parser.get_nodes_from_documents(documents)

File /usr/local/lib/python3.10/dist-packages/llama_index/core/node_parser/interface.py:70, in NodeParser.get_nodes_from_documents(self, documents, show_progress, kwargs) 57 def get_nodes_from_documents( 58 self, 59 documents: Sequence[Document], 60 show_progress: bool = False, 61 kwargs: Any, 62 ) -> List[BaseNode]: 63 """Parse documents into nodes. 64 65 Args: (...) 68 69 """ ---> 70 doc_id_todocument = {doc.id: doc for doc in documents} 72 with self.callback_manager.event( 73 CBEventType.NODE_PARSING, payload={EventPayload.DOCUMENTS: documents} 74 ) as event: 75 nodes = self._parse_nodes(documents, show_progress=show_progress, **kwargs)

File /usr/local/lib/python3.10/dist-packages/llama_index/core/node_parser/interface.py:70, in (.0) 57 def get_nodes_from_documents( 58 self, 59 documents: Sequence[Document], 60 show_progress: bool = False, 61 kwargs: Any, 62 ) -> List[BaseNode]: 63 """Parse documents into nodes. 64 65 Args: (...) 68 69 """ ---> 70 doc_id_todocument = {doc.id: doc for doc in documents} 72 with self.callback_manager.event( 73 CBEventType.NODE_PARSING, payload={EventPayload.DOCUMENTS: documents} 74 ) as event: 75 nodes = self._parse_nodes(documents, show_progress=show_progress, kwargs)

AttributeError: 'Document' object has no attribute 'id_'

[dosubot[bot]]

🤖

Based on the error message you provided, it seems like the issue might be related to the id_func function being called on a string object instead of a Document object. In the SentenceWindowNodeParser class, the id_func function is used in the build_window_nodes_from_documents method:

nodes = build_nodes_from_splits(
    text_splits,
    doc,
    id_func=self.id_func,
)

Here, build_nodes_from_splits is expected to take a Document object as its second argument. However, it's possible that a string is being passed instead. This could happen if the text_splits list contains string objects instead of Document objects.

The expected type of the documents parameter in the get_nodes_from_documents method of the SentenceWindowNodeParser class is Sequence[Document]. This is indicated in the method signature:

def build_window_nodes_from_documents(
    self, documents: Sequence[Document]
) -> List[BaseNode]:

Please ensure that the data being passed to build_nodes_from_splits is of the correct type. If the problem persists, could you provide more details about your usage and the data you're working with? This will help in identifying the root cause of the issue.

You can refer to the source code of the SentenceWindowNodeParser class in the LlamaIndex repository for more details.

If you're not sure about the attributes of the 'Document' class, you can use the built-in Python function 'dir()' to list all the attributes and methods of the 'Document' object. Here's an example:

doc = Document()  # or any instance of Document
print(dir(doc))

This will print out a list of all the attributes and methods of the 'Document' object, and you can check if 'id_' is in the list. If it's not, you'll need to use a valid attribute name instead.

If you're still having trouble, please provide more information about the 'Document' class and the part of your code where the error occurs.

-Dosu

Sources

• ,[Bug]: when i change "SentenceWindowNodeParser" chunk size, i get 'str' object has no attribute 'id_'
• llama-index-legacy/llama_index/legacy/indices/base.py
• llama-index-core/llama_index/core/schema.py

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.

[MatteoRiva95]

@dosu-bot this could be due to the fact that I am using "documents" as a list. In fact, I tried to parse only one PDF and it worked (even if I need PymuPDFReader to work on more than one PDF file).

Anyway, I am trying to use HuggingFace instead of OpenAI for the "Building Evaluation from Scratch" problem (https://docs.llamaindex.ai/en/latest/examples/low_level/evaluation.html#). But how can I use it? Since it gives me back this error:

qa_pairs = generate_qa_pairs(

nodes[:1],

nodes,
llm,
num_questions_per_chunk=10,

)

---

AttributeError Traceback (most recent call last) Cell In[15], line 1 ----> 1 qa_pairs = generate_qa_pairs( 2 # nodes[:1], 3 nodes, 4 llm, 5 num_questions_per_chunk=10, 6 )

Cell In[14], line 38, in generate_qa_pairs(nodes, llm, num_questions_per_chunk) 33 context_str = node.get_content(metadata_mode="all") 34 fmt_messages = question_gen_template.format_messages( 35 num_questions_per_chunk=10, 36 context_str=context_str, 37 ) ---> 38 chat_response = llm.chat(fmt_messages) 39 raw_output = chat_response.message.content 40 result_list = str(raw_output).strip().split("\n")

AttributeError: 'HuggingFacePipeline' object has no attribute 'chat'

[dosubot[bot]]

🤖

The error you're encountering is due to the fact that the HuggingFacePipeline object does not have a chat method. The chat method is actually part of the HuggingFaceInferenceAPI class, not the HuggingFacePipeline class.

To resolve this issue, you should create an instance of HuggingFaceInferenceAPI and use that to call the chat method. Here's an example of how you can do this:

from llama_index.legacy.llms.huggingface import HuggingFaceInferenceAPI
from llama_index.legacy.llms import ChatMessage, MessageRole

# Create an instance of HuggingFaceInferenceAPI
hf_inference_api = HuggingFaceInferenceAPI(model_name="your_model_name")

# Define your chat messages
messages = [
    ChatMessage(content="Which movie is the best?"),
    ChatMessage(content="It's Die Hard for sure.", role=MessageRole.ASSISTANT),
    ChatMessage(content="Can you explain why?"),
]

# Call the chat method
response = hf_inference_api.chat(messages=messages)

Please replace "your_model_name" with the name of the model you want to use.

As for the generate_qa_pairs function, without the exact context, it's hard to provide a precise answer. However, in the LlamaIndex codebase, there are similar functions like agenerate_dataset_from_nodes and generate_dataset_from_nodes that generate a dataset of question-answer pairs from given nodes (documents). If you're referring to a different function named generate_qa_pairs, please provide more context.

For more details, you can refer to the LlamaIndex codebase and the test cases for HuggingFaceInferenceAPI.

Sources

• llama-index-legacy/tests/llms/test_huggingface.py
• llama-index-core/llama_index/core/llama_dataset/generator.py

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.

[MatteoRiva95]

@dosu-bot okay, I changed the script like this, in order to have a quantized version of the model:

from llama_index.legacy.llms.huggingface import HuggingFaceInferenceAPI

model_id = "anakin87/zephyr-7b-alpha-sharded" # model repo id

other models:

"Trelis/Llama-2-7b-chat-hf-sharded-bf16"

"bn22/Mistral-7B-Instruct-v0.1-sharded"

"HuggingFaceH4/zephyr-7b-beta"

"anakin87/zephyr-7b-alpha-sharded"

device = 'cuda' # Run on gpu if available else run on cpu

# tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, device_map = "auto", quantization_config=quant_config)

create a pipeline

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_length=2048, do_sample=True, return_full_text = True, repetition_penalty = 1.1, num_beams=5, no_repeat_ngram_size=4)

llm = HuggingFaceInferenceAPI(pipeline=pipe)

but it gives me this error back:

---

ValueError Traceback (most recent call last) Cell In[17], line 31 20 # create a pipeline 21 pipe = pipeline("text-generation", 22 model=model, 23 tokenizer=tokenizer, (...) 28 num_beams=5, 29 no_repeat_ngram_size=4) ---> 31 llm = HuggingFaceInferenceAPI(pipeline=pipe)

File /usr/local/lib/python3.10/dist-packages/llama_index/legacy/llms/huggingface.py:528, in HuggingFaceInferenceAPI.init(self, **kwargs) 525 task = kwargs.get("task", "") 526 # NOTE: task being None or empty string leads to ValueError, 527 # which ensures model is present --> 528 kwargs["model_name"] = InferenceClient.get_recommended_model(task=task) 529 logger.debug( 530 f"Using Hugging Face's recommended model {kwargs['model_name']}" 531 f" given task {task}." 532 ) 533 if kwargs.get("task") is None:

File /usr/local/lib/python3.10/dist-packages/huggingface_hub/inference/_client.py:1935, in InferenceClient.get_recommended_model(task) 1933 model = _fetch_recommended_models().get(task) 1934 if model is None: -> 1935 raise ValueError( 1936 f"Task {task} has no recommended model. Please specify a model" 1937 " explicitly. Visit https://huggingface.co/tasks for more info." 1938 ) 1939 return model

ValueError: Task has no recommended model. Please specify a model explicitly. Visit https://huggingface.co/tasks for more info.

[dosubot[bot]]

🤖

Based on the error message you're encountering, it seems like the HuggingFaceInferenceAPI is not recognizing the model you're trying to use. This could be due to a variety of reasons, but one common cause is that the model name is not valid or recognized by the HuggingFaceInferenceAPI.

In the LlamaIndex codebase, the model name is typically specified when creating an instance of HuggingFaceInferenceAPI. For example, in the fixture_hf_inference_api function and the test_instantiation method of the TestHuggingFaceInferenceAPI class, a constant STUB_MODEL_NAME with the value "placeholder_model" is used to instantiate the HuggingFaceInferenceAPI.

@pytest.fixture(name="hf_inference_api")
def fixture_hf_inference_api() -> HuggingFaceInferenceAPI:
    with patch.dict("sys.modules", huggingface_hub=MagicMock()):
        return HuggingFaceInferenceAPI(model_name=STUB_MODEL_NAME)

def test_instantiation(self) -> None:
    mock_hub = MagicMock()
    with patch.dict("sys.modules", huggingface_hub=mock_hub):
        llm = HuggingFaceInferenceAPI(model_name=STUB_MODEL_NAME)

You can find this in the llama-index-legacy/tests/llms/test_huggingface.py file.

In your case, you're passing the model as part of the pipeline to the HuggingFaceInferenceAPI. It's possible that the HuggingFaceInferenceAPI is not able to extract the model name from the pipeline correctly, leading to the error.

One way to resolve this could be to explicitly pass the model name when creating the HuggingFaceInferenceAPI instance, like so:

llm = HuggingFaceInferenceAPI(model_name=model_id, pipeline=pipe)

This way, the HuggingFaceInferenceAPI has a direct reference to the model name and should be able to recognize it correctly.

I hope this helps! If you're still encountering issues, could you provide more details about the quant_config and how you're using the HuggingFaceInferenceAPI instance? This could help in identifying any other potential issues.

Sources

• specifying the model name isn't working in the latest version
• llama-index-legacy/tests/llms/test_huggingface.py

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.