[2.0] Adding Hugging Face TGI Generator component to pipeline fails

[lfunderburk]

Advent of Haystack Day 2

Describe the bug Building RAG with ranker and with HFTGI Generator component throws data structure error message, whereas if I use GPT Generator, the pipeline works fine.

Error message

AttributeError                            Traceback (most recent call last)
File [~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/core/pipeline/pipeline.py:634](https://file+.vscode-resource.vscode-cdn.net/Users/macpro/Documents/GitHub/Advent-of-Haystack/day2-ranking-rag-responses/notebooks/~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/core/pipeline/pipeline.py:634), in Pipeline._run_component(self, name, inputs)
    [632](https://file+.vscode-resource.vscode-cdn.net/Users/macpro/Documents/GitHub/Advent-of-Haystack/day2-ranking-rag-responses/notebooks/~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/core/pipeline/pipeline.py:632) logger.debug("   '%s' inputs: %s", name, inputs)
--> [634](https://file+.vscode-resource.vscode-cdn.net/Users/macpro/Documents/GitHub/Advent-of-Haystack/day2-ranking-rag-responses/notebooks/~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/core/pipeline/pipeline.py:634) outputs = instance.run(**inputs)
    [636](https://file+.vscode-resource.vscode-cdn.net/Users/macpro/Documents/GitHub/Advent-of-Haystack/day2-ranking-rag-responses/notebooks/~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/core/pipeline/pipeline.py:636) # Unwrap the output

File [~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/components/generators/hugging_face_tgi.py:190](https://file+.vscode-resource.vscode-cdn.net/Users/macpro/Documents/GitHub/Advent-of-Haystack/day2-ranking-rag-responses/notebooks/~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/components/generators/hugging_face_tgi.py:190), in HuggingFaceTGIGenerator.run(self, prompt, generation_kwargs)
    [188](https://file+.vscode-resource.vscode-cdn.net/Users/macpro/Documents/GitHub/Advent-of-Haystack/day2-ranking-rag-responses/notebooks/~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/components/generators/hugging_face_tgi.py:188)     return self._run_streaming(prompt, prompt_token_count, generation_kwargs)
--> [190](https://file+.vscode-resource.vscode-cdn.net/Users/macpro/Documents/GitHub/Advent-of-Haystack/day2-ranking-rag-responses/notebooks/~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/components/generators/hugging_face_tgi.py:190) return self._run_non_streaming(prompt, prompt_token_count, num_responses, generation_kwargs)

File [~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/components/generators/hugging_face_tgi.py:228](https://file+.vscode-resource.vscode-cdn.net/Users/macpro/Documents/GitHub/Advent-of-Haystack/day2-ranking-rag-responses/notebooks/~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/components/generators/hugging_face_tgi.py:228), in HuggingFaceTGIGenerator._run_non_streaming(self, prompt, prompt_token_count, num_responses, generation_kwargs)
    [223](https://file+.vscode-resource.vscode-cdn.net/Users/macpro/Documents/GitHub/Advent-of-Haystack/day2-ranking-rag-responses/notebooks/~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/components/generators/hugging_face_tgi.py:223) tgr: TextGenerationResponse = self.client.text_generation(prompt, details=True, **generation_kwargs)
    [224](https://file+.vscode-resource.vscode-cdn.net/Users/macpro/Documents/GitHub/Advent-of-Haystack/day2-ranking-rag-responses/notebooks/~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/components/generators/hugging_face_tgi.py:224) all_metadata.append(
    [225](https://file+.vscode-resource.vscode-cdn.net/Users/macpro/Documents/GitHub/Advent-of-Haystack/day2-ranking-rag-responses/notebooks/~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/components/generators/hugging_face_tgi.py:225)     {
    [226](https://file+.vscode-resource.vscode-cdn.net/Users/macpro/Documents/GitHub/Advent-of-Haystack/day2-ranking-rag-responses/notebooks/~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/components/generators/hugging_face_tgi.py:226)         "model": self.client.model,
    [227](https://file+.vscode-resource.vscode-cdn.net/Users/macpro/Documents/GitHub/Advent-of-Haystack/day2-ranking-rag-responses/notebooks/~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/components/generators/hugging_face_tgi.py:227)         "index": _i,
--> [228](https://file+.vscode-resource.vscode-cdn.net/Users/macpro/Documents/GitHub/Advent-of-Haystack/day2-ranking-rag-responses/notebooks/~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/components/generators/hugging_face_tgi.py:228)         "finish_reason": tgr.details.finish_reason.value,
    [229](https://file+.vscode-resource.vscode-cdn.net/Users/macpro/Documents/GitHub/Advent-of-Haystack/day2-ranking-rag-responses/notebooks/~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/components/generators/hugging_face_tgi.py:229)         "usage": {
    [230](https://file+.vscode-resource.vscode-cdn.net/Users/macpro/Documents/GitHub/Advent-of-Haystack/day2-ranking-rag-responses/notebooks/~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/components/generators/hugging_face_tgi.py:230)             "completion_tokens": len(tgr.details.tokens),
    [231](https://file+.vscode-resource.vscode-cdn.net/Users/macpro/Documents/GitHub/Advent-of-Haystack/day2-ranking-rag-responses/notebooks/~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/components/generators/hugging_face_tgi.py:231)             "prompt_tokens": prompt_token_count,
    [232](https://file+.vscode-resource.vscode-cdn.net/Users/macpro/Documents/GitHub/Advent-of-Haystack/day2-ranking-rag-responses/notebooks/~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/components/generators/hugging_face_tgi.py:232)             "total_tokens": prompt_token_count + len(tgr.details.tokens),
    [233](https://file+.vscode-resource.vscode-cdn.net/Users/macpro/Documents/GitHub/Advent-of-Haystack/day2-ranking-rag-responses/notebooks/~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/components/generators/hugging_face_tgi.py:233)         },
    [234](https://file+.vscode-resource.vscode-cdn.net/Users/macpro/Documents/GitHub/Advent-of-Haystack/day2-ranking-rag-responses/notebooks/~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/components/generators/hugging_face_tgi.py:234)     }
    [235](https://file+.vscode-resource.vscode-cdn.net/Users/macpro/Documents/GitHub/Advent-of-Haystack/day2-ranking-rag-responses/notebooks/~/anaconda3/envs/advent-haystack/lib/python3.10/site-packages/haystack/components/generators/hugging_face_tgi.py:235) )
...

PipelineRuntimeError: llm raised 'AttributeError: 'str' object has no attribute 'value'' 
Inputs: {'prompt': '\nAccording to these documents:\n\n\n  answer a simple question:  “What is our favorite animal?”.\nHint: The pipeline from the first challenge will be really useful here. Below is also an image of a pipeline that might work very well for this challenge.\n\U0001fa75 Here’s the Starter Colab\nIn the starter colab you will only find what you will have to run once you’ve got a working pipeline, and a PromptBuilder. We’ve also included the imports to some useful components.\nLinkContentFetcher `: This will allow you to fetch the contents of https://haystack.deepset.ai/advent-of-haystack/day-1#challenge\nHTMLToDocument : Once you’ve fetched the contents, this component will allow you to convert it to a Document\nDocumentSplitter \n\n  Try out Haystack 2.0-Beta to discover what’s coming in the next major release\nwith 10 challenges in the month of December 🎉\nEvery few days one of the doors in this page will open to reveal a new challenge\nSubmit your results and discuss solutions with the community 🎄\nSubmit\nIn this challenge, the Haystack elves are sending you off on a bit of a scavenger hunt! Somewhere, in the first challenge, they have hidden (not very well hidden) some information. That information is about what their favorite animal is!\nYour task is to create a retrieval-augmented generative pipeline that will answer a simple question:  \n\n  version: Haystack 2.0-Beta . They’ve been trying to prepare some preview examples for you, but elves are forgetful beings and easily distracted. This challenge series will include some “gifts” the elves prepared for you, but we need you to take it over the finish line.\nHopefully, by the end of each challenge, you’ll also have learned something new about the upcoming Haystack 2.0!\nThe Haystack elves wanted to gift you a retrieval-augmented generative pipeline that can answer questions based on the contents of a given URL. Our favorite animal is a capybara!!\n\U0001fa75 Here is the Starter Colab\nFor this, they used:\n\n\n  convert it to a Document\nDocumentSplitter (Optional): This is useful if you want to split your Document into chunks\nTransformersSimilarityRanker (Optional): This is useful if you want to rank your documents (chunked with the splitter above) so that the most relevant is at the top.\nPromptBuilder : This is used to define how you want to prompt an LLM so that it generates an accurate response for you. We’ve included one for you in the starter Colab that will help you with this challenge.\nGPTGenerator : This component is used to query GPT. You can change this to one of our other generators as \n\n  Try out Haystack 2.0-Beta to discover what’s coming in the next major release\nwith 10 challenges in the month of December 🎉\nEvery few days one of the doors in this page will open to reveal a new challenge\nSubmit your results and discuss solutions with the community 🎄\nSubmit\nWelcome to the first challenge of our Advent of Haystack series (inspired by the Advent of Code )!\nWe have some news! The Haystack elves have been working hard in their workshops to develop Haystack 2.0 for the last few months. And this Christmas season, they’re gifting you with a preview version: Haystack 2.0-Beta . They’ve \n\n  of our other generators as well!\nCommunity\n\n\n  Try out Haystack 2.0-Beta to discover what’s coming in the next major release\nwith 10 challenges in the month of December 🎉\nEvery few days one of the doors in this page will open to reveal a new challenge\nSubmit your results and discuss solutions with the community 🎄\n\n\n\nAnswer the given question: What is our favourite animal?\nAnswer:'}

Expected behavior I should be able to use another generator and obtain the same functionality.

Additional context

Advent of Haystack Day 2

To Reproduce

from getpass import getpass
from haystack.components.generators import HuggingFaceTGIGenerator
from haystack.components.fetchers.link_content import LinkContentFetcher
from haystack.components.converters import HTMLToDocument
from haystack.components.preprocessors import DocumentSplitter
from haystack.components.rankers import TransformersSimilarityRanker
from haystack.components.generators import GPTGenerator
from haystack.components.builders.prompt_builder import PromptBuilder
from haystack import Pipeline
from haystack.components.retrievers import InMemoryBM25Retriever
from haystack.document_stores.in_memory import InMemoryDocumentStore

hugging_face_token = getpass("Enter Hugging Face Token: ")
mistral_hf_endpoint = "https://api-inference.huggingface.co/models/mistralai/Mistral-7B-v0.1"

prompt_template = """
According to these documents:

{% for doc in documents %}
  {{ doc.content }}
{% endfor %}

Answer the given question: {{question}}
Answer:
"""

prompt_builder = PromptBuilder(template=prompt_template)
fetcher = LinkContentFetcher()
converter = HTMLToDocument()
splitter = DocumentSplitter(split_length=100, split_overlap=5)
llm = HuggingFaceTGIGenerator(model="mistralai/Mistral-7B-v0.1",
                                     url=mistral_hf_endpoint,
                                     token=hugging_face_token)
ranker = TransformersSimilarityRanker()
ranker.warm_up()

pipeline = Pipeline()
pipeline.add_component(name="fetcher", instance=fetcher)
pipeline.add_component(name="converter", instance=converter)
pipeline.add_component(name="splitter", instance=splitter)
pipeline.add_component(name="ranker", instance=ranker)
pipeline.add_component(name="prompt_builder", instance=prompt_builder)
pipeline.add_component(name="llm", instance=llm)

pipeline.connect("fetcher", "converter")
pipeline.connect("converter", "splitter")
pipeline.connect("splitter", "ranker")
pipeline.connect("ranker.documents", "prompt_builder.documents")
pipeline.connect("prompt_builder", "llm")

question = "What is our favorite animal?"
result = pipeline.run({"ranker": {"query": question},
                   "prompt_builder": {"question": question},
                   "fetcher": {"urls": ["https://haystack.deepset.ai/advent-of-haystack/day-1#challenge"]}})
print(result['llm']['replies'][0])

FAQ Check

• [x] Have you had a look at our new FAQ page?

System:

• OS: MacOS Ventura, 13.5.2, M1 chip
• GPU/CPU:
• Haystack version (commit or version number): 2.0.0b
• DocumentStore: NA
• Reader: NA
• Retriever: NA

[julian-risch]

Hello @lfunderburk Thank you for reporting this. I have some trouble to reproduce the issue. Using the code you have under "To Reproduce", I just added "ranker": {"query": question}, to the pipeline.run(...) call and I get the expected answer Capybara. Could you please check again whether you can reproduce the error? Here is a colab for running the code: https://colab.research.google.com/drive/1fjsevrTRYd5FFcScD8ryjQVyTCgdRhyr?usp=sharing

[Joricz]

Hello.

I have a similar issue, some details as a context.

1. I've tried with both Mixtral and Mistral to discard an issue with the model, in both cases I get the same error.
2. The same code runs OK in a google colab notebook but gives an error in Windows VS Code.
3. Python version is 3.11.7
4. Haystack version is haystack-ai==2.0.0b3
5. There is an exclusive virtual environment for this code to avoid conflict with other dependencies.

This is the code:

hf_token = "XXXXXXXXXXXXXXXXXXXX" from haystack.components.generators import HuggingFaceTGIGenerator

generator = HuggingFaceTGIGenerator("mistralai/Mixtral-8x7B-Instruct-v0.1", token=hf_token)

generator = HuggingFaceTGIGenerator("mistralai/Mistral-7B-v0.1", token=hf_token) generator.warm_up() result = generator.run("What is Natural Language Processing?", generation_kwargs={"max_new_tokens": 350}) print(result["replies"][0])

This is the output:

(mixtral) PS C:\Users\52331\Desktop\Main\Python\LanguageModels\Mixtral_Haystack_HF-TGI> & c:/Users/52331/Desktop/Main/Python/LanguageModels/Mixtral_Haystack_HF-TGI/mixtral/Scripts/python.exe c:/Users/52331/Desktop/Main/Python/LanguageModels/Mixtral_Haystack_HF-TGI/mix01.py Traceback (most recent call last): File "c:\Users\52331\Desktop\Main\Python\LanguageModels\Mixtral_Haystack_HF-TGI\mix01.py", line 15, in result = generator.run("What is Natural Language Processing?", generation_kwargs={"max_new_tokens": 350}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\52331\Desktop\Main\Python\LanguageModels\Mixtral_Haystack_HF-TGI\mixtral\Lib\site-packages\haystack\components\generators\hugging_face_tgi.py", line 190, in run return self._run_non_streaming(prompt, prompt_token_count, num_responses, generation_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\52331\Desktop\Main\Python\LanguageModels\Mixtral_Haystack_HF-TGI\mixtral\Lib\site-packages\haystack\components\generators\hugging_face_tgi.py", line 228, in _run_non_streaming "finish_reason": tgr.details.finish_reason.value, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'str' object has no attribute 'value' (mixtral) PS C:\Users\52331\Desktop\Main\Python\LanguageModels\Mixtral_Haystack_HF-TGI>

[vblagoje]

@Joricz thank you for this report. I can't reproduce this issue on Python versions (3.8-3.11) on OSX. Please put a debugger breakpoint just before this line to inspect the value of tgr.details.finish_reason and let us know what you get there. You should get the FinishReason enum. Also, which huggingface_hub version do you have installed in your environment?

[Joricz]

Hello. I've attached pictures On line 12 "generator = HuggingFaceTGIGenerator("mistralai/Mistral-7B-v0.1", token=hf_token)", I got: "tgr.details.finish_reason: not available" On line 13, "generator.warm_up()", I got: "tgr.details.finish_reason:NameError: name 'tgr' is not defined"

hf_hub version is: 0.19.4

Best regards and happy new year :)

[ajaysurya1221]

haystack-ai 2.0.0b4 Fixed this!

[mathislucka]

@julian-risch could you please verify that the fix works and close this if it does?

[Joricz]

Hello.

In my case I've upgraded to 2.0.0b4 and the issue has been fixed. Now I can go on with the rest of the tutorials, thank you!

Jorge.

[julian-risch]

Can't reproduce with 2.0.0b4 and I suspect that initially an old transformers version caused the issue. Feel free to re-open if the issue remains with up to date Haystack and transformers versions.