How to extract answer from RetrievalQAWithSourcesChain with ggml-model-q4_0.bin?

[fabmeyer]

I am working on a streamlit prototype to query text documents with an LLM.

Everything works fine with the openAI model.

However if I am using LlamaCpp the output only gets written in the console and LangChain returns an empty object at the end.

# model
callback_manager = BaseCallbackManager([StreamingStdOutCallbackHandler()])
model_LLAMA = LlamaCpp(model_path='./models/ggml-model-q4_0.bin', n_ctx=4096, callback_manager=callback_manager, verbose=True)

# chain
chain = RetrievalQAWithSourcesChain.from_chain_type(llm=model_LLAMA, chain_type='refine', retriever=docsearch.as_retriever())

# in streamlit
st.session_state['query'] = st.session_state['chain']({'question': st.session_state['user_input']}, return_only_outputs=False)
print('query: ' + str(st.session_state['query']))

On the console the following output gets printet word after word including the empty object at the end:

Trends in travel and vacation can be classified as follows:

1. Adventure travel: This type of travel involves visiting remote destinations with an emphasis on outdoor activities such as hiking, mountain climbing, whitewater rafting, etc. Tourists are looking for exciting adventures in nature to escape the hustle and bustle of their everyday lives.

2. Backpacking: This type of travel involves exploring new destinations on a budget. It allows tourists to experience different cultures while staying within their means financially.

3. Nature vacation: This type of travel involves spending time outdoors in nature, such as hiking in national parks or camping under the stars. It has become popular among tourists who want a more authentic and immersive experience of the natural world.

4. Mountain climbing: This type of travel involves scaling mountains or rocky terrains to get an up-close view of nature's most spectacular creations. Tourists are drawn to this thrilling challenge for their next adventure trip. 

5. Surfing vacation: This type of travel involvesquery: {'question': 'What are trends in travel and vacation?', 'answer': '', 'sources': ''}

How can I extract or write the output into an object like with the openAI model also?

[LuciferUchiha]

I am also having the same issue:

  callback_manager = CallbackManager([StreamingStdOutCallbackHandler()])

  model = LlamaCpp(
      model_path=(MODELS_PATH / "7B" / "ggml-model-q4_0.bin").as_posix(),
      n_ctx=CONTEXT_SIZE,
      callback_manager=callback_manager, verbose=True
  )
  question = opts.question

  chain = RetrievalQAWithSourcesChain.from_chain_type(llm=model,
                                                      chain_type="refine",
                                                      retriever=store.as_retriever())

  result = chain({"question": question})
  print(result)

[praneetreddy017]

Is this only a problem with this model?

[fabmeyer]

Is this only a problem with this model?

@praneetreddy017 No it's the exact same problem with all open-source models I have tested so far. To be precise: Everything works when using OpenAI API and everything works when using RetrievalQA instead of RetrievalQAWithSourcesChain. Just RetrievalQAWithSourcesChain in combination with open-source models does not work.

[mirodrr]

Having this same problem with Claude by Anthropic. So it's not just open source having this problem

[mirodrr]

Figured out a workaround to get the answer and sources with a non OpenAI model (Claude in this case):

qa_with_sources = RetrievalQAWithSourcesChain.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever(),
    return_source_documents=True
)

result = qa_with_sources(query, return_only_outputs=False)

resultAnswer = result['answer']
resultSourceDocuments = result['source_documents']

print('Answer: ')
print()
print(resultAnswer)
print()
print('Sources: ')
for document in resultSourceDocuments:
    print(document.metadata['source'])

This is what the output looks like:

Answer: 

 Benito Mussolini was an Italian politician who became the leader of the National Fascist Party and served as Prime Minister of Italy from 1922 to 1943. 

Mussolini started out as a socialist but was expelled from the party for supporting Italy's entry into World War I. He then founded the Fascist movement, which emphasized nationalism and anti-communism. In 1922, Mussolini marched on Rome with his paramilitary Blackshirts and was invited by King Victor Emmanuel III to form a government. Over the following years, Mussolini dismantled democratic institutions and consolidated power, eventually becoming dictator of Italy.

Mussolini pursued an aggressive foreign policy, including the invasion of Ethiopia and alliance with Nazi Germany. Italy entered World War II in 1940 but suffered defeats in Greece and North Africa. In 1943, Mussolini was deposed and imprisoned. He was rescued by German commandos but was captured and executed by Italian partisans in 1945.

Mussolini believed democracy had failed and that fascism would unify and strengthen the Italian nation. However, his regime severely curtailed civil liberties and political dissent. Mussolini promoted militarism and believed in the superiority of the Italian race, though not to the same degree as the Nazis. His fasc

Sources: 
https://simple.wikipedia.org/wiki/Benito%20Mussolini
https://simple.wikipedia.org/wiki/Benito%20Mussolini
https://simple.wikipedia.org/wiki/Fascism
https://simple.wikipedia.org/wiki/August%2031

This is how it should look like without the workaround, but it currenlty only looks like this with openAI Code:

from langchain.chains import RetrievalQAWithSourcesChain

qa_with_sources = RetrievalQAWithSourcesChain.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever()
)
qa_with_sources(query)

Output:

{'question': 'who was Benito Mussolini?',
 'answer': 'Benito Mussolini was an Italian politician and journalist who was the Prime Minister of Italy from 1922 until 1943. He was the leader of the National Fascist Party and invented the ideology of Fascism. He became dictator of Italy by the end of 1927 and was friends with German dictator Adolf Hitler. Mussolini attacked Greece and failed to conquer it. He was removed by the Great Council of Fascism in 1943 and was executed by a partisan on April 28, 1945. After the war, several Neo-Fascist movements have had success in Italy, the most important being the Movimento Sociale Italiano. His granddaughter Alessandra Mussolini has outspoken views similar to Fascism. \n',
 'sources': 'https://simple.wikipedia.org/wiki/Benito%20Mussolini, https://simple.wikipedia.org/wiki/Fascism'}

[dosubot[bot]]

Hi, @fabmeyer! I'm Dosu, and I'm helping the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, the issue is about extracting the answer from the RetrievalQAWithSourcesChain using the ggml-model-q4_0.bin model. Currently, the output can only be seen in the console and you are looking for a way to extract or write the output into an object like with the OpenAI model. Another user has reported a similar issue with a different model, but one user has found a workaround to extract the answer and sources with a non-OpenAI model.

Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself or it will be automatically closed in 7 days.

Thank you for your understanding and contribution to the LangChain project!