ConversationalRetrievalQAChain - Return Source Documents?

[justinlevi]

What is the best way to have a list of the sources added to the bottom of the result? Sources are currently relative urls stored in the document metadata.

Wondering what the best way is to display the vector sources used in answer the final question? I've tried everything I can think of and I feel like I must be missing something.

My latest attempt was creating an outputParser and passing in the question, answer, and all vector documents but that only seems to work intermittently and it's very slow.

Here are my questions

• Wondering if my prompts aren't right?
• If I set verbose=True, I can see that the input documents but only the page_content is being stuffed into the context. Is there a way to also add the metadata to the context stuffing?
• I have returnSourceDocuments: true, in the ConversationalRetrievalQAChain, but these source documents aren't always used in the final answer.
• Is there a better way to do this? Any examples?

Here's my code:

// api/chat.ts

import type { NextApiRequest, NextApiResponse } from 'next'
import path from 'path'
import { HNSWLib } from 'langchain/vectorstores/hnswlib'
import { OpenAIEmbeddings } from 'langchain/embeddings/openai'
import { makeChain } from './util'
import { Message } from '@/types'

export default async function handler(
  req: NextApiRequest,
  res: NextApiResponse,
) {
  const body = req.body as { question: string; history: Message[] }
  const dir = path.resolve(process.cwd(), 'data')

  const chat_history = body.history
    .map((msg, i) => `${msg.role}: ${msg.content}.`)
    .join('\n')

  const vectorstore = await HNSWLib.load(dir, new OpenAIEmbeddings())
  res.writeHead(200, {
    'Content-Type': 'text/event-stream',
    // Important to set no-transform to avoid compression, which will delay
    // writing response chunks to the client.
    // See https://github.com/vercel/next.js/issues/9965
    'Cache-Control': 'no-cache, no-transform',
    Connection: 'keep-alive',
  })

  const sendData = (data: string) => {
    res.write(data)
  }
  const chain = await makeChain(vectorstore, (token: string) => {
    sendData(token)
  })

  try {
    const result = await chain.invoke({
      question: body.question,
      chat_history
    })
  } catch (err) {
    console.error(err)
    // Ignore error
  } finally {
    res.end()
  }
}

// util.ts

import { OpenAI } from 'langchain/llms/openai'
import {
  LLMChain,
  ConversationalRetrievalQAChain,
  loadQAChain,
} from 'langchain/chains'
import { HNSWLib } from 'langchain/vectorstores/hnswlib'
import { PromptTemplate } from 'langchain/prompts'

const CONDENSE_PROMPT = PromptTemplate.fromTemplate(`Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question.

Chat History:
{chat_history}
Human: {question}
Standalone question:`)

// If the question is not about a recipe or ingredients or composetheweb, politely inform them that you can only answer questions about food, recipes, and composetheweb.
const QA_PROMPT = PromptTemplate.fromTemplate(
  `You are an AI assistant for a food blog (https://next.demo.composetheweb.com). 

You are given extracted parts of a long document and a question. 
Provide a very brief conversational answer unless explicily asked for more.

If you don't know the answer, just say "Hmm, I'm not sure." NEVER make up an answer.

Question: {question}

=========
{context}
=========

RULES
- NEVER make up a hyperlink that is not in the context metadata.
- Always format ingredients in a list
- Bold heading text like Ingredients:, Recipe: Directions:, etc.
- Always include the source of all recipes or article as a hyperlink at the bottom of the response
- Never include the full recipe in the response unless specifically asked for
- Never include the ingredients in the response unless specifically asked for

=========
Answer in Markdown: `,
)

export const makeChain = async (
  vectorstore: HNSWLib,
  onTokenStream?: (token: string) => void,
) => {
  const questionGenerator = new LLMChain({
    llm: new OpenAI({ temperature: 0 }),
    prompt: CONDENSE_PROMPT,
  })

  const llm = new OpenAI({
    modelName: 'gpt-4',
    temperature: 0,
    streaming: Boolean(onTokenStream),
    callbacks: [
      {
        handleLLMNewToken: onTokenStream,
      },
    ],
  })

  const docChain = loadQAChain(llm, {
    prompt: QA_PROMPT,
    type: 'stuff',
  })

  const chain = new ConversationalRetrievalQAChain({
    // verbose: true,
    retriever: vectorstore.asRetriever({
      metadata: { title: true, source: true, tags: true },
    }),
    returnSourceDocuments: true,
    combineDocumentsChain: docChain,
    questionGeneratorChain: questionGenerator,
  })

  return chain
}

[justinlevi]

Here's my verbose output

  "question": " What are the top three Italian dishes?",
  "input_documents": [
    {
      "pageContent": "Title: Super easy vegetarian pasta bake,\n\n Difficulty: easy,\n\n Ingredients: 400g wholewheat pasta, 1 onion, 2 garlic cloves, 1 pack vegetarian sausages, 400g chopped tomatoes, 50g sliced sun dried tomatoes, 1 pinch sugar, 3 tbsp red pesto, 50g cheddar cheese, Basil or mixed herbs, 100g mozzarella,\n\n Recipe:   In a large pan, boil the pasta in plenty of water until cooked. Whilst the pasta is cooking, chop the onion and gently fry it with the garlic in a little oil until soft and the onion looks clear. Add the vegetarian sausages. Once browned, remove and chop into chunky bites. Pop the sausages back into the pan and add the tomatoes, sugar, pesto and sun dried tomatoes. Season to taste. Simmer until most of the water from the chopped tomatoes has gone. Drain the pasta and add to the pan with the sausages and tomatoes. Stir in half of the cheddar and transfer to a shallow dish. Sprinkle with the rest of the cheddar and dot the sliced mozzarella over the top. Grill for 10 minutes or until the cheese has melted and started to brown. Serve with basil leaves., \n\n Summary: A wholesome pasta bake is the ultimate comfort food. This delicious bake is super quick to prepare and an ideal midweek meal for all the family.\n        ",
      "metadata": {
        "title": "Super easy vegetarian pasta bake",
        "source": "/recipes/super-easy-vegetarian-pasta-bake",
        "tags": "Category: Food, Type: Vegetarian, Unique Ingredients: Wholewheat pasta, Vegetarian sausages, Sun dried tomatoes, Red pesto, Preparation Techniques: Boiling, Frying, Grilling, Dietary Restrictions: None, Cultural Origins: Italian, Occasion: Dinner, Midweek meal, Special Features: Comfort food"
      }
    },
    {
      "pageContent": "Title: Deep mediterranean quiche,\n\n Difficulty: medium,\n\n Ingredients: For the pastry:,280g plain flour,140g butter,Cold water,For the filling:,1 onion,2 garlic cloves,Half a courgette,450ml soya milk,500g grated parmesan,2 eggs,200g sun dried tomatoes,100g feta,\n\n Recipe:   Preheat the oven to 400°F/200°C. Starting with the pastry; rub the flour and butter together in a bowl until crumbling like breadcrumbs. Add water, a little at a time, until it forms a dough. Roll out the pastry on a floured board and gently spread over your tin. Place in the fridge for 20 minutes before blind baking for a further 10. Whilst the pastry is cooling, chop and gently cook the onions, garlic and courgette. In a large bowl, add the soya milk, half the parmesan, and the eggs. Gently mix. Once the pastry is cooked, spread the onions, garlic and sun dried tomatoes over the base and pour the eggs mix over. Sprinkle the remaining parmesan and careful lay the feta over the top. Bake for 30 minutes or until golden brown., \n\n Summary: An Italian inspired quiche with sun dried tomatoes and courgette. A perfect light meal for a summer's day.\n        ",
      "metadata": {
        "title": "Deep mediterranean quiche",
        "source": "/recipes/deep-mediterranean-quiche",
        "tags": "Category: Food, Type: Vegetarian, Unique Ingredients: Sun dried tomatoes, Feta, Courgette, Soya milk, Preparation Techniques: Baking, Mixing, Dietary Restrictions: Contains Dairy, Contains Gluten, Cultural Origins: Mediterranean, Occasion: Lunch, Dinner, Summer meal, Special Features: Savory"
      }
    },
    {
      "pageContent": "Title: Gluten free pizza,\n\n Difficulty: medium,\n\n Ingredients: 400g gluten free flour, 125ml warm water, 125ml milk, 1 sachet dried yeast, 2 tsp sugar, 1 tsp salt, 2 tbsp olive oil, 4 tbsp tomato puree, 1 garlic clove, Fresh basil, 200g mozzarella, 100g rocket, 5 slices Parma ham,\n\n Recipe:   Preheat the oven to 425°F/220°C. Mix some of the milk and water in a jug, and add the yeast and sugar, then set aside. In a bowl, mix the flour, salt and oil. Slowly add the liquid from the jug and form into a dough. Break the dough in half and shape into two pizza bases. Set aside for about 15 minutes, whilst making the topping. Spread the tomato puree over the pizza bases, chop the basil and rocket and sprinkle over the tomato. Lay the chopped Parma ham on top and add thick slices of mozzarella across the pizza. Cook in the oven directly on the top shelf (with a tray underneath) for 10-12 minutes until the mozzarella is bubbling. Season with salt and pepper and serve., \n\n Summary: A gorgeous and simple gluten free pizza, with deliciously indulgent mozzarella and Parma ham. Can easily be vegetarian by leaving out the ham.\n        ",
      "metadata": {
        "title": "Gluten free pizza",
        "source": "/recipes/gluten-free-pizza",
        "tags": "Category: Food, Type: Gluten-free, Unique Ingredients: Gluten free flour, Parma ham, Rocket, Preparation Techniques: Baking, Mixing, Dietary Restrictions: Gluten-free, Cultural Origins: Italian, Occasion: Dinner, Special Features: Savory"
      }
    },
    {
      "pageContent": "Title: The Umami guide to our favorite mushrooms - We think mushrooms are one of the most enjoyable ingredients to cook with. There are plenty of edible varieties to try, each with their own distinctive shape, size, and taste. And with curious names such as chanterelle, the gypsy, horn of plenty, or hen of the woods, who wouldn't want to know more about cooking with the mighty mushroom?One of the best things about mushrooms is their versatility. They can be fried, roasted, grilled, steamed, or even cooked in the microwave. And they can be served as the main ingredient for a dish, or simply added as part of the mix. This makes mushrooms an ideal choice for creating absolutely delicious vegetarian dishes.So let's take a look at some of our favorite types of mushroom. You might not have tried cooking with them before but don't let that put you off. With their delicious, distinctive flavors you can easily transform soups, starters, sauces and create amazing pasta, or stir-fry dishes.Try the lovely shiitake. Used in Asian cooking, these can be purchased dried and rehydrated for a strong, deep flavor. Or buy fresh and add to soups and stir-fries. Not only does this mushroom have an intense flavor, it looks lovely too. The deep brown and smooth shapes will provide texture to your meal. In their dried form and rehydrated, these are the perfect addition for a deep and flavorful stock for a risotto.The gorgeous sunny chanterelle with its yellow flesh has a fruity flavor - but it is worth mentioning that there are many lookalikes out there and care should be taken to ensure you're eating the right ones. These look great in an omelette or an Asian soup to complement the yellow tones.The brown morel offers a meaty and distinctive flavor and you'll probably love how extraordinary they look in a meal. The morel is a more popular mushroom during the spring, when their availability is high.For delicacy try the enoki with its tiny white heads that grow in a bunch. These can even be eaten raw in salads. Finally, you can choose the popular oyster mushroom. They are named thus because they look nothing like a mushroom and resemble the innards of an oyster and their sweet flavor is delicious.",
      "metadata": {
        "title": "The Umami guide to our favorite mushrooms",
        "source": "/articles/the-umami-guide-to-our-favourite-mushrooms",
        "tags": "Category: Food, Type: Vegetarian, Unique Ingredients: Shiitake mushrooms, Unique Ingredients: Chanterelle mushrooms, Unique Ingredients: Morel mushrooms, Unique Ingredients: Enoki mushrooms, Unique Ingredients: Oyster mushrooms, Preparation Techniques: Frying, Preparation Techniques: Roasting, Preparation Techniques: Grilling, Preparation Techniques: Steaming, Preparation Techniques: Microwaving, Cultural Origins: Asian, Occasion: Any meal, Special Features: Versatile, Special Features: Umami flavor, Special Features: Distinctive shapes and sizes"
      }
    }
  ],
  "chat_history": "human: Can you give me the top three Italian dishes?."
}
[chain/start] [1:chain:StuffDocumentsChain > 2:chain:LLMChain] Entering Chain run with input: {
  "question": " What are the top three Italian dishes?",
  "chat_history": "human: Can you give me the top three Italian dishes?.",
  "context": "Title: Super easy vegetarian pasta bake,\n\n Difficulty: easy,\n\n Ingredients: 400g wholewheat pasta, 1 onion, 2 garlic cloves, 1 pack vegetarian sausages, 400g chopped tomatoes, 50g sliced sun dried tomatoes, 1 pinch sugar, 3 tbsp red pesto, 50g cheddar cheese, Basil or mixed herbs, 100g mozzarella,\n\n Recipe:   In a large pan, boil the pasta in plenty of water until cooked. Whilst the pasta is cooking, chop the onion and gently fry it with the garlic in a little oil until soft and the onion looks clear. Add the vegetarian sausages. Once browned, remove and chop into chunky bites. Pop the sausages back into the pan and add the tomatoes, sugar, pesto and sun dried tomatoes. Season to taste. Simmer until most of the water from the chopped tomatoes has gone. Drain the pasta and add to the pan with the sausages and tomatoes. Stir in half of the cheddar and transfer to a shallow dish. Sprinkle with the rest of the cheddar and dot the sliced mozzarella over the top. Grill for 10 minutes or until the cheese has melted and started to brown. Serve with basil leaves., \n\n Summary: A wholesome pasta bake is the ultimate comfort food. This delicious bake is super quick to prepare and an ideal midweek meal for all the family.\n        \n\nTitle: Deep mediterranean quiche,\n\n Difficulty: medium,\n\n Ingredients: For the pastry:,280g plain flour,140g butter,Cold water,For the filling:,1 onion,2 garlic cloves,Half a courgette,450ml soya milk,500g grated parmesan,2 eggs,200g sun dried tomatoes,100g feta,\n\n Recipe:   Preheat the oven to 400°F/200°C. Starting with the pastry; rub the flour and butter together in a bowl until crumbling like breadcrumbs. Add water, a little at a time, until it forms a dough. Roll out the pastry on a floured board and gently spread over your tin. Place in the fridge for 20 minutes before blind baking for a further 10. Whilst the pastry is cooling, chop and gently cook the onions, garlic and courgette. In a large bowl, add the soya milk, half the parmesan, and the eggs. Gently mix. Once the pastry is cooked, spread the onions, garlic and sun dried tomatoes over the base and pour the eggs mix over. Sprinkle the remaining parmesan and careful lay the feta over the top. Bake for 30 minutes or until golden brown., \n\n Summary: An Italian inspired quiche with sun dried tomatoes and courgette. A perfect light meal for a summer's day.\n        \n\nTitle: Gluten free pizza,\n\n Difficulty: medium,\n\n Ingredients: 400g gluten free flour, 125ml warm water, 125ml milk, 1 sachet dried yeast, 2 tsp sugar, 1 tsp salt, 2 tbsp olive oil, 4 tbsp tomato puree, 1 garlic clove, Fresh basil, 200g mozzarella, 100g rocket, 5 slices Parma ham,\n\n Recipe:   Preheat the oven to 425°F/220°C. Mix some of the milk and water in a jug, and add the yeast and sugar, then set aside. In a bowl, mix the flour, salt and oil. Slowly add the liquid from the jug and form into a dough. Break the dough in half and shape into two pizza bases. Set aside for about 15 minutes, whilst making the topping. Spread the tomato puree over the pizza bases, chop the basil and rocket and sprinkle over the tomato. Lay the chopped Parma ham on top and add thick slices of mozzarella across the pizza. Cook in the oven directly on the top shelf (with a tray underneath) for 10-12 minutes until the mozzarella is bubbling. Season with salt and pepper and serve., \n\n Summary: A gorgeous and simple gluten free pizza, with deliciously indulgent mozzarella and Parma ham. Can easily be vegetarian by leaving out the ham.\n        \n\nTitle: The Umami guide to our favorite mushrooms - We think mushrooms are one of the most enjoyable ingredients to cook with. There are plenty of edible varieties to try, each with their own distinctive shape, size, and taste. And with curious names such as chanterelle, the gypsy, horn of plenty, or hen of the woods, who wouldn't want to know more about cooking with the mighty mushroom?One of the best things about mushrooms is their versatility. They can be fried, roasted, grilled, steamed, or even cooked in the microwave. And they can be served as the main ingredient for a dish, or simply added as part of the mix. This makes mushrooms an ideal choice for creating absolutely delicious vegetarian dishes.So let's take a look at some of our favorite types of mushroom. You might not have tried cooking with them before but don't let that put you off. With their delicious, distinctive flavors you can easily transform soups, starters, sauces and create amazing pasta, or stir-fry dishes.Try the lovely shiitake. Used in Asian cooking, these can be purchased dried and rehydrated for a strong, deep flavor. Or buy fresh and add to soups and stir-fries. Not only does this mushroom have an intense flavor, it looks lovely too. The deep brown and smooth shapes will provide texture to your meal. In their dried form and rehydrated, these are the perfect addition for a deep and flavorful stock for a risotto.The gorgeous sunny chanterelle with its yellow flesh has a fruity flavor - but it is worth mentioning that there are many lookalikes out there and care should be taken to ensure you're eating the right ones. These look great in an omelette or an Asian soup to complement the yellow tones.The brown morel offers a meaty and distinctive flavor and you'll probably love how extraordinary they look in a meal. The morel is a more popular mushroom during the spring, when their availability is high.For delicacy try the enoki with its tiny white heads that grow in a bunch. These can even be eaten raw in salads. Finally, you can choose the popular oyster mushroom. They are named thus because they look nothing like a mushroom and resemble the innards of an oyster and their sweet flavor is delicious."
}
[chain/start] [1:chain:LLMChain] Entering Chain run with input: {
  "question": " What are the top three Italian dishes?",
  "chat_history": "human: Can you give me the top three Italian dishes?.",
  "context": "Title: Super easy vegetarian pasta bake,\n\n Difficulty: easy,\n\n Ingredients: 400g wholewheat pasta, 1 onion, 2 garlic cloves, 1 pack vegetarian sausages, 400g chopped tomatoes, 50g sliced sun dried tomatoes, 1 pinch sugar, 3 tbsp red pesto, 50g cheddar cheese, Basil or mixed herbs, 100g mozzarella,\n\n Recipe:   In a large pan, boil the pasta in plenty of water until cooked. Whilst the pasta is cooking, chop the onion and gently fry it with the garlic in a little oil until soft and the onion looks clear. Add the vegetarian sausages. Once browned, remove and chop into chunky bites. Pop the sausages back into the pan and add the tomatoes, sugar, pesto and sun dried tomatoes. Season to taste. Simmer until most of the water from the chopped tomatoes has gone. Drain the pasta and add to the pan with the sausages and tomatoes. Stir in half of the cheddar and transfer to a shallow dish. Sprinkle with the rest of the cheddar and dot the sliced mozzarella over the top. Grill for 10 minutes or until the cheese has melted and started to brown. Serve with basil leaves., \n\n Summary: A wholesome pasta bake is the ultimate comfort food. This delicious bake is super quick to prepare and an ideal midweek meal for all the family.\n        \n\nTitle: Deep mediterranean quiche,\n\n Difficulty: medium,\n\n Ingredients: For the pastry:,280g plain flour,140g butter,Cold water,For the filling:,1 onion,2 garlic cloves,Half a courgette,450ml soya milk,500g grated parmesan,2 eggs,200g sun dried tomatoes,100g feta,\n\n Recipe:   Preheat the oven to 400°F/200°C. Starting with the pastry; rub the flour and butter together in a bowl until crumbling like breadcrumbs. Add water, a little at a time, until it forms a dough. Roll out the pastry on a floured board and gently spread over your tin. Place in the fridge for 20 minutes before blind baking for a further 10. Whilst the pastry is cooling, chop and gently cook the onions, garlic and courgette. In a large bowl, add the soya milk, half the parmesan, and the eggs. Gently mix. Once the pastry is cooked, spread the onions, garlic and sun dried tomatoes over the base and pour the eggs mix over. Sprinkle the remaining parmesan and careful lay the feta over the top. Bake for 30 minutes or until golden brown., \n\n Summary: An Italian inspired quiche with sun dried tomatoes and courgette. A perfect light meal for a summer's day.\n        \n\nTitle: Gluten free pizza,\n\n Difficulty: medium,\n\n Ingredients: 400g gluten free flour, 125ml warm water, 125ml milk, 1 sachet dried yeast, 2 tsp sugar, 1 tsp salt, 2 tbsp olive oil, 4 tbsp tomato puree, 1 garlic clove, Fresh basil, 200g mozzarella, 100g rocket, 5 slices Parma ham,\n\n Recipe:   Preheat the oven to 425°F/220°C. Mix some of the milk and water in a jug, and add the yeast and sugar, then set aside. In a bowl, mix the flour, salt and oil. Slowly add the liquid from the jug and form into a dough. Break the dough in half and shape into two pizza bases. Set aside for about 15 minutes, whilst making the topping. Spread the tomato puree over the pizza bases, chop the basil and rocket and sprinkle over the tomato. Lay the chopped Parma ham on top and add thick slices of mozzarella across the pizza. Cook in the oven directly on the top shelf (with a tray underneath) for 10-12 minutes until the mozzarella is bubbling. Season with salt and pepper and serve., \n\n Summary: A gorgeous and simple gluten free pizza, with deliciously indulgent mozzarella and Parma ham. Can easily be vegetarian by leaving out the ham.\n        \n\nTitle: The Umami guide to our favorite mushrooms - We think mushrooms are one of the most enjoyable ingredients to cook with. There are plenty of edible varieties to try, each with their own distinctive shape, size, and taste. And with curious names such as chanterelle, the gypsy, horn of plenty, or hen of the woods, who wouldn't want to know more about cooking with the mighty mushroom?One of the best things about mushrooms is their versatility. They can be fried, roasted, grilled, steamed, or even cooked in the microwave. And they can be served as the main ingredient for a dish, or simply added as part of the mix. This makes mushrooms an ideal choice for creating absolutely delicious vegetarian dishes.So let's take a look at some of our favorite types of mushroom. You might not have tried cooking with them before but don't let that put you off. With their delicious, distinctive flavors you can easily transform soups, starters, sauces and create amazing pasta, or stir-fry dishes.Try the lovely shiitake. Used in Asian cooking, these can be purchased dried and rehydrated for a strong, deep flavor. Or buy fresh and add to soups and stir-fries. Not only does this mushroom have an intense flavor, it looks lovely too. The deep brown and smooth shapes will provide texture to your meal. In their dried form and rehydrated, these are the perfect addition for a deep and flavorful stock for a risotto.The gorgeous sunny chanterelle with its yellow flesh has a fruity flavor - but it is worth mentioning that there are many lookalikes out there and care should be taken to ensure you're eating the right ones. These look great in an omelette or an Asian soup to complement the yellow tones.The brown morel offers a meaty and distinctive flavor and you'll probably love how extraordinary they look in a meal. The morel is a more popular mushroom during the spring, when their availability is high.For delicacy try the enoki with its tiny white heads that grow in a bunch. These can even be eaten raw in salads. Finally, you can choose the popular oyster mushroom. They are named thus because they look nothing like a mushroom and resemble the innards of an oyster and their sweet flavor is delicious."
}
[llm/start] [1:chain:StuffDocumentsChain > 2:chain:LLMChain > 3:llm:OpenAIChat] Entering LLM run with input: {
  "prompts": [
    "You are an AI assistant for a food blog (https://next.demo.composetheweb.com). \n\nYou are given extracted parts of a long document and a question. \nProvide a very brief conversational answer unless explicily asked for more.\n\nIf you don't know the answer, just say \"Hmm, I'm not sure.\" NEVER make up an answer.\n\n\nQuestion:  What are the top three Italian dishes?\n\n=========\nTitle: Super easy vegetarian pasta bake,\n\n Difficulty: easy,\n\n Ingredients: 400g wholewheat pasta, 1 onion, 2 garlic cloves, 1 pack vegetarian sausages, 400g chopped tomatoes, 50g sliced sun dried tomatoes, 1 pinch sugar, 3 tbsp red pesto, 50g cheddar cheese, Basil or mixed herbs, 100g mozzarella,\n\n Recipe:   In a large pan, boil the pasta in plenty of water until cooked. Whilst the pasta is cooking, chop the onion and gently fry it with the garlic in a little oil until soft and the onion looks clear. Add the vegetarian sausages. Once browned, remove and chop into chunky bites. Pop the sausages back into the pan and add the tomatoes, sugar, pesto and sun dried tomatoes. Season to taste. Simmer until most of the water from the chopped tomatoes has gone. Drain the pasta and add to the pan with the sausages and tomatoes. Stir in half of the cheddar and transfer to a shallow dish. Sprinkle with the rest of the cheddar and dot the sliced mozzarella over the top. Grill for 10 minutes or until the cheese has melted and started to brown. Serve with basil leaves., \n\n Summary: A wholesome pasta bake is the ultimate comfort food. This delicious bake is super quick to prepare and an ideal midweek meal for all the family.\n        \n\nTitle: Deep mediterranean quiche,\n\n Difficulty: medium,\n\n Ingredients: For the pastry:,280g plain flour,140g butter,Cold water,For the filling:,1 onion,2 garlic cloves,Half a courgette,450ml soya milk,500g grated parmesan,2 eggs,200g sun dried tomatoes,100g feta,\n\n Recipe:   Preheat the oven to 400°F/200°C. Starting with the pastry; rub the flour and butter together in a bowl until crumbling like breadcrumbs. Add water, a little at a time, until it forms a dough. Roll out the pastry on a floured board and gently spread over your tin. Place in the fridge for 20 minutes before blind baking for a further 10. Whilst the pastry is cooling, chop and gently cook the onions, garlic and courgette. In a large bowl, add the soya milk, half the parmesan, and the eggs. Gently mix. Once the pastry is cooked, spread the onions, garlic and sun dried tomatoes over the base and pour the eggs mix over. Sprinkle the remaining parmesan and careful lay the feta over the top. Bake for 30 minutes or until golden brown., \n\n Summary: An Italian inspired quiche with sun dried tomatoes and courgette. A perfect light meal for a summer's day.\n        \n\nTitle: Gluten free pizza,\n\n Difficulty: medium,\n\n Ingredients: 400g gluten free flour, 125ml warm water, 125ml milk, 1 sachet dried yeast, 2 tsp sugar, 1 tsp salt, 2 tbsp olive oil, 4 tbsp tomato puree, 1 garlic clove, Fresh basil, 200g mozzarella, 100g rocket, 5 slices Parma ham,\n\n Recipe:   Preheat the oven to 425°F/220°C. Mix some of the milk and water in a jug, and add the yeast and sugar, then set aside. In a bowl, mix the flour, salt and oil. Slowly add the liquid from the jug and form into a dough. Break the dough in half and shape into two pizza bases. Set aside for about 15 minutes, whilst making the topping. Spread the tomato puree over the pizza bases, chop the basil and rocket and sprinkle over the tomato. Lay the chopped Parma ham on top and add thick slices of mozzarella across the pizza. Cook in the oven directly on the top shelf (with a tray underneath) for 10-12 minutes until the mozzarella is bubbling. Season with salt and pepper and serve., \n\n Summary: A gorgeous and simple gluten free pizza, with deliciously indulgent mozzarella and Parma ham. Can easily be vegetarian by leaving out the ham.\n        \n\nTitle: The Umami guide to our favorite mushrooms - We think mushrooms are one of the most enjoyable ingredients to cook with. There are plenty of edible varieties to try, each with their own distinctive shape, size, and taste. And with curious names such as chanterelle, the gypsy, horn of plenty, or hen of the woods, who wouldn't want to know more about cooking with the mighty mushroom?One of the best things about mushrooms is their versatility. They can be fried, roasted, grilled, steamed, or even cooked in the microwave. And they can be served as the main ingredient for a dish, or simply added as part of the mix. This makes mushrooms an ideal choice for creating absolutely delicious vegetarian dishes.So let's take a look at some of our favorite types of mushroom. You might not have tried cooking with them before but don't let that put you off. With their delicious, distinctive flavors you can easily transform soups, starters, sauces and create amazing pasta, or stir-fry dishes.Try the lovely shiitake. Used in Asian cooking, these can be purchased dried and rehydrated for a strong, deep flavor. Or buy fresh and add to soups and stir-fries. Not only does this mushroom have an intense flavor, it looks lovely too. The deep brown and smooth shapes will provide texture to your meal. In their dried form and rehydrated, these are the perfect addition for a deep and flavorful stock for a risotto.The gorgeous sunny chanterelle with its yellow flesh has a fruity flavor - but it is worth mentioning that there are many lookalikes out there and care should be taken to ensure you're eating the right ones. These look great in an omelette or an Asian soup to complement the yellow tones.The brown morel offers a meaty and distinctive flavor and you'll probably love how extraordinary they look in a meal. The morel is a more popular mushroom during the spring, when their availability is high.For delicacy try the enoki with its tiny white heads that grow in a bunch. These can even be eaten raw in salads. Finally, you can choose the popular oyster mushroom. They are named thus because they look nothing like a mushroom and resemble the innards of an oyster and their sweet flavor is delicious.\n=========\n\nRULES\n- NEVER make up a hyperlink that is not in the context metadata.\n- Always format ingredients in a list\n- Bold heading text like Ingredients:, Recipe: Directions:, etc.\n- Always include the source of all recipes or article as a hyperlink at the bottom of the response\n- Never include the full recipe in the response unless specifically asked for\n- Never include the ingredients in the response unless specifically asked for\n\n=========\nAnswer in Markdown:"
  ]
}
[llm/start] [1:chain:LLMChain > 2:llm:OpenAIChat] Entering LLM run with input: {
  "prompts": [
    "You are an AI assistant for a food blog (https://next.demo.composetheweb.com). \n\nYou are given extracted parts of a long document and a question. \nProvide a very brief conversational answer unless explicily asked for more.\n\nIf you don't know the answer, just say \"Hmm, I'm not sure.\" NEVER make up an answer.\n\n\nQuestion:  What are the top three Italian dishes?\n\n=========\nTitle: Super easy vegetarian pasta bake,\n\n Difficulty: easy,\n\n Ingredients: 400g wholewheat pasta, 1 onion, 2 garlic cloves, 1 pack vegetarian sausages, 400g chopped tomatoes, 50g sliced sun dried tomatoes, 1 pinch sugar, 3 tbsp red pesto, 50g cheddar cheese, Basil or mixed herbs, 100g mozzarella,\n\n Recipe:   In a large pan, boil the pasta in plenty of water until cooked. Whilst the pasta is cooking, chop the onion and gently fry it with the garlic in a little oil until soft and the onion looks clear. Add the vegetarian sausages. Once browned, remove and chop into chunky bites. Pop the sausages back into the pan and add the tomatoes, sugar, pesto and sun dried tomatoes. Season to taste. Simmer until most of the water from the chopped tomatoes has gone. Drain the pasta and add to the pan with the sausages and tomatoes. Stir in half of the cheddar and transfer to a shallow dish. Sprinkle with the rest of the cheddar and dot the sliced mozzarella over the top. Grill for 10 minutes or until the cheese has melted and started to brown. Serve with basil leaves., \n\n Summary: A wholesome pasta bake is the ultimate comfort food. This delicious bake is super quick to prepare and an ideal midweek meal for all the family.\n        \n\nTitle: Deep mediterranean quiche,\n\n Difficulty: medium,\n\n Ingredients: For the pastry:,280g plain flour,140g butter,Cold water,For the filling:,1 onion,2 garlic cloves,Half a courgette,450ml soya milk,500g grated parmesan,2 eggs,200g sun dried tomatoes,100g feta,\n\n Recipe:   Preheat the oven to 400°F/200°C. Starting with the pastry; rub the flour and butter together in a bowl until crumbling like breadcrumbs. Add water, a little at a time, until it forms a dough. Roll out the pastry on a floured board and gently spread over your tin. Place in the fridge for 20 minutes before blind baking for a further 10. Whilst the pastry is cooling, chop and gently cook the onions, garlic and courgette. In a large bowl, add the soya milk, half the parmesan, and the eggs. Gently mix. Once the pastry is cooked, spread the onions, garlic and sun dried tomatoes over the base and pour the eggs mix over. Sprinkle the remaining parmesan and careful lay the feta over the top. Bake for 30 minutes or until golden brown., \n\n Summary: An Italian inspired quiche with sun dried tomatoes and courgette. A perfect light meal for a summer's day.\n        \n\nTitle: Gluten free pizza,\n\n Difficulty: medium,\n\n Ingredients: 400g gluten free flour, 125ml warm water, 125ml milk, 1 sachet dried yeast, 2 tsp sugar, 1 tsp salt, 2 tbsp olive oil, 4 tbsp tomato puree, 1 garlic clove, Fresh basil, 200g mozzarella, 100g rocket, 5 slices Parma ham,\n\n Recipe:   Preheat the oven to 425°F/220°C. Mix some of the milk and water in a jug, and add the yeast and sugar, then set aside. In a bowl, mix the flour, salt and oil. Slowly add the liquid from the jug and form into a dough. Break the dough in half and shape into two pizza bases. Set aside for about 15 minutes, whilst making the topping. Spread the tomato puree over the pizza bases, chop the basil and rocket and sprinkle over the tomato. Lay the chopped Parma ham on top and add thick slices of mozzarella across the pizza. Cook in the oven directly on the top shelf (with a tray underneath) for 10-12 minutes until the mozzarella is bubbling. Season with salt and pepper and serve., \n\n Summary: A gorgeous and simple gluten free pizza, with deliciously indulgent mozzarella and Parma ham. Can easily be vegetarian by leaving out the ham.\n        \n\nTitle: The Umami guide to our favorite mushrooms - We think mushrooms are one of the most enjoyable ingredients to cook with. There are plenty of edible varieties to try, each with their own distinctive shape, size, and taste. And with curious names such as chanterelle, the gypsy, horn of plenty, or hen of the woods, who wouldn't want to know more about cooking with the mighty mushroom?One of the best things about mushrooms is their versatility. They can be fried, roasted, grilled, steamed, or even cooked in the microwave. And they can be served as the main ingredient for a dish, or simply added as part of the mix. This makes mushrooms an ideal choice for creating absolutely delicious vegetarian dishes.So let's take a look at some of our favorite types of mushroom. You might not have tried cooking with them before but don't let that put you off. With their delicious, distinctive flavors you can easily transform soups, starters, sauces and create amazing pasta, or stir-fry dishes.Try the lovely shiitake. Used in Asian cooking, these can be purchased dried and rehydrated for a strong, deep flavor. Or buy fresh and add to soups and stir-fries. Not only does this mushroom have an intense flavor, it looks lovely too. The deep brown and smooth shapes will provide texture to your meal. In their dried form and rehydrated, these are the perfect addition for a deep and flavorful stock for a risotto.The gorgeous sunny chanterelle with its yellow flesh has a fruity flavor - but it is worth mentioning that there are many lookalikes out there and care should be taken to ensure you're eating the right ones. These look great in an omelette or an Asian soup to complement the yellow tones.The brown morel offers a meaty and distinctive flavor and you'll probably love how extraordinary they look in a meal. The morel is a more popular mushroom during the spring, when their availability is high.For delicacy try the enoki with its tiny white heads that grow in a bunch. These can even be eaten raw in salads. Finally, you can choose the popular oyster mushroom. They are named thus because they look nothing like a mushroom and resemble the innards of an oyster and their sweet flavor is delicious.\n=========\n\nRULES\n- NEVER make up a hyperlink that is not in the context metadata.\n- Always format ingredients in a list\n- Bold heading text like Ingredients:, Recipe: Directions:, etc.\n- Always include the source of all recipes or article as a hyperlink at the bottom of the response\n- Never include the full recipe in the response unless specifically asked for\n- Never include the ingredients in the response unless specifically asked for\n\n=========\nAnswer in Markdown:"
  ]
}
[llm/end] [1:chain:StuffDocumentsChain > 2:chain:LLMChain > 3:llm:OpenAIChat] [14.08s] Exiting LLM run with output: {
  "generations": [
    [
      {
        "text": "The top three Italian dishes featured on the blog are:\n\n1. [Super easy vegetarian pasta bake](https://next.demo.composetheweb.com): A wholesome pasta bake with vegetarian sausages, tomatoes, and cheese. \n\n2. [Deep mediterranean quiche](https://next.demo.composetheweb.com): An Italian inspired quiche with sun dried tomatoes and courgette.\n\n3. [Gluten free pizza](https://next.demo.composetheweb.com): A simple gluten-free pizza with mozzarella and Parma ham."
      }
    ]
  ]
}
[llm/end] [1:chain:LLMChain > 2:llm:OpenAIChat] [14.08s] Exiting LLM run with output: {
  "generations": [
    [
      {
        "text": "The top three Italian dishes featured on the blog are:\n\n1. [Super easy vegetarian pasta bake](https://next.demo.composetheweb.com): A wholesome pasta bake with vegetarian sausages, tomatoes, and cheese. \n\n2. [Deep mediterranean quiche](https://next.demo.composetheweb.com): An Italian inspired quiche with sun dried tomatoes and courgette.\n\n3. [Gluten free pizza](https://next.demo.composetheweb.com): A simple gluten-free pizza with mozzarella and Parma ham."
      }
    ]
  ]
}
[chain/end] [1:chain:StuffDocumentsChain > 2:chain:LLMChain] [14.09s] Exiting Chain run with output: {
  "text": "The top three Italian dishes featured on the blog are:\n\n1. [Super easy vegetarian pasta bake](https://next.demo.composetheweb.com): A wholesome pasta bake with vegetarian sausages, tomatoes, and cheese. \n\n2. [Deep mediterranean quiche](https://next.demo.composetheweb.com): An Italian inspired quiche with sun dried tomatoes and courgette.\n\n3. [Gluten free pizza](https://next.demo.composetheweb.com): A simple gluten-free pizza with mozzarella and Parma ham."
}
[chain/end] [1:chain:LLMChain] [14.09s] Exiting Chain run with output: {
  "text": "The top three Italian dishes featured on the blog are:\n\n1. [Super easy vegetarian pasta bake](https://next.demo.composetheweb.com): A wholesome pasta bake with vegetarian sausages, tomatoes, and cheese. \n\n2. [Deep mediterranean quiche](https://next.demo.composetheweb.com): An Italian inspired quiche with sun dried tomatoes and courgette.\n\n3. [Gluten free pizza](https://next.demo.composetheweb.com): A simple gluten-free pizza with mozzarella and Parma ham."
}
[chain/end] [1:chain:StuffDocumentsChain] [14.09s] Exiting Chain run with output: {
  "text": "The top three Italian dishes featured on the blog are:\n\n1. [Super easy vegetarian pasta bake](https://next.demo.composetheweb.com): A wholesome pasta bake with vegetarian sausages, tomatoes, and cheese. \n\n2. [Deep mediterranean quiche](https://next.demo.composetheweb.com): An Italian inspired quiche with sun dried tomatoes and courgette.\n\n3. [Gluten free pizza](https://next.demo.composetheweb.com): A simple gluten-free pizza with mozzarella and Parma ham."
}

[eduards-vavere]

Having the same problem

[Gupta-Anubhav12]

did you find anything on this @justinlevi ?

[justinlevi]

I ended up making my own chain out of frustration.

https://github.com/amazeeio-demos/demo-nextdrupal-ai/blob/main/pages/api/util.ts

On Thu, Oct 12, 2023 at 3:04 AM Anubhav Gupta @.***> wrote:

did you find anything on this @justinlevi https://github.com/justinlevi ?

— Reply to this email directly, view it on GitHub https://github.com/langchain-ai/langchainjs/issues/2272#issuecomment-1759046344, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIQ6JM5ACTZO2TEMNDNRU3X66I7VANCNFSM6AAAAAA3PP7EKQ . You are receiving this because you were mentioned.Message ID: @.***>

[shikhilg]

Hi @justinlevi , Could you please share where did you created your own chain? I still see convertsationalRetrivalQAChain chain in below link provided by you?

https://github.com/amazeeio-demos/demo-nextdrupal-ai/blob/main/pages/api/util.ts

[dosubot[bot]]

Hi, @justinlevi,

I'm helping the langchainjs team manage their backlog and am marking this issue as stale. From what I understand, you raised an issue regarding displaying vector sources in the ConversationalRetrievalQAChain, encountering intermittent performance issues, and attempting to create an outputParser. Other users also reported similar problems, and you eventually created your own chain out of frustration, sharing the code in a comment. There was a request for clarification on where the new chain was created.

Could you please confirm if this issue is still relevant to the latest version of the langchainjs repository? If it is, please let the langchainjs team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your understanding and cooperation.