Are there any ways to increase response speed?

[vforv]

Currently it takes 10-15s to get response from OpenAI I am using example similar to this https://langchain.readthedocs.io/en/latest/modules/agents/examples/agent_vectorstore.html

[miguelemosreverte]

bump For me it takes two minutes to get a paragraph

[nikolaspapastavrou]

bump

[vishnumg]

Bump

[RemcoGoy]

bump

[jamescash]

bump

[elizabethsiegle]

bump

[tybalex]

bump

[jamescash]

bump

[vishnuss33]

bump

[navohu]

bump

[nikolaspapastavrou]

Hi all, I have found a way to increase speed when using Agents. The idea is to use the OpenAI LLM for complex tasks such as text generation and small LLMs for Chain of Thought completion. This is inspired by this paper: https://arxiv.org/pdf/2305.17390.pdf

I can create an Agent class for this and make a PR. Would this be of interest?

[Slumber-techlord]

Hey everyone,

I got a slightly faster turnaround time ~5 seconds using the custom QA chain like this:

const docs = await loader.load();

  const textSplitter = new RecursiveCharacterTextSplitter({ chunkSize: 1000, chunkOverlap: 200 });
  const docOutput = await textSplitter.splitDocuments(docs);

  const privateKey = process.env.SUPABASE_PRIVATE_KEY;
  if (!privateKey) throw new Error(`Expected env var SUPABASE_PRIVATE_KEY`);

  const url = process.env.SUPABASE_URL;
  if (!url) throw new Error(`Expected env var SUPABASE_URL`);

  const client = createClient(url, privateKey);

  const vectorStore = await SupabaseVectorStore.fromDocuments(docOutput, new OpenAIEmbeddings(), {
    client,
    tableName: "documents",
  });

const mapreduce_chain = loadQAMapReduceChain(llm);

  const document_chain = new RetrievalQAChain({
    combineDocumentsChain: mapreduce_chain,
    retriever: vectorStore.asRetriever(),
  });

Here, I'm using a VectorStore as the Retriever and a MapReduceDocumentsChain as the QA chain. The VectorStore I've used here is Supabase.

[quynguyen2303]

bump

[adesokanayo]

bump

[tmin97]

bump

[kwehmeyer]

bump

[adesokanayo]

What I have done is to use streaming instead of waiting for the complete openai response. You can get the response via a stream and this has improved the speed of my AI.

You will need to

1. import the LangchainStream

const { stream, handlers } = LangChainStream();

2. set you streaming to true as shown below

const model = new ChatOpenAI({ modelName: "gpt-4", temperature : 0.7, streaming : true, });

and

3. return a streaing text response return new StreamingTextResponse(stream);

Remember not to await the call to once you are streaming.

[thierrymoudiki]

Hi @vforv, did you find a workaround, please?

[gauravingalkar]

facing same issue with llama2 model. the qa chain is super slow. almost 30 seconds to return the answer

[TechGuyVN]

any config to solve this, i need it to make Chatbot, but it too slow

[yishairasowsky]

Hi all, I have found a way to increase speed when using Agents. The idea is to use the OpenAI LLM for complex tasks such as text generation and small LLMs for Chain of Thought completion. This is inspired by this paper: https://arxiv.org/pdf/2305.17390.pdf

I can create an Agent class for this and make a PR. Would this be of interest?

Please if you can provide that it would be greatly appreciated - thank you!

[yishairasowsky]

Hey everyone,

I got a slightly faster turnaround time ~5 seconds using the custom QA chain like this:

const docs = await loader.load();

  const textSplitter = new RecursiveCharacterTextSplitter({ chunkSize: 1000, chunkOverlap: 200 });
  const docOutput = await textSplitter.splitDocuments(docs);

  const privateKey = process.env.SUPABASE_PRIVATE_KEY;
  if (!privateKey) throw new Error(`Expected env var SUPABASE_PRIVATE_KEY`);

  const url = process.env.SUPABASE_URL;
  if (!url) throw new Error(`Expected env var SUPABASE_URL`);

  const client = createClient(url, privateKey);

  const vectorStore = await SupabaseVectorStore.fromDocuments(docOutput, new OpenAIEmbeddings(), {
    client,
    tableName: "documents",
  });

const mapreduce_chain = loadQAMapReduceChain(llm);

  const document_chain = new RetrievalQAChain({
    combineDocumentsChain: mapreduce_chain,
    retriever: vectorStore.asRetriever(),
  });

Here, I'm using a VectorStore as the Retriever and a MapReduceDocumentsChain as the QA chain. The VectorStore I've used here is Supabase.

I use Python so if there's a way you can teach us how to do it that would be great because at the moment I see you using JavaScript