Langchain Research and if RLHF can be used for tool selection and composition

[danielpatrickhug]

Langchain

Langchain created an OS project for composing and connecting large language model and webtools primarily through the use of chaining prompt templates to obtain some desired behavior. Some examples include:

• agnostic vector database context question answering
• vector database question answering with sources using ApproxNN
• conversation using in-memory chat history and prompt template
• LLM bash command templating
• LLM code completion templating for math problems
• LLM requests chain to get reference context from website and then answer question

Langchain also provides the ability to use the Google search SERP API as well as chain requests similar to how Coheres Search based QA works. However, Lanchain it is done in a way that allows for more complex pipelines. see coheres search.py.

The following is an example of how to use the RequestChain to get the raw text from a google search and then pass it to the OpenAI model to answer a question.

This allows you to compose chains like the following:

from langchain.prompts import PromptTemplate

template = """Between >>> and <<< are the raw search result text from google.
Extract the answer to the question '{query}' or say "not found" if the information is not contained.
Use the format
Extracted:
>>> {requests_result} <<<
Extracted:"""

PROMPT = PromptTemplate(
    input_variables=["query", "requests_result"],
    template=template,
)
chain = LLMRequestsChain(llm_chain = LLMChain(llm=OpenAI(temperature=0), prompt=PROMPT))
question = "What are the Three (3) biggest countries, and their respective sizes?"
inputs = {
    "query": question,
    "url": "https://www.google.com/search?q=" + question.replace(" ", "+")
}
chain(inputs)
'''
{'query': 'What are the Three (3) biggest countries, and their respective sizes?',
 'url': 'https://www.google.com/search?q=What+are+the+Three+(3)+biggest+countries,+and+their+respective+sizes?',
 'output': ' Russia (17,098,242 km²), Canada (9,984,670 km²), United States (9,826,675 km²)'}
'''

For example, you could create a chain that makes an API request to a weather service, processes the response to extract the current temperature and weather conditions, and then passes this information to a reasoning chain that determines whether the weather is suitable for outdoor activities or to suggest clothing using a LLM or category based api.

create a chain that stores and retrieves data from a vector database, and then passes this data to a general llm chain that the data retrieved from the database as context. ("The possibilities are endless." - copilot)

This organically leads to the concept of an Agent which uses prompt templates for complex tasks like self ask behavior using Thought/Action/Action Input/Observation which can repeat N times. prompts can be used to get the agent to use tools for specific problem-solving tasks and self-ask its self reasoning questions and then answer them. For example, the agent could be asked to use a vector database to answer a question, use a knowledge graph like wiki to aggregate links for querying or use a general request chain to get information from a website or api. we could teach the model to generate a image prompt and interact with a midjourney or stable diffusion api or image/audio database and output to discord chat.

Implications for a RLHF model.(Open discussion)

The model could be presented with a set of tools and a task to be completed, and it would be trained to select and compose the tools or prompt templates in a way that maximizes the reward it receives based on the feedback it receives from a human evaluator. i.e the evaluators rank prompts that best uses the set of tools to solve a problem. Ideally, The model would learn to adjust its tool/model selection and usage strategies based on the feedback it receives and the different problems it's solving. How could RLHF be used to reduce the complexity, could it learn to suggest and run its own tools for problems.

Please, let me know what you think. Do you think it would be useful, is this too complex for the current plan?

[mrcabbage972]

Hi again!

Here's a paper that discusses how to train an LLM that would be able to delegate tasks to tools and/or other LLM's: Decomposed Prompting : A MODULAR APPROACH FOR SOLVING COMPLEX TASKS. From the abstract:

a new approach to solve complex tasks by decomposing them (via prompting) into simpler sub-tasks that can be delegated to a shared library of prompting-based LLMs dedicated to these sub-tasks. This modular structure allows each prompt to be optimized for its specific sub-task, further decomposed if necessary, and even easily replaced with more effective prompts, trained models, or symbolic functions if desired.

This should play well with the LangChain approach.

[yk]

This is really cool, I see a lot of potential here, especially as you say, in combination with RLHF. We have to make a more detailed plan on what exactly we want to do with it, or at least come up with a first proof-of-concept use-case to combine this with the assistant.

[danielpatrickhug]

I was thinking of some good success cases:

One could be to ask OA on discord to craft a story, essay or business idea about a topic, and then for each of the 'scenes' in the story create a midjourney style prompt(or find the most similar prompt in a prompt database) and then ask it to interact with a in thread discord bot having it query the image prompts and tell the story step by step. This would be a proof of concept where we can leverage discord infrastructure and show how the bot can be social with other bots in a thread.

Chat GPT does storytelling and breaking up stories to image prompts really well, minus mid journey-specific prompts and the in-thread image gen. I used this strategy to generate this video using stable-diffusion-videos

Another would be to give a video url for youtube, have it download the transcript, (optional)store the transcript in a db, then load the transcript into context and allow the user to ask questions against the transcript. This would be a proof of concept for the bot querying and using info from an arbitrary media source. Downstream, I would like to ask the bot to load a textbook chapter or arxiv paper into context and talk about the concepts, ask it to make quizzes, and work through the problems.

Another idea would to have an instance of the bot managing and using its own prompt template database or ground truth links for making requests and getting context for specific topics.

[danielpatrickhug]

@mrcabbage972 Hi again :) thank you! going through it now

[danielpatrickhug]

Not sure how much this has been discussed but I think the potential for the model to be social with other bots/interfaces in a thread and roll out its reasoning and collaboration. Potentially also, a moderator bot that could stop a chain of collaboration if things get out of hand and then explain its reasoning in the thread. would be something new and cool to see in action.

Chat GPT's summary:

• Collaborating with other bots or models to solve problems or complete tasks: The RLHF model could learn to collaborate with other bots or models in order to solve problems or complete tasks more effectively. For example, it could learn to delegate certain subtasks to other bots or models or to request assistance when it is stuck on a problem.

• Sharing information or resources with other bots or models: The RLHF model could learn to share information or resources with other bots or models in order to improve their performance. For example, it could learn to share data or knowledge that it has acquired with other bots or models in order to help them make better decisions.

• Communicating with other bots or models using natural language: The RLHF model could learn to communicate with other bots or models using natural language, either directly or through a human intermediary. This could involve learning to generate responses to messages from other bots or models, or to initiate conversations with them on its own.

• Participating in group discussions or debates: The RLHF model could learn to participate in group discussions or debates with other bots or models, either by offering its own opinions or by responding to the ideas of others. This could involve learning to identify relevant topics, to present arguments and evidence, and to respond to counterarguments.

[danielpatrickhug]

I attempted to simulate this on chatGPT. it would then need to know to execute a queue of messages in order and wait for another bot to finish. as opposed to cramming it all in one message.

Daniel Hug

I want you to pretend like youre in a discord channel and have access to a set of tools(you can use these tools by typing '/' and then the name of the tool "midjourney" followed by the input "an image of an apple on a hourse" 
the full output will be "/midjourney an image on a horse" only output the prompt "/midjourney an image on a horse" and nothing else when asked to create an image prompt of something.
user: "query mid journey for these two prompts "a horse on an apple --v4" and "an apple on a horse" you would out put first "/midjourney a horse on an apple --v4" and then "an apple on a horse" if you understand just say yes and nothing else and wait for me to ask you about image prompts

ChatGPT

Yes, I understand. Here are the image prompts you requested:

"/midjourney a horse on an apple --v4"
"/midjourney an apple on a horse"

Daniel Hug

Hey chat gpt write me a story about a dog and a cat and then create image prompts for the different scenes in the story and then tell the story querying the midjourney app along the way

ChatGPT

Sure! Here is a story about a dog and a cat:

Once upon a time, there was a friendly dog named Max and a curious cat named Whiskers. Max was a big, fluffy golden retriever with a wagging tail and a loving personality. Whiskers was a small, sleek black cat with bright green eyes and a mischievous streak.

One day, Max and Whiskers were playing in the garden when they stumbled upon a mysterious hole in the ground. "What could be down there?" wondered Max.

"I'll go check it out!" said Whiskers, and she scurried into the hole.

Max, being the brave dog he was, followed close behind. As they descended into the darkness, they discovered a magical underground world filled with glittering jewels and sparkling streams.

"Wow, look at all this treasure!" exclaimed Max.

"We should take some back with us!" said Whiskers.

So, they gathered as many jewels as they could carry and started to make their way back to the surface. But on the way, they were confronted by a group of fierce goblins.

"Give us all your treasure, or else!" growled the goblins.

Max and Whiskers knew they couldn't defeat the goblins on their own, so they came up with a plan. Max distracted the goblins while Whiskers snuck up behind them and used her sharp claws to cut the ropes that held their treasure sacks.

The goblins were so surprised, they ran away in fear. Max and Whiskers emerged from the underground world triumphant, with more treasure than they could have ever imagined.

From then on, Max and Whiskers were the best of friends and went on many more adventures together.

Here are the image prompts for the different scenes in the story:

"/midjourney Max and Whiskers playing in the garden"
"/midjourney Max and Whiskers discovering a magical underground world"
"/midjourney Max and Whiskers gathering treasure"
"/midjourney Max and Whiskers facing a group of goblins"
"/midjourney Max and Whiskers defeating the goblins and emerging triumphant"

Daniel Hug

ok now step by step tell the story and then query the midjourney bot for that specific scene interleaving the story and the images

Chat GPT

Sure! Here is the story of Max and Whiskers, with the image prompts for each scene interleaved:

Once upon a time, there was a friendly dog named Max and a curious cat named Whiskers. Max was a big, fluffy golden retriever with a wagging tail and a loving personality. Whiskers was a small, sleek black cat with bright green eyes and a mischievous streak.

"/midjourney Max and Whiskers playing in the garden"

One day, Max and Whiskers were playing in the garden when they stumbled upon a mysterious hole in the ground. "What could be down there?" wondered Max.

"I'll go check it out!" said Whiskers, and she scurried into the hole.

Max, being the brave dog he was, followed close behind.

"/midjourney Max and Whiskers discovering a magical underground world"

As they descended into the darkness, they discovered a magical underground world filled with glittering jewels and sparkling streams.

"Wow, look at all this treasure!" exclaimed Max.

"We should take some back with us!" said Whiskers.

"/midjourney Max and Whiskers gathering treasure"

So, they gathered as many jewels as they could carry and started to make their way back to the surface. But on the way, they were confronted by a group of fierce goblins.

"Give us all your treasure, or else!" growled the goblins.

"/midjourney Max and Whiskers facing a group of goblins"

Max and Whiskers knew they couldn't defeat the goblins on their own, so they came up with a plan. Max distracted the goblins while Whiskers snuck up behind them and used her sharp claws to cut the ropes that held their treasure sacks.

The goblins were so surprised, they ran away in fear. Max and Whiskers emerged from the underground world triumphant, with more treasure than they could have ever imagined.

"/midjourney Max and Whiskers defeating the goblins and emerging triumphant"

From then on, Max and Whiskers were the best of friends and went on many more adventures together.

[mrcabbage972]

Hey @danielpatrickhug, want to brainstorm this together and figure out what could be a good PoC? I might be soon working on something similar on my day job, so very interested in seeing how well the approach works.

[joshuasv]

Recent Toolformer may be of inspiration.

Takeaways:

• They show step by step how to augment an existing dataset to incorporate API-like prompts.
• Fine-tuned GPT-J in a self-supervised way, no human annotators needed.

Ping me for further discussions, you can find me on the Discord channel by my GitHub handle.

[mrcabbage972]

@joshuasv I'm working on implementing Toolformer. Got the basic algorithm written down but still debugging it. Here's the repo.

[xrsrke]

@mrcabbage972 hi. i implemented ToolFormer (https://github.com/xrsrke/toolformer) a week ago. Feel free to ask me if you need any help

[olliestanley]

OA is now using LangChain and next steps include Toolformer experiments so I think this can be closed