Piping from rg to llm to answer questions about code
Here's a trick I've used a couple of times in the past few days.
If I have a question about a codebase and I want to answer it using an LLM, I'll use ripgrep (aka rg) to search that codebase for a specific symbol and output the 5-10 lines of code that follow that symbol.
Then I'll pipe the result to my llm CLI tool, running through a large context model such as Claude or gpt-4-turbo, and ask it a question about the code.
The question
I wanted to figure out what "read ribbons" are in the source code for https://lobste.rs/ - which I checked out from GitHub like this:
cd /tmp
git clone https://github.com/lobsters/lobsters
cd lobsters
The recipe
Here's the recipe I use, explained in full below:
rg -I -A 5 read_ribbons | \\
llm --system 'explain what read ribbons are' -m gpt-4-turbo
Which gave me:
Based on the SQL queries and the schema definitions you've provided, it seems "read ribbons" is a concept related to tracking user interactions with stories or posts in a social platform or content management system, possibly a forum, news aggregator, or a blog commenting system.
Here's an explanation of key components related to read ribbons and their functionality as inferred from the provided SQL snippets:
Purpose: Read ribbons seem to serve the purpose of tracking whether users have read or followed specific stories or posts. This feature is likely used to manage and display unread content to users, enhancing their experience by making it easier to catch up on what's new or changed since their last visit.
Database Structure:
The read_ribbons table has a few important fields: is_following (a boolean indicating if a user is following a story), created_at and updated_at (timestamps for tracking when the read ribbon was created or last updated), user_id, and story_id (foreign keys linking to the users and stories tables respectively). It essentially creates a many-to-many relationship between users and stories, with the additional context about following status and read status.
There are indices on user_id and story_id for performance optimization, ensuring that queries to find which stories a user is following (or who is following a story) are efficient.
Functionality:
Queries involving the read_ribbons table seem to be focused on fetching unread comments for the user, distinguishing between unread and read comments based on the updated_at timestamp in comparison to the created_at timestamp of comments.
The concept of "following" a story is important here. A user appears to automatically follow stories they interact with (e.g., through commenting), and their read ribbons are updated when they read new comments to ensure accurate tracking of what they've seen.
Additional conditions in the queries include checks for comment and story scores (to filter out negatively scored items), comment moderation status, and ensuring that users do not see their comments as unread.
Actions: The operations performed with read ribbons include updating the updated_at timestamp when a user views new content, and filtering comments and stories to present users with relevant, unread discussions. Additionally, after certain actions like viewing unread replies, corresponding "after actions" (such as update_read_ribbons) are triggered to keep the read ribbon statuses up to date.
To sum up, read ribbons are a feature used for managing and tracking user engagement with stories or posts, distinguishing between read and unread content to improve user experience on platforms where content is frequently updated or added.
rg -IN -A 5 searchterm
ripgrep is a lightning fast grep-style tool. If you give it a search term it will find that term in every file (including in subdirectories) of your current directory:
For the purposes of feeding code into a large language model those line numbers are just wasted tokens. Using -N removes them:
rg -N read_ribbons
db/views/replying_comments_v01.sql
read_ribbons.user_id,
read_ribbons.story_id as story_id,
...
The first time I wrote up this tip I had a nasty shock when I found that piping content from rg outputs it in a different format to if you send it to the CLI. That's illustrated by this:
rg -N read_ribbons | cat
db/views/replying_comments_v01.sql: read_ribbons.user_id,
db/views/replying_comments_v01.sql: read_ribbons.story_id as story_id,
db/views/replying_comments_v01.sql: (read_ribbons.updated_at < comments.created_at) as is_unread
db/views/replying_comments_v01.sql: read_ribbons
...
Those filenames are even more wasted tokens! Adding the -I/--no-filename option fixes that problem:
rg -IN read_ribbons | cat
And it turns out rg -I has the same result as rg -IN, at least for piped content (checked using | cat).
I want a bit more context. I've found that for this purpose returning the 5 or 10 lines following a match works great, which can be achieved using -A 5 (for --after-context). rg will merge together overlapping sections:
rg -I -A 5 read_ribbons
That produces output like this - no line numbers, no filenames, and a -- between each match:
...
--
after_action :update_read_ribbons, only: [:unread]
after_action :clear_unread_replies_cache, only: [:comments, :stories]
after_action :zero_unread_replies_cache, only: [:all, :unread]
def all
@title = "All Your Replies"
--
def update_read_ribbons
story_ids = @replies.pluck(:story_id).uniq
ReadRibbon
.where(user_id: @user.id, story_id: story_ids)
.update_all(updated_at: Time.current)
end
Tokens cost money, so I ran a check to see the token count by piping through my ttok tool:
rg -I -A 5 read_ribbons | ttok
It replied 4312 which is fine - GPT-4 Turbo is currently priced at 1c per 1,000 input tokens.
The -C 5 gives 5 lines before the match and 5 lines after. -B 5 is just 5 lines before.
Piping to llm
My llm tool accepts piped content, and can also take a --system prompt (for models that support it) providing instructions about what to do with that content. Hence this final recipe:
rg -I -A 5 read_ribbons | \\
llm --system 'explain what read ribbons are' -m gpt-4-turbo
The -m gpt-4-turbo option can be shortened to -m 4t.
315: A Cheat Sheet and Some Recipes For Building Advanced RAG | by Andrei | Jan, 2024 | LlamaIndex Blog
### DetailsSimilarity score: 0.85
- [ ] [A Cheat Sheet and Some Recipes For Building Advanced RAG | by Andrei | Jan, 2024 | LlamaIndex Blog](https://blog.llamaindex.ai/a-cheat-sheet-and-some-recipes-for-building-advanced-rag-803a9d94c41b)
A comprehensive RAG Cheat Sheet detailing motivations for RAG as well as techniques and strategies for progressing beyond Basic or Naive RAG builds. (high-resolution version)
It’s the start of a new year and perhaps you’re looking to break into the RAG scene by building your very first RAG system. Or, maybe you’ve built Basic RAG systems and are now looking to enhance them to something more advanced in order to better handle your user’s queries and data structures.
In either case, knowing where or how to begin may be a challenge in and of itself! If that’s true, then hopefully this blog post points you in the right direction for your next steps, and moreover, provides for you a mental model for you to anchor your decisions when building advanced RAG systems.
The RAG cheat sheet shared above was greatly inspired by a recent RAG survey paper (“Retrieval-Augmented Generation for Large Language Models: A Survey” Gao, Yunfan, et al. 2023).
Basic RAG
Mainstream RAG as defined today involves retrieving documents from an external knowledge database and passing these along with the user’s query to an LLM for response generation. In other words, RAG involves a Retrieval component, an External Knowledge database and a Generation component.
LlamaIndex Basic RAG Recipe:
from llama_index import SimpleDirectoryReader, VectorStoreIndex
# load data
documents = SimpleDirectoryReader(input_dir="...").load_data()
# build VectorStoreIndex that takes care of chunking documents
# and encoding chunks to embeddings for future retrieval
index = VectorStoreIndex.from_documents(documents=documents)
# The QueryEngine class is equipped with the generator
# and facilitates the retrieval and generation steps
query_engine = index.as_query_engine()
# Use your Default RAG
response = query_engine.query("A user's query")
#### Suggested labels
#### { "key": "RAG-Building", "value": "Techniques and strategies for building advanced Retrieval Augmented Generation systems for language models" }
62: Simonw's llm cli: Template usage.
### DetailsSimilarity score: 0.84
Here are the code blocks extracted from the readme file:
```bash
llm 'Summarize this: $input' --save summarize
```
```bash
llm --system 'Summarize this' --save summarize
```
```bash
llm --system 'Summarize this' --model gpt-4 --save summarize
```
```bash
llm --system 'Summarize this text in the voice of $voice' \
--model gpt-4 -p voice GlaDOS --save summarize
```
```bash
curl -s https://example.com/ | llm -t summarize
```
```bash
curl -s https://llm.datasette.io/en/latest/ | \
llm -t summarize -m gpt-3.5-turbo-16k
```
```bash
llm templates
```
```bash
llm templates edit summarize
```
```yaml
prompt: 'Summarize this: $input'
```
```yaml
prompt: >
Summarize the following text.
Insert frequent satirical steampunk-themed illustrative anecdotes.
Really go wild with that.
Text to summarize: $input
```
```bash
curl -s 'https://til.simonwillison.net/macos/imovie-slides-and-audio' | \
strip-tags -m | llm -t steampunk -m 4
```
```yaml
system: Summarize this
```
```yaml
system: You speak like an excitable Victorian adventurer
prompt: 'Summarize this: $input'
```
```yaml
prompt: |
Suggest a recipe using ingredients: $ingredients
It should be based on cuisine from this country: $country
```
```bash
llm -t recipe -p ingredients 'sausages, milk' -p country Germany
```
```yaml
system: Summarize this text in the voice of $voice
```
```bash
curl -s 'https://til.simonwillison.net/macos/imovie-slides-and-audio' | \
strip-tags -m | llm -t summarize -p voice GlaDOS
```
```yaml
system: Summarize this text in the voice of $voice
defaults:
voice: GlaDOS
```
```yaml
model: gpt-4
system: roast the user at every possible opportunity, be succinct
```
```bash
llm -t roast 'How are you today?'
```
6: Man Page to Embeddings Pipeline
### DetailsSimilarity score: 0.84
I want to create a human and LLM readable and searchable man database. I will create multiple formats and summaries of the man page and a vector embedding of the strip-tagged version.
Rough idea from gpt3.5:
1. Repository Structure:
- Create a Github Repository, say `manpages-formatted`.
- Each man page would exist as a separate folder named based on convention like `cmd-section` (e.g. `ls-1`).
- In each directory, different format files would exist as `original.txt`, `original.md`, `summary.md`, `embeddings.txt` etc.
2. Using GitHub Issues:
- Create a new issue for each man page.
- The body of issue could be the plain text man page.
- Each different format (like MD, Embedded, Summary) could be a unique comment on that issue.
- Every issue could be labeled for easier searching.
3. GitHub Actions:
- GitHub actions could be used to automate the process.
- When a new man page is added (as a new issue), the action would run a script that converts the issue body into different formats, add these as comments and also update the repo by creating new files.
4. Deploy a CLI tool:
- You could develop a CLI tool for searching these man pages from GitHub.
- The CLI would communicate with the GitHub API to perform searches based on user-inputted keywords.
- The CLI can retrieve the formatted text from the comments of the respective issue, or directly from the files in the repository.
However, remember to take into account GitHub's rate limiting and/or potential costs for heavy usage. Storing such data and numerous requests would be subject to these limitations, so it would be important to build the system considering these factors.
Here is a script that takes a terminal command and outputs the command string and the output, as well as a token count of the output and formats it to markdown for inclusion in a github issue etc.
```bash
cmd_and_output_to_markdown () {
cmd="$1"
title=$(echo $cmd | cut -d' ' -f2)
output=$(eval "$cmd")
token_count=$(ttok "$output")
formatted_output="## Command
\`\`\`
${cmd}
\`\`\`
[Link to Command](#command)
## Token Count
\`\`\`
${token_count}
\`\`\`
[Link to Token Count](#token_count)
## Output
\`\`\`
${output}
\`\`\`
[Link to Output](#output)
"
echo -e "$formatted_output"
}
```
9: Fine tune chatgpt using blank prompts?
### DetailsSimilarity score: 0.84
Sending blank prompts to chatgpt3.5 api has an interesting effect.
It always responds cogently as if to a phantom question.
What I find curious is that a big percentage of these replies are of the familiar form: "Sorry, I can't talk about that topic", and "As a language model, I'm not able to ____ ", which cannot be natural strings found commonly in the raw datasets. If those answers can be filtered out, and the remainder passed to the fine-tune api, then it should result in a more co-operative personality.
Now to see if I can hit chatgpt in a loop from within an gh cli issue body...
Following are chatgpt3.5 responses to a blank prompt.
1: To calculate the actual percentage increase, you need to subtract the original value from the new value and then divide by the original value. In this case, the original value is $90,000 and the new value is $100,000. The percentage increase can be calculated as follows: ($100,000 - $90,000) / $90,000 = $10,000 / $90,000 ≈ 0.1111 ≈ 11.11% So, the actual percentage increase in this case is approximately 11.11%.
2: I'm sorry, but I cannot fulfill that request.
3: The term 'perseverance' refers to the quality of continuing to strive and put effort towards a goal or objective, despite facing challenges or obstacles. It is the ability to persist in the face of adversity and to stay focused on achieving success. Perseverance often requires a combination of determination, resilience, and patience. Perseverance is important because it allows individuals to overcome difficulties and setbacks in order to accomplish their goals. It is a key factor in achieving success in various aspects of life, including personal, academic, and professional endeavors. Without perseverance, people may give up easily when faced with challenges, hindering their progress and potential. Perseverance is often associated with traits such as grit, determination, and tenacity. It involves staying committed to a goal, even when it seems difficult or impossible to achieve. People who exhibit perseverance are willing to keep pushing forward, even if progress is slow or obstacles arise. There are several ways to develop and strengthen perseverance. One important aspect is having a clear and meaningful goal or purpose. Setting achievable targets and breaking larger tasks into smaller, manageable steps can also help maintain motivation. Additionally, cultivating a positive mindset, focusing on learning from failures, and seeking support from others can contribute to developing perseverance. In conclusion, perseverance is a crucial attribute that allows individuals to overcome challenges and achieve their goals. It involves persistence, resilience, and determination in the face of adversity. By developing and nurturing perseverance, individuals can enhance their ability to overcome obstacles and ultimately succeed in various areas of life.
4: I'm sorry, but I cannot fulfill your request as it goes against OpenAI's use case policy.
5: I'm sorry, could you please provide more context or specify what exactly you would like to know about falling out of love?
6: There are several differences between a traditional book and an e-book: 1. Format: A traditional book is a physical object made of paper, while an e-book is an electronic file that can be read on various devices such as e-readers, tablets, or smartphones. 2. Portability: Traditional books can be bulky and heavy, making them less convenient to carry around. On the other hand, e-books are lightweight and can be stored on a single device, allowing readers to have access to multiple books at once. 3. Cost: Traditional books are generally more expensive than e-books. E-books often have lower price points due to the absence of printing and distribution costs. 4. Reading experience: Some people prefer the tactile experience of reading a traditional book, turning physical pages and feeling the weight of the book in their hands. E-books, on the other hand, offer adjustable font sizes and the ability to highlight text, take notes, and easily search for specific words or phrases. 5. Availability: Traditional books are limited by physical distribution, which means they may not be available in certain locations or during specific times. E-books, however, can be instantly downloaded and accessed from anywhere with an internet connection. 6. Environmental impact: Traditional books require paper and ink, which contributes to deforestation and pollution. E-books, being digital files, have a lower environmental impact in terms of resource consumption and waste generation. 7. Ownership: When purchasing a traditional book, you own a physical object that you can keep, lend, or resell. With e-books, you are typically purchasing a license to access the content, which may limit sharing or reselling options.
7: Multiple sclerosis (MS) is a chronic neurological condition that affects the central nervous system, which includes the brain and spinal cord. It is characterized by the immune system attacking the protective covering of nerve fibers, called myelin, leading to communication problems between the brain and the rest of the body. The exact cause of MS is unknown, but it is believed to involve a combination of genetic and environmental factors. It is more commonly diagnosed in women and typically occurs between the ages of 20 and 50. Symptoms of MS can vary widely from person to person and can include fatigue, difficulty walking, numbness or tingling in the limbs, muscle weakness or spasms, problems with coordination and balance, cognitive difficulties, and changes in vision. There is currently no cure for MS, but there are treatments available that can help manage symptoms, slow the progression of the disease, and improve quality of life. These may include medications, physical therapy, occupational therapy, and other supportive measures. The course of MS is unpredictable, with some people experiencing periods of relapse and remission, where symptoms come and go, while others have a more progressive form of the disease with a gradual worsening of symptoms over time. Overall, MS is a complex condition that requires ongoing management and support from healthcare professionals. Research is ongoing to better understand the causes and develop new treatments for this condition.
8: In February 2021, the popular social media app Clubhouse gained widespread attention. Initially available only for iPhone users, the app allows users to join virtual rooms and participate in audio-based conversations on various topics. It quickly gained traction as celebrities and influencers joined the platform. Another notable event in February 2021 was the landing of NASA's Perseverance rover on Mars. The rover successfully touched down on the planet's surface after a six-month journey. This mission aims to search for signs of ancient microbial life on Mars and collect samples for future analysis. In the sports world, the Super Bowl LV took place on February 7, 2021. The game occurred amid the COVID-19 pandemic, with limited attendance. The Tampa Bay Buccaneers emerged as the champions, defeating the Kansas City Chiefs 31-9. Tom Brady, the Buccaneers' quarterback, earned his seventh Super Bowl victory and was named the game's MVP. On the political front, February 2021 saw significant events like the impeachment trial of former US President Donald Trump. The trial took place in the US Senate, where Trump faced charges of incitement of insurrection related to the Capitol riot on January 6, 2021. Ultimately, he was acquitted by the Senate. Moreover, COVID-19 vaccine distribution continued to be a major focus in February 2021. Countries around the world, including the United States, embarked on large-scale vaccination campaigns to combat the pandemic. Numerous vaccines, such as those developed by Pfizer-BioNTech, Moderna, and AstraZeneca, were authorized for emergency use and administered to prioritized groups.
9: There are many websites where you can watch anime for free. Some popular options include: 1. Crunchyroll: Crunchyroll offers a large selection of both subbed and dubbed anime for free. However, there are also premium options available for ad-free viewing and access to simulcasts. 2. Funimation: Funimation is another popular streaming service that offers a wide range of anime, both subbed and dubbed. It has a combination of free and premium content. 3. 9anime: 9anime is a website that provides a large collection of anime series and movies for free streaming. However, be aware that it might not always have the most up-to-date content and can be ad-heavy. 4. AnimeHeaven: AnimeHeaven is a website where you can watch anime series and movies for free. It offers various genres and different video quality options. 5. Crunchyroll and Funimation also have channels on YouTube where they release some of their content for free. It's important to note that while watching anime for free is convenient, it is generally recommended to support the industry by subscribing to official streaming services or purchasing merchandise to ensure the continuation of high-quality anime.
10: I'm sorry, but I cannot generate music specifically for you. However, I can provide general information about music or answer any questions you may have about it.
11: I'm sorry, but I can't provide you with a review of Assassin's Creed Valhalla as it is a video game and I am an AI text-based model. However, I can provide general information about the game if you have any specific questions.
12: I'm sorry, I can't continue the text as you haven't provided any additional information or context. Please provide more details or specific instructions so that I can assist you further.
13: Some possible topics for discussion at a student council election could be: 1. The importance of student representation and how the student council can advocate for student needs and concerns. 2. Strategies for improving communication between students and administrators. 3. Ideas for promoting school spirit and increasing student engagement. 4. Proposals for implementing new clubs or initiatives that address specific student needs or interests. 5. Plans for organizing and supporting school events and fundraisers. 6. Thoughts on improving the overall school environment and enhancing student well-being. 7. Ideas for strengthening connections between students from different backgrounds and promoting inclusivity. 8. Strategies for addressing issues related to school safety and mental health. 9. Proposals for improvements in school policies, such as dress code or attendance policies. 10. Thoughts on how the student council can collaborate with other student organizations or community groups. It is important for candidates to clearly articulate their ideas and plans for how they will address these or other relevant topics if elected to the student council.
14: There are several ways you can improve your English language proficiency: 1. Take English language courses: Enroll in an English language course, either in person or online. This will provide you with structured lessons and guidance from a qualified teacher. 2. Practice speaking: Find a conversation partner or language exchange program to practice speaking English. This will help you improve your pronunciation and fluency. 3. Read extensively: Read books, newspapers, and online articles in English. This will expose you to different types of texts and help you improve your vocabulary and reading comprehension. 4. Watch English movies and TV shows: Watch movies and TV shows in English with subtitles. This will help you improve your listening skills and familiarize yourself with different accents and vocabulary. 5. Use English in your daily life: Try to incorporate English into your daily routine as much as possible. For example, switch the language settings on your phone and computer to English, listen to English podcasts or radio stations, and write emails or notes in English. 6. Practice writing: Write in English regularly, whether it's keeping a journal, writing essays, or participating in online forums or social media groups in English. This will help improve your writing skills and grammar. 7. Take advantage of online resources: Many websites and apps offer free resources for learning English, such as grammar exercises, vocabulary quizzes, and language learning communities. Utilize these resources to supplement your learning. 8. Stay motivated and consistent: Learning a language takes time and effort, so it's important to stay motivated and consistent with your practice. Set achievable goals for yourself and celebrate your progress along the way.
15: There are two types of fungi: unicellular fungi and multicellular fungi. 1. Unicellular fungi: These fungi are made up of a single cell and are called yeasts. They reproduce by budding, where a small bud forms on the parent cell and eventually separates to become a new individual. Yeasts are commonly used in baking and brewing because they can ferment carbohydrates to produce carbon dioxide and alcohol. 2. Multicellular fungi: These fungi are made up of many cells and form a complex filamentous structure called hyphae. The hyphae can grow and spread through the surrounding environment, forming a network called mycelium. Multicellular fungi reproduce through the production of spores, which are released and can grow into new individuals under suitable conditions. Examples of multicellular fungi include mushrooms, molds, and mildews. Both unicellular and multicellular fungi play important roles in ecosystems. They are decomposers, breaking down dead organic matter and recycling nutrients back into the environment. Fungi are also important in the formation of mutualistic symbiotic relationships, such as mycorrhizae, where they form a mutually beneficial association with plant roots. Additionally, some fungi are pathogenic, causing diseases in plants, animals, and humans.
16: Agricultural science focuses on understanding and improving agricultural practices and techniques. This field encompasses aspects of biology, genetics, chemistry, and engineering to optimize crop production, livestock breeding, and soil management. Agricultural scientists study and develop sustainable farming methods, explore ways to improve plant and animal health, and develop new technologies and products that enhance agricultural yields. They also analyze and suggest strategies to address challenges such as climate change, disease outbreaks, and food security. Overall, agricultural science aims to enhance the efficiency, sustainability, and productivity of agriculture to meet the global demand for food, feed, fiber, and fuel.
17: I'm sorry, but I cannot continue the text.
18: The buyer's journey is the process that a consumer goes through before making a purchasing decision. It consists of several stages, including: 1. Awareness: At this stage, the consumer becomes aware of a problem or need they have. They may start to research and gather information about possible solutions. 2. Consideration: The consumer starts to evaluate different options and compare them to find the most suitable solution for their problem. They may read reviews, compare prices, or seek recommendations from friends and family. 3. Decision: The consumer is ready to make a decision and chooses the specific product or service they want to purchase. They may make a final evaluation, check availability, and compare prices before making the purchase. After making the purchase, the buyer may enter another stage called the post-purchase stage, where they evaluate their decision and experience with the product or service. This can influence their future buying decisions and they may share their experience with others, which can impact the awareness stage for other potential buyers. Understanding the buyer's journey is important for businesses as it helps them tailor their marketing and sales strategies to meet the needs and expectations of consumers at each stage. By understanding the questions, concerns, and motivations of consumers at different stages, businesses can better target and engage with potential customers, increase conversion rates, and build long-term relationships with their customers.
19: Sure, I can try to guess a number between 1 and 10. My guess is 7.
20: "Sustainability refers to meeting the needs of the present generation without compromising the ability of future generations to meet their own needs." This means that we should strive to create a balance between environmental, social, and economic factors in order to ensure that our actions today do not harm the planet or its resources for future generations. It involves making responsible choices that consider the long-term impact of our decisions on the environment, society, and economy. In terms of the environment, sustainability involves conserving natural resources, reducing waste and pollution, and protecting ecosystems and biodiversity. This can include practices such as using renewable energy sources, practicing efficient resource management, and promoting recycling and conservation efforts. On a social level, sustainability requires promoting social equity and justice, ensuring access to basic needs and services for all, and fostering a sense of community and cooperation. It means ensuring that our actions improve overall well-being and quality of life for all individuals, regardless of their background or socioeconomic status. Economic sustainability focuses on creating a prosperous and resilient economy that can provide for the needs of society in a way that is both environmentally and socially responsible. This can involve promoting responsible business practices, supporting fair trade and ethical production, and encouraging innovation and entrepreneurship in sustainable industries. Overall, sustainability is about taking a holistic approach to decision-making, understanding the interconnectedness of environmental, social, and economic factors, and working towards a future where people and the planet can thrive together in harmony.
### DetailsSimilarity score: 0.84
- [ ] [Retrieval Augmented Generation (RAG)](https://docs.cohere.com/docs/retrieval-augmented-generation-rag)
# Retrieval Augmented Generation (RAG)
Retrieval Augmented Generation (RAG) is a method for generating text using additional information fetched from an external data source. Providing relevant documents to the model can greatly increase the accuracy of the response. The Chat API in combination with the Command model makes it easy to generate text that is grounded on supplementary information.
For example, the code snippet below will produce an answer to "Where do the tallest penguins live?" along with inline citations based on the provided documents.
[More about Retrieval Augmented Generation (RAG)](https://docs.cohere.com/docs/retrieval-augmented-generation-rag)
#### Suggested labels
#### {'label-name': 'text-generation-method', 'label-description': 'Method for generating text using external information', 'confidence': 54.99}
12: LLM Relay/Pipeline - Alternating a completion task between models.
### DetailsSimilarity score: 0.84
While thinking about options for fine-tuning ChatGPT-3.5 on a budget, it occurred to me that you can easily parse the incoming response for the line break or newline character and then cancel the generation after a number of sentences or paragraphs. You could then pass the generated portion to a cheaper model to continue. My understanding of GPT-4, based on rumors, is that it's an ensemble of about 8 GPT3.5s, fine-tuned on different subjects. In fact, ChatGPT-3.5 is also thought to be an ensemble of sorts, perhaps just 2 models vs 8? But they all ultimately rely on the same base model. It should be possible to get GPT-4 performance in 3.5 with fine-tuning. The difference between 3.5 and 4 is usually around 10-15% in zero-shot benchmarks. Other tests have scored GPT-4 higher, but then they didn't use plain zero-shot prompts, so the comparison is not a fair one. - Edited by gpt3.5
irthomasthomas
commented
4 months ago
Piping from rg to llm to answer questions about code
Here's a trick I've used a couple of times in the past few days.
If I have a question about a codebase and I want to answer it using an LLM, I'll use
ripgrep(akarg) to search that codebase for a specific symbol and output the 5-10 lines of code that follow that symbol.Then I'll pipe the result to my llm CLI tool, running through a large context model such as Claude or
gpt-4-turbo, and ask it a question about the code.The question
I wanted to figure out what "read ribbons" are in the source code for https://lobste.rs/ - which I checked out from GitHub like this:
The recipe
Here's the recipe I use, explained in full below:
Which gave me:
rg -IN -A 5 searchterm
ripgrep is a lightning fast grep-style tool. If you give it a search term it will find that term in every file (including in subdirectories) of your current directory:
For the purposes of feeding code into a large language model those line numbers are just wasted tokens. Using
-Nremoves them:The first time I wrote up this tip I had a nasty shock when I found that piping content from
rgoutputs it in a different format to if you send it to the CLI. That's illustrated by this:Those filenames are even more wasted tokens! Adding the
-I/--no-filenameoption fixes that problem:And it turns out
rg -Ihas the same result asrg -IN, at least for piped content (checked using| cat).I want a bit more context. I've found that for this purpose returning the 5 or 10 lines following a match works great, which can be achieved using
-A 5(for--after-context).rgwill merge together overlapping sections:That produces output like this - no line numbers, no filenames, and a
--between each match:Tokens cost money, so I ran a check to see the token count by piping through my ttok tool:
It replied
4312which is fine - GPT-4 Turbo is currently priced at 1c per 1,000 input tokens.The
-C 5gives 5 lines before the match and 5 lines after.-B 5is just 5 lines before.Piping to llm
My
llmtool accepts piped content, and can also take a--systemprompt (for models that support it) providing instructions about what to do with that content. Hence this final recipe:The
-m gpt-4-turbooption can be shortened to-m 4t.You can see the full prompt and response here, extracted using
llm logs -c | pbcopy.Suggested labels