https://eugeneyan.com/writing/llm-patterns/

[utterances-bot]

Patterns for Building LLM-based Systems & Products

Evals, RAG, fine-tuning, caching, guardrails, defensive UX, and collecting user feedback.

https://eugeneyan.com/writing/llm-patterns/

[JungeAlexander]

Thanks for the article, Eugene - a long but excellent read!

[eugeneyan]

Thank you for the kind words 🙇 And wow you're fast! I hadn't even shared it widely yet.

[andrewyu0]

reading it piecemeal and printed it out - amazing thanks for putting this labor of love together! 🙏

[adidoit]

Wonderful post and love the links to the papers. Thank you!! What do you think about Gorilla LLM and Toolformer? Still early days but wonder if there's a future pattern in there

[kirankumarbathula]

great article, thanks for sharing. Lot of things to dig in

[mameshini]

Thank you for creating such a comprehensive overview of LLM patterns.

[AhmedTremo]

Great Read, thanks for sharing Eugene

[KDmytro]

Lots of wisdom packed in one place. Thank you for putting it all together.

[davegoldblatt]

I summarized this with Claude 2. How did it do?

Here is a bullet point summary with chapter headings for the article:

Introduction

• The article provides practical patterns for integrating large language models (LLMs) into systems and products.
• It builds on academic research, industry resources, and practitioner know-how to distill key ideas and practices.

Key Patterns

Evals: To Measure Performance

• Evals are benchmarks and metrics to assess model performance on tasks.
• They enable tracking improvements or regressions when updating systems.
• Metrics can be context-dependent or context-free. Popular metrics include BLEU, ROUGE, and BERTScore.

Retrieval-Augmented Generation (RAG): To Add Knowledge

• RAG retrieves relevant data to provide context and grounding for the LLM.
• It helps reduce hallucination and provides access to recent data.
• Popular techniques include dense passage retrieval (DPR) and fusion-in-decoder (FiD).

Fine-tuning: To Get Better at Specific Tasks

• Fine-tuning further trains a pre-trained LLM on a downstream task.
• It can improve performance and provides more control than using a 3rd party LLM.
• Methods include instruction tuning, single-task tuning, and techniques like LoRA.

Caching: To Reduce Latency and Cost

• Caching stores previous LLM responses to serve faster for repeated requests.
• It reduces latency and cost but needs to be applied carefully.
• Cache based on IDs or constrained inputs for reliability.

Guardrails: To Ensure Output Quality

• Guardrails validate LLM outputs for correctness, safety, and coherence.
• They provide quality control and help ensure outputs are reliable.
• Methods include guidance, syntactic checks, semantic checks, and safety checks.

Defensive UX: To Handle Errors Gracefully

• Defensive UX anticipates and gracefully handles LLM inaccuracies and errors.
• It helps users understand system capabilities and increases trust.
• Patterns include setting expectations, efficient dismissal, and attribution.

Collect User Feedback: To Improve Models

• User feedback provides data to evaluate and fine-tune models.
• Feedback can be explicit (ratings) or implicit (usage metrics).
• UX should make providing feedback easy and frictionless.

Conclusion

• The article provides a useful reference for patterns to consider when building LLM systems and products.
• Additional insights are shared regarding how to apply each pattern.
• There are still many open questions as the field is rapidly evolving.

[Tylersuard]

Thank you for putting all this knowledge in one place, it is really helpful. I understood literally not a word of your summary on Retrieval-Enhanced Transformers. It might be an idea to write it a little more simply.

[charlestsang]

Thanks for sharing the knowlegde in such a good way. A lot of inputs gain from your blog. Well done.

[caioabe]

"be waifu"

[eusden]

Thanks for putting this together Eugene. Thorough and comprehensible explanation of the SOTA. The domain and applications are expansive, so this clear and practical summary will be a useful reference for many including me!

[mon95]

Thank you so much for this article! This was a great read :)

[pedro3087]

thanks for the great information. Do you have any samples or a blog post where we can find more information about how to automate evals and Guardrails? thanks again.

[eugeneyan]

Here's some thinking on guardrails, especially how to bootstrap them with out-of-domain data.

[knownbymanoj]

Thank you for the article had an amazing time reading it and got so much clarity on the concepts.