LLMs in Production - Conference

[manisnesan]

Related

• https://huyenchip.com/2023/04/11/llm-engineering.html

Tweet thread - LLMs in Production hosted by MLOps Community

Considerations

• High Affordability (eg: code suggestions, email suggestions) vs Low Affordability (eg: Medical diagnosis, financial decisions) Use Cases
• Guardrails to ensure safety, accuracy, reliability

Iteration Cycle

• Getting Started with ML (Data Warehouse -> Hire/Train ML expert -> Build Model -> Build Inference Endpoint -> MVP)
• Getting Started with LLMs (LLM API -> Prompt Design -> Vector DB / Data Integration -> MVP )

[manisnesan]

It's not a knowledge answering engine but a reasoning engine.

Limitations

• Prompt Hacks that skips guardrails
• Hallucinations
• Using LLMs to write malware or attacks
• Using LLMs to reveal internal info

Point of Failure Areas

• Prompt Point Failure (b/w user & model) -> user side vs model side | Prevent LLMs model not provide info about undesirable things (eg: how to steal a car) | Prompt Padding with Heuristics/Rules
• AI Middleware - chaining together commands on API & let LLM perform the operations | wrong order of operations -
• LLM Response -> Grounding -> connecting LLMs to external DBs or search & inject context | Self Reflecting LLMs | Model "Patching" | Continual Learning

[manisnesan]

Reasoning Machines

• LLMs should be used as reasoning engines rather than fact engines

Problem: LLMs can generate responses only based on the training data.

Solution:

• Prompt Engineering can be used to supply factual info.

Limitation: Scaling

Option 1 using external data: Read the webpage and ask the question about the document. This helps the model to reason about the document.

[manisnesan]

How LlamaIndex can connect your LLMs with your External data

Slides | Presenter

• LLMs for knowledge generation & reasoning. Pretrained on large amounts of publicly available data
• Use Cases: QA, Text Generation, Planning, Summarization
• Context: How to augment LLMs with our private data?
  • Input: Raw Files, API
• Paradigms for inserting knowledge
  • Fine tuning with task specific domain data
  • In-context learning: putting context into prompt ie Combine a pre-trained LLM with a retrieval model to retrieve context. | Here's the context $CONTEXT. Given the context, answer the following question $question
  • Key Challenges
  • deal with logn context
  • retrieve the right context
  • LlamaIndex
  • Use Case: Semantic Search
  • Use Case: Summarization
  • Use Case: Text-to-SQL (Structured Data)
  • Synthesis over Heterogeneous Data
  • Multi-step Query : break a complex into a simple queries over the data source

[manisnesan]

Role of ML Engineers in the time of LLMs

Lessons Learned

• Impressive first results
• Convincing hallucinations
• Inconsistent Outputs

Commoditized ML

• Projects based on public data
• Project without specific env req (eg: on device)
• Projects without security requirements

In-house ML

• Projects based on proprietary data
• Projects with specific security requirements
• Low Latency requirements
• Importance of intellectual property

Objectives of ML (until now)

• Unbiased preds
• Data & training transparency (e.g model cards)
• Closed feedback loops to improve model perf
• User privacy
• On device inferences

What does it mean for ML Engineers?

• Role evolves from purely developing model to "moderators between stakeholders"
• Advisors around risks & benefits

Conclusions

• Prompt Design (is it really engineering??) will be part of ML but it won't replace ML
• Lots of MLOps Challenges with LLMs
• Integration needs a good ML understanding (Eg: regarding bias, safety)

Focus on

• Projects with proprietary data
• Projects with "subjective" ML eg. similarity ML (recsys could be an example)
• Avoid plain vanilla projects (eg: sentiment classification can be easily done through API)
• ML with specific requirements (eg: user privacy, security, low latency)
• Be the moderator & advisor b/w or to stakeholders.

[manisnesan]

Obstacles to successful LLM deployments

Presenter: Tanmay Chopra

Infra Thorns

• Latency Expectation Readjustment (LLM o/p generation can take longer than the user normally expects)
  • Sol: Make model faster (model distillation & pruning) or seem faster (using HCI such as animations, streaming o/p)
• Cost & decision complexities (Buy ie using API vs Build ie finetune)
  • Sol : Low upfront investment for Buy to validate use case. Over time fine tune models inhourse
• Foundational Model API Reliability (Serving Infra for foundation model) is still being build up and downtime can significantly lose trust)
  • Sol: multi-provider fallback | Fail gracefully
• Immature Evaluation Infra
  • Fail fast with failsafes & integrate use feedback loops | Top-line metric evaluation (link LLM o/p to task specfici metric)

Output linked Thorns

• O/P Format Variability
  • Sol (Few-shot prompting - > provide output eg in prompt | Output format validation | Fail gracefull as last resort
• Lack of reproducibility
  • sol: Set temp to 0 (ensure same output from the model)
• Prompt Hijacking
• Trust & Safety

Start Strong

• Proj positioning: Focus on deploying to non-critical workflows (Add value & don't become dependency)
• Find relatively higher latency use cases
• Plan to build while you buy
• Don't underestimate HCI component (Faster response, fail gracefully, enable large scale feedback loop)

Tweet

[manisnesan]

Emerging Patterns for LLMs in Prod

Challenges in Prod

• Reliability (hallucination) | Cost | Latency | Safety

Start Simple

Prompting -> Few Shot Prompting -> Retrieval + Prompting (Langchain, LlamaIndex) -> Iterative Refinement (CoT, Decomposition)

Reliability: Add Structure

• Query. You must comply with the following Typescript schema Accuracy: Self-refinement
• Model evaluates its own output.
• Review your own output and score it. Improve the prompt based on the score. Accuracy: Contextual Compression
• Context window limts the amount of external data (it has to be a fresh data)
• Compress and Summarize with examples that are needed to answer the question Latency: Semantic Caching
• uses vector DB for caching
• Cache hits are evaluated by semantic similarity or by LLM
• When: Expensive completions (many trips to LLM) | Selection Problem (tools, caretogires)
• See GPTCache in github Safety: PII & Prompt Injection Detection
• Use LLMs to detect leakage of information (eg: PII), Prompt injections

[manisnesan]

LangChain - Enabling LLMs to use tools

Why tooling is important

• Retrieve relevant context (eg: current events, proprietary data, complex Data Structure)
• Allow the LLM to interact with the outside world.

Different types of tools

• Search Engines | Calculators | Retrieval Systems | Python REPLs | Arbitrary func | APIs How to get LLms to use tools
• When to use them | How to use them | What they return

Challenges:

• Getting them to use tools in right scenario: Instructions/System message | tool description | repeat instructions at the end | Tool retrieval
• Getting them to not always use tools: Instructions/System message | repeat instructions at the end | "Human Clarification/input" tool
• Parsing LLM output to tool invocation: json | output parsers | fix output parsers

Spotlight on OpenAPI tools

• Prompting: Typescript/JSON to communicate the params
• Composability: Wrap each endpoint in its own chain & have the agent call that chain

[manisnesan]

Vector DBs & LLMs

https://mlops.community/vector-similarity-search-from-basics-to-production/

[manisnesan]

How to use LLMs effectively

Tweet thread - Mental model of going from simple to complex