manisnesan
commented
2 weeks ago https://www.oreilly.com/radar/what-we-learned-from-a-year-of-building-with-llms-part-iii-strategy/
Takeaways When you fine tune:
- after collecting lots of examples ie failure modes convincing other approaches are not sufficient
when to self host
- self-hosting may be the only way to use models without sending confidential/private data out of your network
- to reduce costs by fine tuning
what else to focus on
- Evaluation chassis: To reliably measure performance on your task across models
- Guardrails: To prevent undesired outputs no matter the model
- Caching: To reduce latency and cost by avoiding the model altogether
- Data flywheel: To power the iterative improvement of everything above
Employ some “strategic procrastination” here, build what you absolutely need, and await the obvious expansions to capabilities from providers.
What we learned from a year of building with LLMs