eckartal
commented
7 hours ago We've changed the topic.
Title
Running TensorRT-LLM on 10,000 RTX Machines: What We've Learned
Description
In this talk, we'll detail our experience with TensorRT-LLM through the lens of our desktop application, which has enabled over 10,000 RTX machine users to run AI models locally. As specialists in desktop inference, we've experienced significant edge cases on implementing AI at scale using consumer-grade hardware. We'll present our findings on the technical challenges, performance metrics, and key insights gained from this large-scale deployment, offering a data-driven perspective on AI implementation for individual users and teams.
Key Takeaway
- Overview of our desktop application leveraging TensorRT-LLM for local AI model execution
- Performance analysis on consumer-grade RTX GPUs
- Quantitative data and benchmarks from our 10,000-machine deployment
- Evidence-based best practices for the hobbyist and consumer market
- Insights and best practices for AI implementation on from consumer to datacenter-grade hardware at scale
Relavence to NVIDIA
This presentation aligns with NVIDIA's goal of expanding TensorRT-LLM adoption. By providing concrete data on its implementation at scale in the consumer market, we demonstrate the tool's applicability beyond enterprise use cases. Our technical insights could inform NVIDIA's development roadmap and potentially accelerate adoption among individual developers and teams.
Speaker Bio
Daniel Ong is the CEO of Homebrew Research, an AI R&D studio working on local AI, small language models, and multi-modality. He started off his career as an engineer at Palantir and Pivotal Labs. He studied Computer Science at Stanford '12. Previously, Daniel was the CTO of Care, Dana Cita (YC '18).
Overall
GTC 2025 will be held on March 17- 20, 2025 in person in San Jose. The NVIDIA team wants us to share our works there.
At that time, hope we integrate Jade to Jan which is powered by Cortex enable Jan flexible enough to run any hardware. It's like puzzling and gathering puzzle pieces together to solve personal AI assistant problems following LLMs, hardware, UI, a small OS, assistant, and voice assistant.
Goals
[ ] Find a topic to share [ ] Prepare a submission fits NVIDIA's reqs - including title, description and key findings [ ] Submit!
Ideas
We should positionate ourselves as a thought leader in AI dev by sharing our findings providing timelines and full-cycle starting from solved problems to future challenges.
So this talk should be about the recap of our full-cycle development journey.
Flow
I think we should have two sections focused on problems solved and tomorrow's problems to solve, explaining what we've done to build a full-stack product-related company starting from LLMs problems to hardware, to voice assistants showcasing Jan as a UI, Cortex as an engine, Jade as a voice assistant and finally combining all of them to build real-life Jarvis that can communicate well, understand human speech and talk back!
Section 1: Building the Foundation
Section 2: Tomorrow's Problems
Key Milestones
Talk Submission
Title
Building "Jarvis" through puzzle-like problem solving
Description
This talk covers the development of AI assistants capable of solving complex problems like puzzles. We'll discuss the technologies that make these assistants possible, including advanced language models, specialized hardware, and intuitive interfaces. The presentation will highlight current capabilities and future challenges in AI assistant development, with a focus on NVIDIA's contributions to the field.
This talk covers the full lifecycle of AI assistant development, structured in two parts. First, we cover solved problems: LLM advancements, hardware optimization, software integration, UI, and voice recognition. Second, we address future challenges: enhancing AI problem-solving, achieving device integration, and exploring physical embodiment. We'll share our learnings on building real-life "Jarvis"ish personal assistant that runs 100% offline on from personal laptops to datacenter-level computers, to demonstrate how these elements combine to create AI assistants capable of natural communication and speech understanding.
Key Takeaways
Resources