meta-llama / llama-recipes

Scripts for fine-tuning Meta Llama3 with composable FSDP & PEFT methods to cover single/multi-node GPUs. Supports default & custom datasets for applications such as summarization and Q&A. Supporting a number of candid inference solutions such as HF TGI, VLLM for local or cloud deployment. Demo apps to showcase Meta Llama3 for WhatsApp & Messenger.

9.95k stars 1.4k forks source link

[MLC-LLM] Introducing Llama 3 running locally on Android using MLC-LLM #506

Closed tmoreau89 closed 2 weeks ago

tmoreau89 commented 2 weeks ago

What does this PR do?

This PR introduces a guide on how to run Llama 3 8B instruct on an Android Phone using MLC LLM!

Feature/Issue validation/testing

This tutorial was tested with the following setup:

MacBook Pro 16 inch from 2021 with Apple M1 Max and 32GB of RAM running Sonoma 14.3.1
OnePlus 12 Android Smartphone with a Snapdragon 8Gen3 SoC and 12GB or RAM, running OxygenOS 14.0

Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[X] Did you read the contributor guideline, Pull Request section?
[ ] Was this discussed/approved via a Github issue? Please add a link to it if that's the case.
[X] Did you make sure to update the documentation with your changes?
[ ] Did you write any new necessary tests?

Thanks for contributing 🎉!

tmoreau89 commented 2 weeks ago

CC @HamidShojanazeri - the long awaited MLC-LLM tutorial!

tmoreau89 commented 2 weeks ago

@HamidShojanazeri we're green on spellcheck here, but the linter has issues with the following two URLs:

The error saying Status: 0 Error: Exceeded maxRedirects. Probably stuck in a redirect loop.

Unclear why but is there a way to get the linter to ignore these URLs?

HamidShojanazeri commented 2 weeks ago

sorry @tmoreau89 its the linter bug, can you pls add  , surrounding the link, https://github.com/meta-llama/llama-recipes/pull/291/files

tmoreau89 commented 2 weeks ago

@HamidShojanazeri I think we're good to go on this one! Thanks for the tip!

HamidShojanazeri commented 2 weeks ago

@tmoreau89 seems its still failing on "./mlc-package-config.json → Status: 400 [Error: ENOENT: no such file or directory, access '/github/workspace/recipes/inference/mobile_inference/android_inference/mlc-package-config.json']", would appreciate the fix to make the CI green thanks!

tmoreau89 commented 2 weeks ago

@HamidShojanazeri Morning! The checks are all green now (this time for real 😆 )