[Tracking] SLM Rollout - Githubissues

CharlieFRuan commented 10 months ago

Overview

Recently, the mlc-llm team has been working on migrating to a new model compilation workflow, which we refer to as SLM. SLM is the new approach to bring modularized python first compilation to MLC, allowing users and developers to support new models and features more easily.

Concretely for instance, here is the old file to support mistral: mistral.py, and here is the new file: mistral_model.py.

As SLM's development is wrapping up, this issue tracks the steps for rolling out SLM, including documentation, tutorials, prebuilt models/weights release, and more.

Action Items

S1: Compile-time Documentation Updates

https://github.com/mlc-ai/mlc-llm/pull/1421

[x] Populate Build From Source, such that REST, CLI/C++, Python all point to that
[x] Update Compile Models page (main update here)
[x] Update Configure Quantization (still WIP, but give some pointers)
[x] Deprecate Python API for Model Compilation
[x] Update Distributed Compiled Models accordingly
[x] Make New Model Architectures page a WIP, but provide some initial pointers
[x] Update MLC Chat Config documentation

S1.5: Split compile model w/ MLC into convert model and compile model library

https://github.com/mlc-ai/mlc-llm/pull/1464

This iterates S1, with the goal of making to two cmponents convert weight and model library clearer

[x] Fold distribute compiled models into convert weight
[x] Re-purpose compile model w/ MLC into compile model library

S2: Tutorials

[x] Deprecate previous build Colab tutorials
[x] Make a Colab tutorial for adding a new model under SLIM, including how to debug layer-by-layer
- https://github.com/mlc-ai/notebooks/commit/0745d3c5ec66226b52c826a730fa668c50d89b9e

S3: Update prebuilts

[x] Upload new prebuilt models (with good defaults)
- https://github.com/mlc-ai/binary-mlc-llm-libs/pull/58
- https://github.com/mlc-ai/binary-mlc-llm-libs/pull/59
[x] Upload new prebuilt weights (with good defaults)
- Test usage of delivery.py and update name to f{model_name}-{quant}-MLC
[x] Update Model Prebuilt page correspondingly
- https://github.com/mlc-ai/mlc-llm/pull/1490

S4: Deployment: WebLLM, Android, iOS

[x] F1: WebLLM
- https://github.com/mlc-ai/mlc-llm/pull/1476
- https://github.com/mlc-ai/mlc-llm/pull/1479
[x] F2: Android
- https://github.com/mlc-ai/mlc-llm/pull/1494
[x] F3: iOS
- https://github.com/mlc-ai/mlc-llm/pull/1601

For WebLLM, Android, iOS, reflect new changes in flow after SLM, and:

Add an end-to-end tutorial on "Bring Your Own Model Variant" and "Bring Your Own Model Library"
Mention when to only convert weight -- point to that page; and when to build your own model library -- point to that page. In former, mention how to (and when to) reuse existing model library
Describe how to take /path/to/model, /path/to/lib, setup config so app connects up (e.g. in WebLLM case, how to update app config)

S5 Runtime Documentation Updates (after S3)

https://github.com/mlc-ai/mlc-llm/pull/1491

[x] Update other runtime pages (REST, CLI/C++, Python); ensure examples with prebuilt are all runnable; point to convert weight/compile model library pages when appropriate, explain when to do that
[x] Deprecate the usage of ./dist format to auto-locate the packages; in the example, always ask users to specify both model and model-lib-path. (this affects runtime documentation for CLI/C++ and Python)
[x] Update runtime portion of Prebuilt Models after S4 done (now commented out)
- https://github.com/mlc-ai/mlc-llm/pull/1601

S6 Update Project Overview

https://github.com/mlc-ai/mlc-llm/pull/1491

[x] Emphasize model (weights and mlc-chat-config) and model library
[x] Remove mention of local_id and model_lib field
[x] Runtime flow update: emphasize model and model library

Future items

Make prebuilt and fill up https://llm.mlc.ai/docs/prebuilt_models.html; ideally reach parity with old workflow
Quantization in SLM
- Documentation about how quantization works in SLM
- Fully support AWQ -- test whehter AWQ work for all models (not just llama), release prebuilts with AWQ
- Add other quantization (QuIP-sharp)
Consider using dataclass to represent arguments, so that documentation can be automatically populated; e.g. see https://github.com/mlc-ai/mlc-llm/blob/2fd1bf510c2c5ec54bb8323b5166851cb9007885/mlc_llm/core.py#L44
Update package dependencies (users need to install packages like tqdm, requests, attrs, torch, etc. manually one-by-one).
During runtime, still include auto-lookup, but make it a fallback (perhaps ./dist/libs/{model}-{quant}-{platform}.{suffix}); may need to somehow include a model-lib field like before

tqchen commented 10 months ago

To reduce the amount of extra concepts needed, let us consider moving the model_lib mapping directly to AppConfig in WebLLM, so we no longer need this field.

CharlieFRuan commented 10 months ago

Makes sense, I will make corresponding changes in web-llm.

tqchen commented 10 months ago

note on convention, let us use model-name-quant-MLC as the new convention of converted models

gfvvz commented 2 months ago

Hi, just curious about the SLM, and what is SLM short for?

CharlieFRuan commented 2 months ago

It was mainly an internal name for a more pythonic frontend to define LLMs and a more user-friendly flow for converting models via MLC