How can I deploy a single-card MLC-LLM model? I want the model inference to run only on one card, not distributed.

mlc-ai / mlc-llm

Universal LLM Deployment Engine with ML Compilation

https://llm.mlc.ai/

Apache License 2.0

18.63k stars 1.51k forks source link

Closed 137591 closed 4 months ago

137591 commented 4 months ago

tqchen commented 4 months ago

By default mlc llm deployment should be single card