search

mlc-ai / mlc-llm

Universal LLM Deployment Engine with ML Compilation
https://llm.mlc.ai/
Apache License 2.0
19.13k stars 1.57k forks source link

[Question] Why does the model require a lot of video memory? #2844

Closed BlindDeveloper closed 2 months ago

BlindDeveloper commented 2 months ago

Why Phi 3.5 mini 3.8b require mor vram then Mistral 7b v0.3 for android?

TriDefender commented 2 months ago

It depends on the model size, especially the number of parameters and layers of the neural network

BlindDeveloper commented 2 months ago

@TriDefender thank you for explanation!