search

mlc-ai / mlc-llm

Universal LLM Deployment Engine with ML Compilation
https://llm.mlc.ai/
Apache License 2.0
19.26k stars 1.58k forks source link

[Feature Request] Add vision model flag to model record #3031

Open Neet-Nestor opened 1 week ago

Neet-Nestor commented 1 week ago

🚀 Feature

Add a field to the ModelRecord to indicate whether a model is a vision model like Phi-vision models.

 data class ModelRecord(
    @SerializedName("model_url") val modelUrl: String,
    @SerializedName("model_id") val modelId: String,
    @SerializedName("estimated_vram_bytes") val estimatedVramBytes: Long?,
    @SerializedName("model_lib") val modelLib: String
)

Motivation

Users and apps need to have additional UI and functionality for vision models for picking and loading images. Currently there is no way to retrieve the information from the config files as the information is missing.

Alternatives

Otherwise, the users could hardcode UI for specific models.

Additional context

I'm trying to add UI for iOS app of the MLC-LLM for Phi-vision models.

YangWang92 commented 2 days ago

+1