search

QQQQQQY / MLCChat

0 stars 0 forks source link

如何加载新模型 #1

Closed huangliang113333 closed 2 weeks ago

huangliang113333 commented 2 weeks ago

hf-mirror 下载你需要配置的其他模型, 注意: 需要是转换过权重的以 -MLC 结尾的模型。 修改 mlc-app-config.json

我从hf-mirror] 下载模型落后，mlc-app-config.json 如何修改 ，谢谢。我试了试都失败了 ，感谢指教

QQQQQQY commented 2 weeks ago

hf-mirror 下载你需要配置的其他模型, 注意: 需要是转换过权重的以 -MLC 结尾的模型。 修改 mlc-app-config.json

我从hf-mirror] 下载模型落后，mlc-app-config.json 如何修改 ，谢谢。我试了试都失败了 ，感谢指教

你下载的模型是哪个? 你是如何配置的 mlc-app-config.json导致报错的呢以及报错的信息是什么呢?

huangliang113333 commented 2 weeks ago

感谢回复。和您的指点： 在 https://hf-mirror.com/mlc-ai/Qwen2.5-1.5B-Instruct-q4f16_1-MLC/tree/main 下载的 Qwen2.5-1.5B-Instruct-q4f16_1-MLC

mlc-app-config.json { "model_list": [ { "model_id": "gemma-2-2b-it-q4f16_1-MLC", "model_lib": "gemma2_q4f16_1_5cc7dbd3ae3d1040984d9720b2d7b7d4", "model_url": "https://huggingface.co/mlc-ai/gemma-2-2b-it-q4f16_1-MLC", "estimated_vram_bytes": 3000000000 }, { "model_id": "Qwen2.5-1.5B-Instruct-q4f16_1-MLC", "model_lib": "q4f16_1", "model_url": "https://huggingface.co/mlc-ai/Qwen2.5-1.5B-Instruct-q4f16_1-MLC", "estimated_vram_bytes": 3980990464 } ] }

文件放到了 您说的 files下面：

报错：model_lib 找不到，我也不知道model_lib 到底写什么。

E FATAL EXCEPTION: Thread-4 Process: ai.mlc.mlcchat, PID: 23985 org.apache.tvm.Base$TVMError: InternalError: Check failed: (fload_exec.defined()) is false: Cannot find system lib with q4f16_1, please make sure you set model_lib field consistently with the compilation Stack trace: File "/home/qqyang/mlc-llm/cpp/serve/function_table.cc", line 120

                                                                                                    at org.apache.tvm.Base.checkCall(Base.java:173)
                                                                                                    at org.apache.tvm.Function.invoke(Function.java:130)
                                                                                                    at ai.mlc.mlcllm.JSONFFIEngine.runBackgroundLoop(JSONFFIEngine.java:64)
                                                                                                    at ai.mlc.mlcllm.MLCEngine$backgroundWorker$1.invoke(MLCEngine.kt:42)
                                                                                                    at ai.mlc.mlcllm.MLCEngine$backgroundWorker$1.invoke(MLCEngine.kt:40)
                                                                                                    at ai.mlc.mlcllm.BackgroundWorker$start$1.invoke(MLCEngine.kt:19)
                                                                                                    at ai.mlc.mlcllm.BackgroundWorker$start$1.invoke(MLCEngine.kt:18)
                                                                                                    at kotlin.concurrent.ThreadsKt$thread$thread$1.run(Thread.kt:30)

2024-10-15 18:26:11.552 23985-23985 Compatibil...geReporter ai.mlc.mlcchat D Compat change id reported: 147798919; UID 10346; state: ENABLED

QQQQQQY commented 2 weeks ago

感谢回复。和您的指点： 在 https://hf-mirror.com/mlc-ai/Qwen2.5-1.5B-Instruct-q4f16_1-MLC/tree/main 下载的 Qwen2.5-1.5B-Instruct-q4f16_1-MLC

mlc-app-config.json { "model_list": [ { "model_id": "gemma-2-2b-it-q4f16_1-MLC", "model_lib": "gemma2_q4f16_1_5cc7dbd3ae3d1040984d9720b2d7b7d4", "model_url": "https://huggingface.co/mlc-ai/gemma-2-2b-it-q4f16_1-MLC", "estimated_vram_bytes": 3000000000 }, { "model_id": "Qwen2.5-1.5B-Instruct-q4f16_1-MLC", "model_lib": "q4f16_1", "model_url": "https://huggingface.co/mlc-ai/Qwen2.5-1.5B-Instruct-q4f16_1-MLC", "estimated_vram_bytes": 3980990464 } ] }

文件放到了 您说的 files下面：

报错：model_lib 找不到，我也不知道model_lib 到底写什么。

E FATAL EXCEPTION: Thread-4 Process: ai.mlc.mlcchat, PID: 23985 org.apache.tvm.Base$TVMError: InternalError: Check failed: (fload_exec.defined()) is false: Cannot find system lib with q4f16_1, please make sure you set model_lib field consistently with the compilation Stack trace: File "/home/qqyang/mlc-llm/cpp/serve/function_table.cc", line 120

                                                                                                  at org.apache.tvm.Base.checkCall(Base.java:173)
                                                                                                  at org.apache.tvm.Function.invoke(Function.java:130)
                                                                                                  at ai.mlc.mlcllm.JSONFFIEngine.runBackgroundLoop(JSONFFIEngine.java:64)
                                                                                                  at ai.mlc.mlcllm.MLCEngine$backgroundWorker$1.invoke(MLCEngine.kt:42)
                                                                                                  at ai.mlc.mlcllm.MLCEngine$backgroundWorker$1.invoke(MLCEngine.kt:40)
                                                                                                  at ai.mlc.mlcllm.BackgroundWorker$start$1.invoke(MLCEngine.kt:19)
                                                                                                  at ai.mlc.mlcllm.BackgroundWorker$start$1.invoke(MLCEngine.kt:18)
                                                                                                  at kotlin.concurrent.ThreadsKt$thread$thread$1.run(Thread.kt:30)

2024-10-15 18:26:11.552 23985-23985 Compatibil...geReporter ai.mlc.mlcchat D Compat change id reported: 147798919; UID 10346; state: ENABLED

如果你想在app内在线下载你想要添加的模型可以这么配置: { "model_id": "Qwen2-1.5B-Instruct-q4f16_1-MLC", "model_lib": "qwen2_q4f16_1_2e221f430380225c03990ad24c3d030e", "model_url": "https://huggingface.co/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC", "estimated_vram_bytes": 3980990464 } 这样配置的话, 就不需要将已经下载好的模型放到指定目录下了, 因为 model_url 配置的是链接地址, 所以如果想使用这个模型的话, 需要在app打开后自行进行模型在线下载. 如果想要使用本地已经下载好的模型(这样的话app内就不需要进行重新下载), 需要这么配置: { "model_id": "Qwen2-1.5B-Instruct-q4f16_1-MLC", "model_lib": "qwen2_q4f16_1_2e221f430380225c03990ad24c3d030e", "model_url": "/home/qqyang/Qwen2-1.5B-Instruct-q4f16_1-MLC", // 这里修改为自己模型的本地路径 "estimated_vram_bytes": 3980990464 } 配置完成后, 使用 android studio 重新编译运行app, 然后按照本项目文档将已经下载好的模型复制到相应目录就可以了. 可以参考下这份官方文档(需要代理访问), 里面有相关配置说明: https://llm.mlc.ai/docs/deploy/android.html

huangliang113333 commented 2 weeks ago

麻烦一下， { "model_id": "Qwen2-1.5B-Instruct-q4f16_1-MLC", "model_lib": "qwen2_q4f16_1_2e221f430380225c03990ad24c3d030e", "model_url": "/home/qqyang/Qwen2-1.5B-Instruct-q4f16_1-MLC", // 这里修改为自己模型的本地路径 "estimated_vram_bytes": 3980990464 }

中的这些参数，从哪里来的？比如：qwen2_q4f16_1_2e221f430380225c03990ad24c3d030e？

是不是应该做以下步骤？ mlc_llm compile /ok/Qwen1.5-1.8B-Chat-q0f16-MLC/mlc-chat-config.json --device android -o /ok/libs/Qwen1.5-1.8B-Chat-q0f16-android.tar 编写mlc-package-config.json，改成自己想要的模型，路径等等。下面是一个填写示例 { "model_list": [ { "model_url": "https://huggingface.co/Qwen/Qwen1.5-1.8B-Chat", "model_lib": "qwen2_q0f16", "estimated_vram_bytes": 4348727787, "model_id": "Qwen1.5-1.8B-Chat-q0f16"
} ], "model_lib_path_for_prepare_libs": { "qwen2_q0f16": "/ok/libs/Qwen1.5-1.8B-Chat-q0f16-android.tar" } }

./prepare_libs.sh

等于重新生成 mlc-app-config.json

QQQQQQY commented 2 weeks ago

麻烦一下， { "model_id": "Qwen2-1.5B-Instruct-q4f16_1-MLC", "model_lib": "qwen2_q4f16_1_2e221f430380225c03990ad24c3d030e", "model_url": "/home/qqyang/Qwen2-1.5B-Instruct-q4f16_1-MLC", // 这里修改为自己模型的本地路径 "estimated_vram_bytes": 3980990464 }

中的这些参数，从哪里来的？比如：qwen2_q4f16_1_2e221f430380225c03990ad24c3d030e？

是不是应该做以下步骤？ mlc_llm compile /ok/Qwen1.5-1.8B-Chat-q0f16-MLC/mlc-chat-config.json --device android -o /ok/libs/Qwen1.5-1.8B-Chat-q0f16-android.tar 编写mlc-package-config.json，改成自己想要的模型，路径等等。下面是一个填写示例 { "model_list": [ { "model_url": "https://huggingface.co/Qwen/Qwen1.5-1.8B-Chat", "model_lib": "qwen2_q0f16", "estimated_vram_bytes": 4348727787, "model_id": "Qwen1.5-1.8B-Chat-q0f16" } ], "model_lib_path_for_prepare_libs": { "qwen2_q0f16": "/ok/libs/Qwen1.5-1.8B-Chat-q0f16-android.tar" } }

./prepare_libs.sh

等于重新生成 mlc-app-config.json

{
    "model_id": "Qwen2-1.5B-Instruct-q4f16_1-MLC", 
    "model_lib": "qwen2_q4f16_1_2e221f430380225c03990ad24c3d030e", 
    "model_url": "/home/qqyang/Qwen2-1.5B-Instruct-q4f16_1-MLC",  // 这里修改为自己模型的本地路径
    "estimated_vram_bytes": 3980990464
}

这个配置是需要自己编译模型生成 mlc-app-config.json文件, 正如你下面所说的, 不过现在的编译命令应该不是 mlc_llm compile, 而是通过 mlc_llm package 命令生成的. 可以通过这篇文章了解下完整流程, 也可以通过官方文档 了解详细信息. 这里附上一张官方文档的说明: image

huangliang113333 commented 2 weeks ago

非常感谢您的指教，需要重新生成，那我还是搭建一下环境吧，谢谢。问题解决了