[Question] mlc_llm package

[cjhelloviewing]

❓ General Questions

When proceeding with the mlc_llm package, an error occurs as shown below. I would like some help in solving the problem. https://llm.mlc.ai/docs/index.html I am setting up the environment while looking at the document, but I am asking because I don't know.

Device: MacOS (Apple M2 Pro)

--- --- --- log --- --- ---

(base) kyunghunui-Macmini:MLCChat khkang80$ mlc_llm package [2024-08-22 13:31:01] INFO package.py:327: MLC LLM HOME: "/Users/khkang80/zproject/mlc-llm:/Users/khkang80/zproject/mlc-llm/3rdparty/tvm:/Applications/Android Studio.app/Contents/jbr/Contents/Home:/Users/khkang80/Library/Android/sdk/cmake/3.22.1/bin:/Users/khkang80/Library/Android/sdk/ndk/27.0.12077973/toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android24-clang:/Users/khkang80/Library/Android/sdk/ndk/27.0.12077973:/Users/khkang80/Library/Frameworks/Python.framework/Versions/3.10/bin:/Users/khkang80/Library/flutter/flutter/bin:/Users/khkang80/Library/Android/sdk/platform-tools:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Applications/iTerm.app/Contents/Resources/utilities" [2024-08-22 13:31:01] INFO package.py:28: Clean up all directories under "dist/bundle" [2024-08-22 13:31:01] INFO jit.py:43: MLC_JIT_POLICY = ON. Can be one of: ON, OFF, REDO, READONLY [2024-08-22 13:31:01] INFO download_cache.py:227: Downloading model from HuggingFace: HF://mlc-ai/Phi-3-mini-4k-instruct-q4f16_1-MLC [2024-08-22 13:31:01] INFO download_cache.py:29: MLC_DOWNLOAD_CACHE_POLICY = ON. Can be one of: ON, OFF, REDO, READONLY [2024-08-22 13:31:01] INFO download_cache.py:166: Weights already downloaded: /Users/khkang80/.cache/mlc_llm/model_weights/hf/mlc-ai/Phi-3-mini-4k-instruct-q4f16_1-MLC [2024-08-22 13:31:01] INFO package.py:81: Model lib is not specified for model "Phi-3-mini-4k-instruct-q4f16_1-MLC". Now jit compile the model library. [2024-08-22 13:31:01] INFO jit.py:158: Using cached model lib: /Users/khkang80/.cache/mlc_llm/model_lib/f10bb3a7c5ffc894d40d946fe0528867.tar [2024-08-22 13:31:01] INFO download_cache.py:227: Downloading model from HuggingFace: HF://mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC [2024-08-22 13:31:01] INFO download_cache.py:29: MLC_DOWNLOAD_CACHE_POLICY = ON. Can be one of: ON, OFF, REDO, READONLY [2024-08-22 13:31:01] INFO download_cache.py:166: Weights already downloaded: /Users/khkang80/.cache/mlc_llm/model_weights/hf/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC [2024-08-22 13:31:01] INFO package.py:81: Model lib is not specified for model "Qwen2-1.5B-Instruct-q4f16_1-MLC". Now jit compile the model library. [2024-08-22 13:31:01] INFO jit.py:158: Using cached model lib: /Users/khkang80/.cache/mlc_llm/model_lib/32b77923e5ce45a6001ac73424864101.tar [2024-08-22 13:31:01] INFO download_cache.py:227: Downloading model from HuggingFace: HF://mlc-ai/gemma-2-2b-it-q4f16_1-MLC [2024-08-22 13:31:01] INFO download_cache.py:29: MLC_DOWNLOAD_CACHE_POLICY = ON. Can be one of: ON, OFF, REDO, READONLY [2024-08-22 13:31:01] INFO download_cache.py:166: Weights already downloaded: /Users/khkang80/.cache/mlc_llm/model_weights/hf/mlc-ai/gemma-2-2b-it-q4f16_1-MLC [2024-08-22 13:31:01] INFO package.py:81: Model lib is not specified for model "gemma-2-2b-it-q4f16_1-MLC". Now jit compile the model library. [2024-08-22 13:31:01] INFO jit.py:158: Using cached model lib: /Users/khkang80/.cache/mlc_llm/model_lib/52ab370f0b0aefe65cfaa623106eeac4.tar [2024-08-22 13:31:01] INFO download_cache.py:227: Downloading model from HuggingFace: HF://mlc-ai/Llama-3.1-8B-Instruct-q3f16_1-MLC [2024-08-22 13:31:01] INFO download_cache.py:29: MLC_DOWNLOAD_CACHE_POLICY = ON. Can be one of: ON, OFF, REDO, READONLY [2024-08-22 13:31:01] INFO download_cache.py:166: Weights already downloaded: /Users/khkang80/.cache/mlc_llm/model_weights/hf/mlc-ai/Llama-3.1-8B-Instruct-q3f16_1-MLC [2024-08-22 13:31:01] INFO package.py:81: Model lib is not specified for model "Llama-3.1-8B-Instruct-q3f16_1-MLC". Now jit compile the model library. [2024-08-22 13:31:01] INFO jit.py:158: Using cached model lib: /Users/khkang80/.cache/mlc_llm/model_lib/2524ffad6f43813b3a9baacbf58f7e42.tar [2024-08-22 13:31:01] INFO download_cache.py:227: Downloading model from HuggingFace: HF://mlc-ai/Mistral-7B-Instruct-v0.3-q4f16_1-MLC [2024-08-22 13:31:01] INFO download_cache.py:29: MLC_DOWNLOAD_CACHE_POLICY = ON. Can be one of: ON, OFF, REDO, READONLY [2024-08-22 13:31:01] INFO download_cache.py:166: Weights already downloaded: /Users/khkang80/.cache/mlc_llm/model_weights/hf/mlc-ai/Mistral-7B-Instruct-v0.3-q4f16_1-MLC [2024-08-22 13:31:01] INFO package.py:81: Model lib is not specified for model "Mistral-7B-Instruct-v0.3-q4f16_1-MLC". Now jit compile the model library. [2024-08-22 13:31:01] INFO jit.py:158: Using cached model lib: /Users/khkang80/.cache/mlc_llm/model_lib/d8cb02f5ad0cd0a4e2a8a46a85e1ff2e.tar [2024-08-22 13:31:01] INFO package.py:154: Dump the app config below to "dist/bundle/mlc-app-config.json": { "model_list": [ { "model_id": "Phi-3-mini-4k-instruct-q4f16_1-MLC", "model_lib": "phi3_q4f16_1_5a9dfbccbb0147e0e063927839645159", "model_url": "https://huggingface.co/mlc-ai/Phi-3-mini-4k-instruct-q4f16_1-MLC", "estimated_vram_bytes": 4250586449 }, { "model_id": "Qwen2-1.5B-Instruct-q4f16_1-MLC", "model_lib": "qwen2_q4f16_1_2e221f430380225c03990ad24c3d030e", "model_url": "https://huggingface.co/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC", "estimated_vram_bytes": 3980990464 }, { "model_id": "gemma-2-2b-it-q4f16_1-MLC", "model_lib": "gemma2_q4f16_1_5cc7dbd3ae3d1040984d9720b2d7b7d4", "model_url": "https://huggingface.co/mlc-ai/gemma-2-2b-it-q4f16_1-MLC", "estimated_vram_bytes": 3000000000 }, { "model_id": "Llama-3.1-8B-Instruct-q3f16_1-MLC", "model_lib": "llama_q3f16_1_25bcdc17395bc7aaad0b7f761dc55b85", "model_url": "https://huggingface.co/mlc-ai/Llama-3.1-8B-Instruct-q3f16_1-MLC", "estimated_vram_bytes": 4679979417 }, { "model_id": "Mistral-7B-Instruct-v0.3-q4f16_1-MLC", "model_lib": "mistral_q4f16_1_c2cba77a6def4dd52f7e20b5d8576ab5", "model_url": "https://huggingface.co/mlc-ai/Mistral-7B-Instruct-v0.3-q4f16_1-MLC", "estimated_vram_bytes": 4115131883 } ] } [2024-08-22 13:31:01] INFO package.py:211: Creating lib from ['/Users/khkang80/.cache/mlc_llm/model_lib/f10bb3a7c5ffc894d40d946fe0528867.tar', '/Users/khkang80/.cache/mlc_llm/model_lib/32b77923e5ce45a6001ac73424864101.tar', '/Users/khkang80/.cache/mlc_llm/model_lib/52ab370f0b0aefe65cfaa623106eeac4.tar', '/Users/khkang80/.cache/mlc_llm/model_lib/2524ffad6f43813b3a9baacbf58f7e42.tar', '/Users/khkang80/.cache/mlc_llm/model_lib/d8cb02f5ad0cd0a4e2a8a46a85e1ff2e.tar'] [2024-08-22 13:31:01] INFO package.py:212: Validating the library dist/lib/libmodel_android.a [2024-08-22 13:31:01] INFO package.py:213: List of available model libs packaged: ['phi3_q4f16_1_5a9dfbccbb0147e0e063927839645159', 'qwen2_q4f16_1_2e221f430380225c03990ad24c3d030e', 'gemma2_q4f16_1_5cc7dbd3ae3d1040984d9720b2d7b7d4', 'llama_q3f16_1_25bcdc17395bc7aaad0b7f761dc55b85', 'mistral_q4f16_1_c2cba77a6def4dd52f7e20b5d8576ab5'], if we have '-' in the modellib string, it will be turned into '' [2024-08-22 13:31:01] INFO package.py:256: Validation pass [2024-08-22 13:31:01] INFO package.py:270: Moving "dist/lib/libmodel_android.a" to "build/lib/libmodel_android.a" [2024-08-22 13:31:01] INFO package.py:274: Building mlc4j /Users/khkang80/opt/anaconda3/bin/python: can't open file '/Users/khkang80/zproject/mlc-llm:/Users/khkang80/zproject/mlc-llm/3rdparty/tvm:/Applications/Android Studio.app/Contents/jbr/Contents/Home:/Users/khkang80/Library/Android/sdk/cmake/3.22.1/bin:/Users/khkang80/Library/Android/sdk/ndk/27.0.12077973/toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android24-clang:/Users/khkang80/Library/Android/sdk/ndk/27.0.12077973:/Users/khkang80/Library/Frameworks/Python.framework/Versions/3.10/bin:/Users/khkang80/Library/flutter/flutter/bin:/Users/khkang80/Library/Android/sdk/platform-tools:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Applications/iTerm.app/Contents/Resources/utilities/android/mlc4j/prepare_libs.py': [Errno 2] No such file or directory Traceback (most recent call last): File "/Users/khkang80/opt/anaconda3/bin/mlc_llm", line 8, in sys.exit(main()) File "/Users/khkang80/opt/anaconda3/lib/python3.9/site-packages/mlc_llm/main.py", line 53, in main cli.main(sys.argv[2:]) File "/Users/khkang80/opt/anaconda3/lib/python3.9/site-packages/mlc_llm/cli/package.py", line 64, in main package( File "/Users/khkang80/opt/anaconda3/lib/python3.9/site-packages/mlc_llm/interface/package.py", line 361, in package build_android_binding(mlc_llm_source_dir, output) File "/Users/khkang80/opt/anaconda3/lib/python3.9/site-packages/mlc_llm/interface/package.py", line 275, in build_android_binding subprocess.run([sys.executable, mlc4j_path / "prepare_libs.py"], check=True, env=os.environ) File "/Users/khkang80/opt/anaconda3/lib/python3.9/subprocess.py", line 528, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/Users/khkang80/opt/anaconda3/bin/python', PosixPath('/Users/khkang80/zproject/mlc-llm:/Users/khkang80/zproject/mlc-llm/3rdparty/tvm:/Applications/Android Studio.app/Contents/jbr/Contents/Home:/Users/khkang80/Library/Android/sdk/cmake/3.22.1/bin:/Users/khkang80/Library/Android/sdk/ndk/27.0.12077973/toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android24-clang:/Users/khkang80/Library/Android/sdk/ndk/27.0.12077973:/Users/khkang80/Library/Frameworks/Python.framework/Versions/3.10/bin:/Users/khkang80/Library/flutter/flutter/bin:/Users/khkang80/Library/Android/sdk/platform-tools:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Applications/iTerm.app/Contents/Resources/utilities/android/mlc4j/prepare_libs.py')]' returned non-zero exit status 2.

[MasterJH5574]

@cjhelloviewing Could you check if the environment variable MLC_LLM_SOURCE_DIR is properly set? You can check it with echo $MLC_LLM_SOURCE_DIR in terminal. Please make sure that it points to the mlc-llm directory (e.g., /Users/khkang80/zproject/mlc-llm)

[cjhelloviewing]

@MasterJH5574 There is no issue with the environment variables you provided.

(base) Kyunghunui-MacBookPro:MLCChat khkang80$ echo $MLC_LLM_SOURCE_DIR /Users/khkang80/zproject/mlc-llm

[MasterJH5574]

@cjhelloviewing Interesting, based on the error message,

/Users/khkang80/opt/anaconda3/bin/python: can't open file '/Users/khkang80/zproject/mlc-llm:/Users/khkang80/zproject/mlc-llm/3rdparty/tvm:/Applications/Android Studio.app/Contents/jbr/Contents/Home:/Users/khkang80/Library/Android/sdk/cmake/3.22.1/bin:/Users/khkang80/Library/Android/sdk/ndk/27.0.12077973/toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android24-clang:/Users/khkang80/Library/Android/sdk/ndk/27.0.12077973:/Users/khkang80/Library/Frameworks/Python.framework/Versions/3.10/bin:/Users/khkang80/Library/flutter/flutter/bin:/Users/khkang80/Library/Android/sdk/platform-tools:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Applications/iTerm.app/Contents/Resources/utilities/android/mlc4j/prepare_libs.py': [Errno 2] No such file or directory

, it just looks to me that the path was not right, since you can see that it contains a lot of : in it.

Given you are sure that MLC_LLM_SOURCE_DIR is properly set, would you mind directly finding the file below and replace mlc_llm_source_dir on line 264 with "/Users/khkang80/zproject/mlc-llm"? If we do this, then supposedly the "file not found" issue is gone. https://github.com/mlc-ai/mlc-llm/blob/7264faa50feb7310b68e7b15e3b4b05446fbfc8c/python/mlc_llm/interface/package.py#L264

[cjhelloviewing]

@MasterJH5574 I modified the relevant part in the mlc-llm/python/mlc_llm/interface/package.py file you told me and checked. The same problem occurs. Is there a problem with the path in the environment settings? I will check other parts more.

[cjhelloviewing]

There was an issue with setting environment variables. I organized the environment variables and succeeded in 'mlc_llm package'.

Thank you.