search

mlc-ai / mlc-llm

Universal LLM Deployment Engine with ML Compilation
https://llm.mlc.ai/
Apache License 2.0
19.2k stars 1.58k forks source link

"dist/vicuna-v1-7b/float16/vicuna-v1-7b_vulkan_float16.so" not found #149

Closed NeoWang9999 closed 1 year ago

NeoWang9999 commented 1 year ago

I installed mlc chat according to the documentation, but encountered an error

Cannot find vicuna-v1-7b lib in preferred path "dist/vicuna-v1-7b/float16/vicuna-v1-7b_vulkan_float16.so" or other candidate paths

image

My environment is Ubuntu 22.04，windows WSL

nvidia-smi output:

Mon May 15 21:34:53 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.46                 Driver Version: 531.61       CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3060         On | 00000000:01:00.0  On |                  N/A |
|  0%   38C    P8               19W / 170W|   3702MiB / 12288MiB |      1%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A        22      G   /Xwayland                                 N/A      |
|    0   N/A  N/A        23      G   /Xwayland                                 N/A      |
|    0   N/A  N/A        25      G   /Xwayland                                 N/A      |
+---------------------------------------------------------------------------------------+
sudeepag commented 1 year ago

Hi @NeoWang9999 , thanks for trying out the MLC LLM project. It is possible that you are attempting to run the model using cuda, but it has been compiled using vulkan. Could you try running the following, and see if that works?

$ mlc_chat_cli --device-name=vulkan
NeoWang9999 commented 1 year ago

Hi @NeoWang9999 , thanks for trying out the MLC LLM project. It is possible that you are attempting to run the model using cuda, but it has been compiled using vulkan. Could you try running the following, and see if that works?

$ mlc_chat_cli --device-name=vulkan

Hi @sudeepag , thanks for reply, I just tried that, it give me the same error

image

sudeepag commented 1 year ago

Interesting, can you try these three things and post the results here?

$ cd /mnt/c/Ubuntu/mlc
$ ls -R
$ mlc_chat_cli -h
$ conda activate mlc-chat
$ conda list mlc-chat-nightly
tantanorange commented 1 year ago

I don't know why the path error happened. But I have a similar problem on mac1: https://github.com/mlc-ai/mlc-llm/issues/138. It seems that following the instruction (here: https://mlc.ai/mlc-llm/#windows-linux-mac) leads to a dist/vicuna-v1-7b folder that does not contain a float16 folder (https://huggingface.co/mlc-ai/demo-vicuna-v1-7b-int3/tree/main)

I don't know, I'm just a beginner that has a similar problem with the M1 machine.

NeoWang9999 commented 1 year ago

Interesting, can you try these three things and post the results here?

$ cd /mnt/c/Ubuntu/mlc
$ ls -R
$ mlc_chat_cli -h
$ conda activate mlc-chat
$ conda list mlc-chat-nightly

here is the result:


(mlc-chat) nw@DESKTOP-BADE1QI:/mnt/c/Ubuntu/mlc$ ls -R
.:
Dockerfile  dist  docker-compose.yml  mlc-llm

./dist:
lib  vicuna-v1-7b  vicuna-v1-7b_int4

./dist/lib:
README.md  mlc-chat.apk  vicuna-v1-7b-q3f16_0-metal.so  vicuna-v1-7b-q3f16_0-metal_x86_64.dylib  vicuna-v1-7b-q3f16_0-vulkan.dll  vicuna-v1-7b-q3f16_0-vulkan.so

./dist/vicuna-v1-7b:
params  tokenizer.model

./dist/vicuna-v1-7b/params:
ndarray-cache.json    params_shard_112.bin  params_shard_128.bin  params_shard_25.bin  params_shard_40.bin  params_shard_56.bin  params_shard_71.bin  params_shard_87.bin
params_shard_0.bin    params_shard_113.bin  params_shard_129.bin  params_shard_26.bin  params_shard_41.bin  params_shard_57.bin  params_shard_72.bin  params_shard_88.bin
params_shard_1.bin    params_shard_114.bin  params_shard_13.bin   params_shard_27.bin  params_shard_42.bin  params_shard_58.bin  params_shard_73.bin  params_shard_89.bin
params_shard_10.bin   params_shard_115.bin  params_shard_130.bin  params_shard_28.bin  params_shard_43.bin  params_shard_59.bin  params_shard_74.bin  params_shard_9.bin
params_shard_100.bin  params_shard_116.bin  params_shard_131.bin  params_shard_29.bin  params_shard_44.bin  params_shard_6.bin   params_shard_75.bin  params_shard_90.bin
params_shard_101.bin  params_shard_117.bin  params_shard_14.bin   params_shard_3.bin   params_shard_45.bin  params_shard_60.bin  params_shard_76.bin  params_shard_91.bin
params_shard_102.bin  params_shard_118.bin  params_shard_15.bin   params_shard_30.bin  params_shard_46.bin  params_shard_61.bin  params_shard_77.bin  params_shard_92.bin
params_shard_103.bin  params_shard_119.bin  params_shard_16.bin   params_shard_31.bin  params_shard_47.bin  params_shard_62.bin  params_shard_78.bin  params_shard_93.bin
params_shard_104.bin  params_shard_12.bin   params_shard_17.bin   params_shard_32.bin  params_shard_48.bin  params_shard_63.bin  params_shard_79.bin  params_shard_94.bin
params_shard_105.bin  params_shard_120.bin  params_shard_18.bin   params_shard_33.bin  params_shard_49.bin  params_shard_64.bin  params_shard_8.bin   params_shard_95.bin
params_shard_106.bin  params_shard_121.bin  params_shard_19.bin   params_shard_34.bin  params_shard_5.bin   params_shard_65.bin  params_shard_80.bin  params_shard_96.bin
params_shard_107.bin  params_shard_122.bin  params_shard_2.bin    params_shard_35.bin  params_shard_50.bin  params_shard_66.bin  params_shard_81.bin  params_shard_97.bin
params_shard_108.bin  params_shard_123.bin  params_shard_20.bin   params_shard_36.bin  params_shard_51.bin  params_shard_67.bin  params_shard_82.bin  params_shard_98.bin
params_shard_109.bin  params_shard_124.bin  params_shard_21.bin   params_shard_37.bin  params_shard_52.bin  params_shard_68.bin  params_shard_83.bin  params_shard_99.bin
params_shard_11.bin   params_shard_125.bin  params_shard_22.bin   params_shard_38.bin  params_shard_53.bin  params_shard_69.bin  params_shard_84.bin
params_shard_110.bin  params_shard_126.bin  params_shard_23.bin   params_shard_39.bin  params_shard_54.bin  params_shard_7.bin   params_shard_85.bin
params_shard_111.bin  params_shard_127.bin  params_shard_24.bin   params_shard_4.bin   params_shard_55.bin  params_shard_70.bin  params_shard_86.bin

./dist/vicuna-v1-7b_int4:
float16  tokenizer.model

./dist/vicuna-v1-7b_int4/float16:
ndarray-cache.json    params_shard_112.bin  params_shard_128.bin  params_shard_25.bin  params_shard_40.bin  params_shard_56.bin  params_shard_71.bin  params_shard_87.bin
params_shard_0.bin    params_shard_113.bin  params_shard_129.bin  params_shard_26.bin  params_shard_41.bin  params_shard_57.bin  params_shard_72.bin  params_shard_88.bin
params_shard_1.bin    params_shard_114.bin  params_shard_13.bin   params_shard_27.bin  params_shard_42.bin  params_shard_58.bin  params_shard_73.bin  params_shard_89.bin
params_shard_10.bin   params_shard_115.bin  params_shard_130.bin  params_shard_28.bin  params_shard_43.bin  params_shard_59.bin  params_shard_74.bin  params_shard_9.bin
params_shard_100.bin  params_shard_116.bin  params_shard_131.bin  params_shard_29.bin  params_shard_44.bin  params_shard_6.bin   params_shard_75.bin  params_shard_90.bin
params_shard_101.bin  params_shard_117.bin  params_shard_14.bin   params_shard_3.bin   params_shard_45.bin  params_shard_60.bin  params_shard_76.bin  params_shard_91.bin
params_shard_102.bin  params_shard_118.bin  params_shard_15.bin   params_shard_30.bin  params_shard_46.bin  params_shard_61.bin  params_shard_77.bin  params_shard_92.bin
params_shard_103.bin  params_shard_119.bin  params_shard_16.bin   params_shard_31.bin  params_shard_47.bin  params_shard_62.bin  params_shard_78.bin  params_shard_93.bin
params_shard_104.bin  params_shard_12.bin   params_shard_17.bin   params_shard_32.bin  params_shard_48.bin  params_shard_63.bin  params_shard_79.bin  params_shard_94.bin
params_shard_105.bin  params_shard_120.bin  params_shard_18.bin   params_shard_33.bin  params_shard_49.bin  params_shard_64.bin  params_shard_8.bin   params_shard_95.bin
params_shard_106.bin  params_shard_121.bin  params_shard_19.bin   params_shard_34.bin  params_shard_5.bin   params_shard_65.bin  params_shard_80.bin  params_shard_96.bin
params_shard_107.bin  params_shard_122.bin  params_shard_2.bin    params_shard_35.bin  params_shard_50.bin  params_shard_66.bin  params_shard_81.bin  params_shard_97.bin
params_shard_108.bin  params_shard_123.bin  params_shard_20.bin   params_shard_36.bin  params_shard_51.bin  params_shard_67.bin  params_shard_82.bin  params_shard_98.bin
params_shard_109.bin  params_shard_124.bin  params_shard_21.bin   params_shard_37.bin  params_shard_52.bin  params_shard_68.bin  params_shard_83.bin  params_shard_99.bin
params_shard_11.bin   params_shard_125.bin  params_shard_22.bin   params_shard_38.bin  params_shard_53.bin  params_shard_69.bin  params_shard_84.bin
params_shard_110.bin  params_shard_126.bin  params_shard_23.bin   params_shard_39.bin  params_shard_54.bin  params_shard_7.bin   params_shard_85.bin
params_shard_111.bin  params_shard_127.bin  params_shard_24.bin   params_shard_4.bin   params_shard_55.bin  params_shard_70.bin  params_shard_86.bin

./mlc-llm:
3rdparty  CMakeLists.txt  LICENSE  README.md  android  build.py  cpp  ios  log_db  mlc_llm  scripts  setup.py  site  tests  version.py

./mlc-llm/3rdparty:
argparse  sentencepiece-js  tokenizers-cpp

./mlc-llm/3rdparty/argparse:

./mlc-llm/3rdparty/sentencepiece-js:

./mlc-llm/3rdparty/tokenizers-cpp:
CMakeLists.txt  Cargo.toml  src  tokenizers.h

./mlc-llm/3rdparty/tokenizers-cpp/src:
lib.rs

./mlc-llm/android:
MLCChat  README.md  prepare_libs.sh

./mlc-llm/android/MLCChat:
app  build.gradle  gradle  gradle.properties  gradlew  gradlew.bat  settings.gradle

./mlc-llm/android/MLCChat/app:
build.gradle  proguard-rules.pro  src

./mlc-llm/android/MLCChat/app/src:
main

./mlc-llm/android/MLCChat/app/src/main:
AndroidManifest.xml  ic_launcher-playstore.png  java  jni  res

./mlc-llm/android/MLCChat/app/src/main/java:
ai

./mlc-llm/android/MLCChat/app/src/main/java/ai:
mlc

./mlc-llm/android/MLCChat/app/src/main/java/ai/mlc:
mlcchat

./mlc-llm/android/MLCChat/app/src/main/java/ai/mlc/mlcchat:
ChatState.java  Config.java  Downloader.java  LLMChat.java  MainActivity.java  MessageAdapter.java  MessageData.java  Utils.java

./mlc-llm/android/MLCChat/app/src/main/jni:
Android.mk  Application.mk  tvm_runtime.h

./mlc-llm/android/MLCChat/app/src/main/res:
drawable  layout  mipmap-anydpi-v26  mipmap-hdpi  mipmap-mdpi  mipmap-xhdpi  mipmap-xxhdpi  mipmap-xxxhdpi  values  values-night  xml

./mlc-llm/android/MLCChat/app/src/main/res/drawable:
bot_rounded_corner.xml  user_rounded_corner.xml

./mlc-llm/android/MLCChat/app/src/main/res/layout:
activity_main.xml  bot_message.xml  user_message.xml

./mlc-llm/android/MLCChat/app/src/main/res/mipmap-anydpi-v26:
ic_launcher.xml  ic_launcher_round.xml

./mlc-llm/android/MLCChat/app/src/main/res/mipmap-hdpi:
ic_launcher.png  ic_launcher_foreground.png  ic_launcher_round.png

./mlc-llm/android/MLCChat/app/src/main/res/mipmap-mdpi:
ic_launcher.png  ic_launcher_foreground.png  ic_launcher_round.png

./mlc-llm/android/MLCChat/app/src/main/res/mipmap-xhdpi:
ic_launcher.png  ic_launcher_foreground.png  ic_launcher_round.png

./mlc-llm/android/MLCChat/app/src/main/res/mipmap-xxhdpi:
ic_launcher.png  ic_launcher_foreground.png  ic_launcher_round.png

./mlc-llm/android/MLCChat/app/src/main/res/mipmap-xxxhdpi:
ic_launcher.png  ic_launcher_foreground.png  ic_launcher_round.png

./mlc-llm/android/MLCChat/app/src/main/res/values:
colors.xml  ic_launcher_background.xml  strings.xml  themes.xml

./mlc-llm/android/MLCChat/app/src/main/res/values-night:
themes.xml

./mlc-llm/android/MLCChat/app/src/main/res/xml:
backup_rules.xml  data_extraction_rules.xml

./mlc-llm/android/MLCChat/gradle:
wrapper

./mlc-llm/android/MLCChat/gradle/wrapper:
gradle-wrapper.jar  gradle-wrapper.properties

./mlc-llm/cpp:
cli_main.cc  llm_chat.cc  llm_chat.h

./mlc-llm/ios:
MLCChat  MLCChat.xcodeproj  README.md  prepare_libs.sh  prepare_params.sh

./mlc-llm/ios/MLCChat:
 Assets.xcassets   ChatView.swift   LLMChat.mm                  MLCChat.entitlements   MessageView.swift   ThreadWorker.swift
 ChatState.swift   Info.plist       MLCChat-Bridging-Header.h   MLCChatApp.swift      'Preview Content'

./mlc-llm/ios/MLCChat/Assets.xcassets:
AccentColor.colorset  AppIcon.appiconset  Contents.json

./mlc-llm/ios/MLCChat/Assets.xcassets/AccentColor.colorset:
Contents.json

./mlc-llm/ios/MLCChat/Assets.xcassets/AppIcon.appiconset:
Contents.json  mlc-logo.png

'./mlc-llm/ios/MLCChat/Preview Content':
'Preview Assets.xcassets'

'./mlc-llm/ios/MLCChat/Preview Content/Preview Assets.xcassets':
Contents.json

./mlc-llm/ios/MLCChat.xcodeproj:
project.pbxproj  project.xcworkspace  xcshareddata

./mlc-llm/ios/MLCChat.xcodeproj/project.xcworkspace:
contents.xcworkspacedata  xcshareddata

./mlc-llm/ios/MLCChat.xcodeproj/project.xcworkspace/xcshareddata:
IDEWorkspaceChecks.plist

./mlc-llm/ios/MLCChat.xcodeproj/xcshareddata:
xcschemes

./mlc-llm/ios/MLCChat.xcodeproj/xcshareddata/xcschemes:
MLCChat.xcscheme

./mlc-llm/log_db:
RedPajama-INCITE-Chat-3B-v1  dolly-v2-3b  vicuna-v1-7b

./mlc-llm/log_db/RedPajama-INCITE-Chat-3B-v1:
database_tuning_record.json  database_workload.json

./mlc-llm/log_db/dolly-v2-3b:
database_tuning_record.json  database_workload.json

./mlc-llm/log_db/vicuna-v1-7b:
database_tuning_record.json  database_workload.json

./mlc-llm/mlc_llm:
__init__.py  conversation.py  dispatch  relax_model  transform  tuning.py  utils.py

./mlc-llm/mlc_llm/dispatch:
__init__.py  dispatch_tir_operator.py  dispatch_tir_operator_adreno.py  gpt_neox  llama

./mlc-llm/mlc_llm/dispatch/gpt_neox:
__init__.py  dolly_v2_3b.py  dolly_v2_3b_mod.py

./mlc-llm/mlc_llm/dispatch/llama:
__init__.py  main.py

./mlc-llm/mlc_llm/relax_model:
__init__.py  gpt_neox.py  llama.py  modules.py  moss.py

./mlc-llm/mlc_llm/transform:
__init__.py  decode_matmul_ewise.py  quantization.py  transpose_matmul.py

./mlc-llm/scripts:
gh_deploy_site.sh  local_deploy_site.sh  prep_deps.sh

./mlc-llm/site:
_config.yml  gif  img  index.md

./mlc-llm/site/gif:
android-demo.gif  ios-demo.gif  linux-demo.gif

./mlc-llm/site/img:
android  diag.svg

./mlc-llm/site/img/android:
android-diagram.png  android-studio.png  android-vs-ios.png  local-advantage.png

./mlc-llm/tests:
chat.py  debug  evaluate.py

./mlc-llm/tests/debug:
compare_lib.py

(mlc-chat) nw@DESKTOP-BADE1QI:/mnt/c/Ubuntu/mlc$ mlc_chat_cli -h
Usage: mlc_chat [--help] [--version] [--device-name VAR] [--device_id VAR] [--artifact-path VAR] [--model VAR] [--dtype VAR] [--params VAR] [--evaluate]

Optional arguments:
  -h, --help            shows help message and exits
  -v, --version         prints version information and exits
  --device-name         [default: "auto"]
  --device_id           [default: 0]
  --artifact-path       [default: "dist"]
  --model               [default: "vicuna-v1-7b"]
  --dtype               [default: "auto"]
  --params              [default: "auto"]
  --evaluate
(mlc-chat) nw@DESKTOP-BADE1QI:/mnt/c/Ubuntu/mlc$ conda activate mlc-chat
(mlc-chat) nw@DESKTOP-BADE1QI:/mnt/c/Ubuntu/mlc$ conda list mlc-chat-nightly
# packages in environment at /home/nw/anaconda3/envs/mlc-chat:
#
# Name                    Version                   Build  Channel
mlc-chat-nightly          0.1.dev34       34_g9129a4f_h1234567_0    mlc-ai
NeoWang9999 commented 1 year ago

I don't know why the path error happened. But I have a similar problem on mac1: #138. It seems that following the instruction (here: https://mlc.ai/mlc-llm/#windows-linux-mac) leads to a dist/vicuna-v1-7b folder that does not contain a float16 folder (https://huggingface.co/mlc-ai/demo-vicuna-v1-7b-int3/tree/main)

I don't know, I'm just a beginner that has a similar problem with the M1 machine.

yes, and I have tried download the other link: "https://huggingface.co/mlc-ai/demo-vicuna-v1-7b-int4" which include float16 folder, but it still doesn't work .

sudeepag commented 1 year ago

@NeoWang9999 Thanks for sharing the output. It looks like you may have an older version of the mlc-chat-nightly library. Could you try the following?

$ conda uninstall mlc-chat-nightly
$ conda install -c mlc-ai -c conda-forge mlc-chat-nightly

And verify that your environment has a newer version of mlc-chat-nightly.

(sudeep-mlc-llm) conda list mlc-chat-nightly
# packages in environment at /home/sudeepag/.conda/envs/sudeep-mlc-llm:
#
# Name                    Version                   Build  Channel
mlc-chat-nightly          0.1.dev40       40_g1a4ee89_h1234567_0    mlc-ai
ScorpionOO8 commented 1 year ago

@NeoWang9999 Thanks for sharing the output. It looks like you may have an older version of the mlc-chat-nightly library. Could you try the following?

$ conda uninstall mlc-chat-nightly
$ conda install -c mlc-ai -c conda-forge mlc-chat-nightly

And verify that your environment has a newer version of mlc-chat-nightly.

(sudeep-mlc-llm) conda list mlc-chat-nightly
# packages in environment at /home/sudeepag/.conda/envs/sudeep-mlc-llm:
#
# Name                    Version                   Build  Channel
mlc-chat-nightly          0.1.dev40       40_g1a4ee89_h1234567_0    mlc-ai

I tried this and for some reason it installed 0.1dev34. This didn't solve my original error at all. Maybe I have the wrong conda? I used Miniconda3.

sudeepag commented 1 year ago

@ScorpionOO8 Try specifying the version number, e.g. 

$ conda install -c mlc-ai mlc-chat-nightly=0.1.dev40
ScorpionOO8 commented 1 year ago

@ScorpionOO8 Try specifying the version number, e.g.

$ conda install -c mlc-ai mlc-chat-nightly=0.1.dev40

I get this error trying to do that:

Collecting package metadata (current_repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Collecting package metadata (repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

Current channels:

To search for alternate channels that may provide the conda package you're looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.

NeoWang9999 commented 1 year ago

@ScorpionOO8 Try specifying the version number, e.g.

$ conda install -c mlc-ai mlc-chat-nightly=0.1.dev40

that works for me， thanks！

hqd19950905 commented 1 year ago

@ScorpionOO8 Try specifying the version number, e.g.

$ conda install -c mlc-ai mlc-chat-nightly=0.1.dev40

I get this error trying to do that:

Collecting package metadata (current_repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Collecting package metadata (repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  • mlc-chat-nightly=0.1.dev40

Current channels:

To search for alternate channels that may provide the conda package you're looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.

I have the same issue, because the newest version for Mac M1 is 0.1.dev34. I checked this website. https://anaconda.org/mlc-ai/mlc-chat-nightly/files

tqchen commented 1 year ago

Thanks for reporting, this is likely due to the lagging of the M1 build, we are upgrading the build and will report back here

tantanorange commented 1 year ago

Great project! Much much appreciated! --Kun

deep-pipeline commented 1 year ago

Looks like osx-64 is getting very regular (almost nightly) uploads from mlc-ai uploader.. but osx-arm64 is only uploaded by junrushao1994 (last two were dev.34 8 days ago, and dev.9 18 days ago) so I guess maybe the builds for osx-arm64 aren't automated like the others?

Thanks for all your work and for looking into this.

mbarsot commented 1 year ago

hello, on macbook air M1 8 GB i get this error:

_Cannot find library "vicuna-v1-7b-q3f160-metal.so" in dist/prebuilt/lib or other search paths

after executing:

conda create -n mlc-chat conda activate mlc-chat conda install git git-lfs conda install -c mlc-ai -c conda-forge mlc-chat-nightly mkdir -p dist/prebuilt cd dist/prebuilt git lfs install git clone https://huggingface.co/mlc-ai/mlc-chat-vicuna-v1-7b-q3f16_0 git clone https://github.com/mlc-ai/binary-mlc-llm-libs.git lib cd ../.. mlc_chat_cli --local-id vicuna-v1-7b-q3f16_0

what is wrong?

-- sidenote: why the great MLC Chat Ios App cannot run via testflight on M1 mac (it is listed as "ios only") ?

tqchen commented 1 year ago

The M1 build should now be fixed, please try to do 

conda install -c mlc-ai -c conda-forge mlc-chat-nightly --force-reinstall

And try run the instructions here again https://mlc.ai/mlc-llm/#windows-linux-mac

osx-arm64 nightly shuold be fixed

ScorpionOO8 commented 1 year ago

Worked for me. I manually deleted the old installation's files I used to have in Dist as the new files loaded into Dist/prebuilt. This runs faster than any other LLM I've run on my Macbook Air M1 (8gb ram). And it doesn't max out my memory and freeze my computer, wild (happened with other, M1 CPU-only programs). Thanks all!

tqchen commented 1 year ago

Thank you @ScorpionOO8 for confirming, closing this for now, and feel free to post followups

mbarsot commented 1 year ago

@tqchen worked great! thanks

logicalpath commented 1 year ago

Hello, I'm running Linux Mint 21.1 When I run this command from the instructions on https://mlc.ai/mlc-llm/#windows-linux-mac, I get an error locating a so file:

mlc-chat) eddie@eddie-XPS-15-9560:~/workspace/mlc$ mlc_chat_cli --local-id vicuna-v1-7b-q3f16_0 Use MLC config: "dist/prebuilt/mlc-chat-vicuna-v1-7b-q3f16_0/mlc-chat-config.json" Use model weights: "dist/prebuilt/mlc-chat-vicuna-v1-7b-q3f16_0/ndarray-cache.json" Cannot find library "vicuna-v1-7b-q3f16_0-vulkan.so" in "dist"/prebuilt/lib or other search paths

However, The file does exist in the ./lib directory:

(mlc-chat) eddie@eddie-XPS-15-9560:~/workspace/mlc$ tree -f | grep vicuna-v1-7b-q3f16_0-vulkan.so ├── ./lib/vicuna-v1-7b-q3f16_0-vulkan.so (mlc-chat) eddie@eddie-XPS-15-9560:~/workspace/mlc$

Should I just move the file or change the search path? Any help appreciated.

Thanks, Eddie

tqchen commented 1 year ago

the lib needs to sit in workspace/mlc/dist/prebuilt/lib

logicalpath commented 1 year ago

Thank you @tqchen - that got me past that hurdle!

tqchen commented 1 year ago

glad it works