EricLBuehler
commented
4 months ago Thank you for reporting this.
Can you please try to run an example from the command line, for example the below, which loads a GGUF model as configurable in the .toml file here:
./mistralrs_server --port 1234 toml -f toml-selectors/gguf.toml
I think that if this works, we can isolate the issue with the Python bindings and not a general issue.
It ran successfully without any errors so I assumed the installation was complete. Is there any verification process for the installation?
You can run import mistralrs. However, if the pip installation succeeds, I think it worked, especially given that you were able to import the packages.
shresht8
Bug:
I am attempting to run mistral rs for inference for my own GGUF files but before that I wanted to test with the example given in the documentation. I installed mistral rs using ubuntu 22.04 since I have a windows machine with RTX 4070 GPU. I installed the dependencies required as well using ubuntu such as the libssl-dev and pkg-config. Once that was done I went into my conda environment and ran the pip install mistralrs-cuda command to install the package. It ran successfully without any errors so I assumed the installation was complete. Is there any verification process for the installation? Once the installation was complete, I successfully imported the packages and then attempted to run the example given in the documentation given for the python bindings:
runner = Runner( which=Which.GGUF( tok_model_id="mistralai/Mistral-7B-Instruct-v0.1", quantized_model_id="TheBloke/Mistral-7B-Instruct-v0.1-GGUF", quantized_filename="mistral-7b-instruct-v0.1.Q4_K_M.gguf", tokenizer_json=None, repeat_last_n=64, ) )
and got the below error:
PanicException Traceback (most recent call last) Cell In[2], line 1 ----> 1 runner = Runner( 2 which=Which.GGUF( 3 tok_model_id="mistralai/Mistral-7B-Instruct-v0.1", 4 quantized_model_id="TheBloke/Mistral-7B-Instruct-v0.1-GGUF", 5 quantized_filename="mistral-7b-instruct-v0.1.Q4_K_M.gguf", 6 tokenizer_json=None, 7 repeat_last_n=64, 8 ) 9 )
PanicException: called
Result::unwrap()on anErrvalue: LoadLibraryExW { source: Os { code: 126, kind: Uncategorized, message: "The specified module could not be found." } }I attempted to run the actual mistral model as well (not the GGUF version) like this: runner = Runner( which=Which.Plain( model_id="mistralai/Mistral-7B-Instruct-v0.1", arch=Architecture.Mistral, tokenizer_json=None, repeat_last_n=64, ) )
But I got the same error as above. Can anyone help me understand the cause of this error?
Latest commit commit 455653c59884e79f471cf910c5fd6daea7f5beca (HEAD -> master, origin/master, origin/HEAD)