LLM model download

[AFMSB]

How are you downloading Mistral-7B-OpenOrca model, ikeep getting this error:

OSError: Incorrect path_or_model_id: '/media/2nvme/llm/Mistral-7B-OpenOrca'. Please provide either the path to a local folder or the repo_id of a model on the Hub.

[moomou]

I git cloned the model to a specific directory (ex git clone https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca; note you will need https://git-lfs.com/).

Please note the jupyterbook scripts and path references are custom to my local setup and haven't been cleaned up. These files are for reference and won't run as-is.

[AFMSB]

Ok, i am really interested in your project, can you gid me to the setup process, what version of python are you using, i am having issues with cuda in python 3.11

[moomou]

Python 3.11 should work. I personally used 3.10.13; as for CUDA, you dont have to install it yourself. When you install pytorch, CUDA dependencies will be pulled automatically. Just make sure your setup has the nvidia driver installed.

Check out this https://stackoverflow.com/a/72415801 on how to verify your pytorch CUDA setup

[AFMSB]

I am currently struggling with this error:

model._modules[name] = bnb.nn.Linear4bit(
                       ^^^^^^^^^^^^^^^^^

AttributeError: module 'bitsandbytes.nn' has no attribute 'Linear4bit'. Did you mean: 'Linear8bitLt'?

also, what shall i puth in this path:

WHISPER_AUDIO_BIN

[moomou]

I just double checked, I can load the OpenOrca model in 4bits. This is the pinned version of my python env

https://github.com/moomou/listening-with-llm/blob/master/requirements.txt

WHISPER_AUDIO_BIN This is the path of the extracted whisper weight.

basically, the output of this script

import whisper

model = whisper.load_model("large-v3")
audio_encoder = model.encoder
torch.save(
    audio_encoder.state_dict(),
    "<output_location>",
)