[Sys]Can't use mpt-7b as llm in the system

[slark-prime]

• [ ] mpt-7b time out
• [ ] bloomz time out
• [ ] vicuna-7b time out

[slark-prime]

sudo add-apt-repository ppa:ubuntu-toolchain-r/test sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 60 --slave /usr/bin/g++ g++ /usr/bin/g++-11

[slark-prime]

sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 60 --slave /usr/bin/g++ g++ /usr/bin/g++-11

[slark-prime]

CUDA_VISIBLE_DEVICES=1 python test.py

[slark-prime]

chmod +x run_tests.sh ./run_tests.sh

[slark-prime]

Test some variables to see where the problem is:

• [x] see whether HuggingFaceEmbedding downloads the model to the local disk (YES, it is local)
• [x] see how many parameters and storage it occupies (Directly go to the file tab to see the size of model.bin file)
• [x] see how to solve the RAM not enough issue on Remote Desktop (Can not be solve, the GRAM should be enough for both the embedding model and LLM)
• [ ] See why remote access to HuggingFace Vicuna is stuck while remote access to Falcon works
• [ ] Guess: the Huggingface downloads a general model to the local disk and leaves the rest of the model remote in the Hub, which explains why still 5.4GB is occupied.
• [ ] Make the test.py's parameter optional to make debugging more convenient.