Can't Find 4Bit Model

[chigkim]

I'm trying to load llama-7b-4bit model.

git clone https://github.com/0cc4m/KoboldAI -b 4bit --recurse-submodules
more setup
cd repos
git clone https://github.com/0cc4m/GPTQ-for-LLaMa -b gptneox
cd GPTQ-for-LLaMa
python setup_cuda.py install
cd ..
cd ..

I put 4bit.pt, config.json, and tokenizer.model in models/. Then I ran python aiserver.py I can open the link, but I can't find the model to load. It just says please a load model on the left, but it just says "read only" under "1) model." There's no error in the console either. I downloaded debug, but there doesn't seem to be any error either. Sorry, first time trying this, but what am I missing?

[0cc4m]

Did you put the model in the models folder? If so it should show up. The best way to get support with setting up this repo is on Discord: https://koboldai.org/discord

[chigkim]

OMG, I found the problem! I was running on colab, and using --colab flag on aiserver.py activates noaimenu and hides "load model" menu for some reason! I deleted the line args.noaimenu = True; from aiserver.py, and I can finally see the load model! lol Then everything works as expected! Why intentionally kill the menu running on colab?

Man, I wasted many hours of troubleshooting and asking around, and people probably thought I was a crazy person who can't find load model! lol

[henk717]

@chigkim Colab has been banning some models, so by doing it the way we have been doing it gives us some flexibility to load Kobold with the right options for that model, but it also allows colab to ban the models they dislike rather than the entire UI. Its likely the only reason we are still allowed on colab.

From a user experience this also prevents people from accessing models that do not work or do not fit on colab. Rather than modifying the files could add --init only after my ckds line so it only installs Kobold and then running aiserver.py --remote instead of using --colab.