What's the largest open-source LLM we can run on Colab?

[lintool]

Can someone poke around ad find out?

Initial poking by me:

• https://github.com/facebookresearch/llama/issues/120
• https://medium.com/@aitor.porcellaburu/run-open-source-llm-for-free-in-google-colab-kaggle-84dd64a0a23a
• https://betterprogramming.pub/set-up-an-llm-project-using-a-free-gpu-in-google-colab-e55453bfc760

[lintool]

Ref: https://twitter.com/1littlecoder/status/1664368229232328709

[AndreSlavescu]

https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Uncensored-GPTQ

largest model by parameter size that fits on free tier colab

[lintool]

@AndreSlavescu is there a notebook demonstrating this?

[AndreSlavescu]

@AndreSlavescu is there a notebook demonstrating this?

Made a notebook here:

https://colab.research.google.com/drive/1LjWNiGyc331IyBogYi1q1FvYTYxLj-BF?usp=sharing

works well with T4 and takes roughly 15 mins end to end (installing deps + downloading weights + loading and inf)

[AndreSlavescu]

An observation I had though was that the T4 now seems to have reduced memory. I thought it used to have 16 gigs previously, but in the event its ever slightly too little, Kaggle's free tier K80 might be a good idea as well.