Notebooks to run SakuraLLM on colab/kaggle
49
stars
4
forks
source link
SakuraLLM-Notebooks
介绍
部署教程
News
- 2024-05-09: 更新 Sakura-32B notebook.
- 2024-04-23: 更换 llama-cpp-python wheel index. (#7)
- 2024-04-11:更新 Kaggle-llama.cpp 模型版本至 Sakura-14B-Qwen2beta-v0.9,更新 llama-cpp-python 安装指令 (#5).
- 2024-03-09: 新增
ollama
后端 Notebook
- 2024-02-07: 新增
llama-cpp-python
prebuilt wheels
- 2024-01-11: 更新 Kaggle-llama.cpp 模型版本至 Sakura-13B-LNovel-v0.9
- 2024-01-01: 新增 localtunnel 内网穿透工具(无需注册),ngrokToken 留空即可启用 localtunnel
模型运行情况
- transformers autogptq模型(使用v0.8版本进行测试)
模型量化类型 |
colab (T4) |
kaggle (T4×2) |
全量 |
❌ |
❌ |
8bit |
❌ |
✔ |
4bit |
✔ |
✔ |
3bit |
✔ |
✔ |
- llama.cpp GGUF模型(使用v0.9版本进行测试)
模型量化类型 |
colab (T4) |
kaggle (T4×2) |
fp16 |
❌ |
❌ |
Q8_0 |
❌ |
✔ |
Q6_K |
❌ |
✔ |
Q5_K_M |
✔ |
✔ |
Q4_K_M |
✔ |
✔ |
Q3_K_M |
✔ |
✔ |
Q2_K |
✔ |
✔ |
致谢