search

RUCAIBox / LLMBox

A comprehensive library for implementing LLMs, including a unified training pipeline and comprehensive model evaluation.
MIT License
578 stars 74 forks source link

[Feat] Support KV Cache #211

Closed huyiwen closed 5 months ago

huyiwen commented 5 months ago
Evaluation Method ppl prob generation
Dataset Hellaswag (0-shot) MMLU (5-shot) GSM (8-shot)
Performance (PC) 75.60 46.52 15.16
Performance (Vanilla) 75.59 46.53 14.56
Time (PC) 0:05:48 0:05:51 0:17:13
Time (Vanilla) 0:05:32 0:18:30 2:10:27