issues
search
huggingface
/
lighteval
LightEval is a lightweight LLM evaluation suite that Hugging Face has been using internally with the recently released LLM data processing library datatrove and LLM training library nanotron.
MIT License
462
stars
53
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Add maj@k metric
#158
clefourrier
closed
2 months ago
7
Nathan add logging to metrics
#157
NathanHB
closed
2 months ago
6
Fix nanotron input size bug
#156
clefourrier
closed
2 months ago
1
How to run 30b plus model with lighteval when accelerate launch failed? OOM
#155
xiechengmude
closed
1 month ago
4
adding aimo custom eval
#154
NathanHB
closed
1 month ago
2
add transformers model to be used as judge
#153
NathanHB
opened
2 months ago
2
launch lighteval using `lighteval --args`
#152
NathanHB
opened
2 months ago
0
`LatexTableWriter` created but never used.
#151
PhilipMay
opened
2 months ago
1
Homogeneize logging system
#150
clefourrier
opened
2 months ago
14
Add `Ger-RAG-eval`tasks.
#149
PhilipMay
closed
2 months ago
11
Add Code-Centric Interface to LightEval for Enhanced Usability
#148
adithya-s-k
opened
2 months ago
4
Added Namespace parameter for InferenceEndpoints, added option for passing model config directly
#147
shaltielshmid
closed
2 months ago
2
add llm as judge in metrics
#146
NathanHB
closed
2 months ago
0
Add fun widgets to the README
#145
clefourrier
closed
2 months ago
2
Add a nanotron model to the test suite
#144
clefourrier
opened
2 months ago
0
Do an intro notebook on how to use `lighteval`
#143
clefourrier
opened
2 months ago
2
CLI list tasks
#142
DimbyTa
closed
2 months ago
20
Add LLM as a judge as a metric
#141
clefourrier
closed
2 months ago
1
[MATH] Too many values to unpack (expected 2)
#140
rkinas
closed
2 months ago
3
[New Task] Add AlpacaEval LC
#139
YannDubs
opened
3 months ago
9
fix: max_length type in base_model.py
#138
csarron
closed
2 months ago
1
bump-lighteval-version-0.4
#137
NathanHB
closed
3 months ago
0
Release: v0.3.0-alpha
#136
NathanHB
closed
3 months ago
0
Add a logger in the metric functions
#135
NathanHB
opened
3 months ago
0
Update test workflow name to 'Tests'
#134
Wauplin
closed
3 months ago
1
Do not use deprecated list_files_info
#133
Wauplin
closed
3 months ago
1
Winogrande degraded results
#132
opherlieber
opened
3 months ago
5
Add config files for models
#131
clefourrier
closed
2 months ago
2
human eval run
#130
meitalbensinai
opened
3 months ago
5
Fix TextGenerationResponse import from hfh
#129
Wauplin
closed
3 months ago
1
Use config files for the model parameters
#128
clefourrier
closed
2 months ago
0
Added revision parameter for Inference Endpoint deployment
#127
shaltielshmid
closed
3 months ago
0
Adding BBH subset
#126
bilgehanertan
closed
3 months ago
2
Add BBH subset back!
#125
clefourrier
closed
3 months ago
0
Added support for launching inference endpoint with different model dtypes
#124
shaltielshmid
closed
3 months ago
1
Simplified system for extended tasks
#123
clefourrier
closed
3 months ago
3
Fix extended tasks loading
#122
NathanHB
closed
3 months ago
0
Add AGIEval
#121
clefourrier
closed
3 months ago
0
add: dtype management (#117)
#120
bilgehanertan
closed
3 months ago
4
Fixed the loglikelihood method in inference endpoints models
#119
clefourrier
closed
3 months ago
0
Homogeneize logging system
#118
clefourrier
opened
3 months ago
0
Add dtype management in inference endpoints
#117
clefourrier
closed
2 months ago
1
Fix import typo autogptq
#116
clefourrier
closed
3 months ago
0
Ensure chat models terminate generation with EOS token
#115
lewtun
closed
3 months ago
2
Add EQ Bench
#114
lewtun
opened
3 months ago
1
[BUG]: lighteval.utils import is_autogptq_available not working
#113
fanminshi
closed
3 months ago
3
Small fixes to InferenceEndpointModel
#112
shaltielshmid
closed
3 months ago
0
Reorder addition of instruction in chat template
#111
clefourrier
closed
3 months ago
0
With chat templates, instructions shouldn't be prepended to system prompt
#110
Whadup
closed
3 months ago
5
[IFEVAL] Stopping criteria fails for models with ChatML special tokens
#109
lewtun
closed
3 months ago
1
Previous
Next