issues
search
huggingface
/
lighteval
Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
MIT License
814
stars
98
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
[FT] Support llama.cpp inference
#402
JoelNiklaus
opened
9 hours ago
3
Update instance type/size in endpoint model_config example
#401
albertvillanova
closed
1 day ago
3
Fix splitting for generative tasks
#400
NathanHB
opened
2 days ago
0
Quickfix: repeated vllm model cleanup when data_parallel_size>1
#399
anton-l
closed
2 days ago
3
Allowing a single prompt to use several formats for one eval.
#398
clefourrier
closed
7 hours ago
6
[FT] Add Gemba MQM Translation Metric
#397
JoelNiklaus
opened
3 days ago
2
[FT] Is it possible to save the predictions to prevent rerunning expensive inference
#396
JoelNiklaus
opened
3 days ago
1
[BUG] Can't use lighteval to evaluate the nanotron
#395
alexchen4ai
opened
3 days ago
3
Fix UKR/RUS literals
#394
hynky1999
closed
3 days ago
0
Pr sadra
#393
clefourrier
closed
2 days ago
0
Added default values to OpenAIModelConfig, updated README, fetched remote changes
#392
xtsoukala
closed
1 week ago
1
Adds template for translation tasks
#391
hynky1999
closed
4 days ago
0
Use the programmatic interface using an already in memory loaded model
#390
clefourrier
closed
3 days ago
2
Add swiss legal evals as new community tasks
#389
JoelNiklaus
opened
1 week ago
8
Fixes an error with getting the golds from the formatted_docs.
#388
JoelNiklaus
opened
1 week ago
0
Fixes a TypeError in Sacrebleu.
#387
JoelNiklaus
closed
4 days ago
2
Fixes a TypeError for generative metrics.
#386
JoelNiklaus
closed
1 week ago
0
Add litellm inference
#385
JoelNiklaus
opened
1 week ago
0
Fix metric error
#384
JoelNiklaus
closed
1 week ago
0
added tatar literals
#383
gaydmi
closed
1 week ago
3
This PR adds translation literals for Belarusian language.
#382
Kryuski
closed
2 weeks ago
0
Add Udmurt (udm) translation literals
#381
codemurt
closed
2 weeks ago
0
Added inference using litellm
#380
JoelNiklaus
closed
1 week ago
0
[FT] Evaluation using a multi-document RAG based on statistical tools and LLM as judge
#379
louisbrulenaudet
opened
3 weeks ago
2
fix: cache directory variable
#378
NazimHAli
closed
2 weeks ago
0
[BUG] Wrong environment variable used for cache directory
#377
NazimHAli
closed
2 weeks ago
1
add Shan (shn) translation literals
#376
NoerNova
closed
4 weeks ago
3
Adding MuSR
#375
clefourrier
closed
4 weeks ago
0
add bashkir variants
#374
AigizK
closed
4 weeks ago
4
[EVAL]: Add more African Benchmarks
#373
dadelani
opened
4 weeks ago
4
Add new Arabic benchmarks (5) and enhance existing tasks
#372
alielfilali01
opened
1 month ago
20
selected tasks for multilingual evaluation
#371
hynky1999
closed
1 month ago
0
[BUG] ImportError for custom tasks
#370
Bachstelze
closed
1 week ago
2
Testing upper bound on torch install to fix test suite
#369
clefourrier
closed
1 month ago
0
Mini fix for inference endpoints
#368
clefourrier
closed
1 month ago
0
wrong repo
#367
yaraksen
closed
1 month ago
0
fix typo
#366
martinscooper
closed
1 month ago
0
[FT] Using lighteval to evaluate a model on a single sample, how?
#365
dxlong2000
closed
4 weeks ago
6
Fix the dataset loading for custom tasks
#364
clefourrier
closed
1 month ago
0
Adds Baseline workflow + fixes
#363
hynky1999
closed
1 month ago
0
[FT] Pipeline does not fully handle `trust_remote_code` to load dataset
#362
Sanahm
opened
1 month ago
1
Quick fix vllm
#361
clefourrier
closed
1 month ago
0
[FT] More general approach than `output_regex` to model answer extraction
#360
sadra-barikbin
opened
1 month ago
0
adds openai models
#359
NathanHB
closed
1 month ago
0
A more general solution to model answer extraction instead of `output_regex`
#358
sadra-barikbin
opened
1 month ago
1
IrokoBench (Afric tasks)
#357
hynky1999
closed
1 month ago
0
Translation literals
#356
hynky1999
closed
1 month ago
1
[FT] Single token completion loglikelihood auto-detection
#355
hynky1999
opened
1 month ago
0
Fix Tokenization + misc fixes
#354
hynky1999
closed
1 month ago
0
[BUG] batch_size = auto:1 issue
#353
alozowski
opened
1 month ago
0
Next