issues
search
huggingface
/
lighteval
Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
MIT License
845
stars
100
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
redo logging
#415
NathanHB
opened
6 hours ago
3
[FT] Adding diskcache
#414
JoelNiklaus
opened
11 hours ago
0
Add docstring docs
#413
albertvillanova
opened
13 hours ago
3
First draft of autoscale
#412
clefourrier
opened
14 hours ago
1
Add doc-builder doc-pr-upload GH Action
#411
albertvillanova
closed
1 day ago
0
[FT] Add System Prompt field in LightevalTaskConfig that can be used by model clients
#410
JoelNiklaus
opened
1 day ago
5
Speed up Bootstrapping Computation
#409
JoelNiklaus
opened
1 day ago
9
[BUG] Bootstrapping compute's stderr is very slow
#408
JoelNiklaus
opened
2 days ago
0
Nathan refacto cli
#407
NathanHB
opened
3 days ago
1
Fix typo in feature-request template
#406
albertvillanova
closed
3 days ago
1
[FT] The word "pretrained" is required in model_args but not in model_config_path
#405
albertvillanova
opened
4 days ago
1
[FT] Support batch metric computation for SampleLevelMetrics
#404
JoelNiklaus
closed
3 days ago
6
Set up docs
#403
albertvillanova
closed
1 day ago
9
[FT] Support llama.cpp inference
#402
JoelNiklaus
opened
1 week ago
3
Update instance type/size in endpoint model_config example
#401
albertvillanova
closed
1 week ago
3
Fix splitting for generative tasks
#400
NathanHB
closed
4 days ago
2
Quickfix: repeated vllm model cleanup when data_parallel_size>1
#399
anton-l
closed
1 week ago
3
Allowing a single prompt to use several formats for one eval.
#398
clefourrier
closed
1 week ago
6
[FT] Add Gemba MQM Translation Metric
#397
JoelNiklaus
opened
1 week ago
2
[FT] Is it possible to save the predictions to prevent rerunning expensive inference
#396
JoelNiklaus
opened
1 week ago
2
[BUG] Can't use lighteval to evaluate the nanotron
#395
alexchen4ai
opened
1 week ago
3
Fix UKR/RUS literals
#394
hynky1999
closed
1 week ago
0
Pr sadra
#393
clefourrier
closed
1 week ago
2
Added default values to OpenAIModelConfig, updated README, fetched remote changes
#392
xtsoukala
closed
2 weeks ago
1
Adds template for translation tasks
#391
hynky1999
closed
1 week ago
0
Use the programmatic interface using an already in memory loaded model
#390
clefourrier
closed
1 week ago
2
Add swiss legal evals as new community tasks
#389
JoelNiklaus
opened
2 weeks ago
11
Fixes an error with getting the golds from the formatted_docs.
#388
JoelNiklaus
closed
3 days ago
1
Fixes a TypeError in Sacrebleu.
#387
JoelNiklaus
closed
1 week ago
2
Fixes a TypeError for generative metrics.
#386
JoelNiklaus
closed
2 weeks ago
0
Add litellm inference
#385
JoelNiklaus
opened
2 weeks ago
1
Fix metric error
#384
JoelNiklaus
closed
2 weeks ago
0
added tatar literals
#383
gaydmi
closed
2 weeks ago
3
This PR adds translation literals for Belarusian language.
#382
Kryuski
closed
3 weeks ago
0
Add Udmurt (udm) translation literals
#381
codemurt
closed
3 weeks ago
0
Added inference using litellm
#380
JoelNiklaus
closed
2 weeks ago
0
[FT] Evaluation using a multi-document RAG based on statistical tools and LLM as judge
#379
louisbrulenaudet
opened
1 month ago
2
fix: cache directory variable
#378
NazimHAli
closed
3 weeks ago
0
[BUG] Wrong environment variable used for cache directory
#377
NazimHAli
closed
3 weeks ago
1
add Shan (shn) translation literals
#376
NoerNova
closed
1 month ago
3
Adding MuSR
#375
clefourrier
closed
1 month ago
0
add bashkir variants
#374
AigizK
closed
1 month ago
4
[EVAL]: Add more African Benchmarks
#373
dadelani
opened
1 month ago
4
Add new Arabic benchmarks (5) and enhance existing tasks
#372
alielfilali01
opened
1 month ago
28
selected tasks for multilingual evaluation
#371
hynky1999
closed
1 month ago
0
[BUG] ImportError for custom tasks
#370
Bachstelze
closed
2 weeks ago
2
Testing upper bound on torch install to fix test suite
#369
clefourrier
closed
1 month ago
0
Mini fix for inference endpoints
#368
clefourrier
closed
1 month ago
0
wrong repo
#367
yaraksen
closed
1 month ago
0
fix typo
#366
martinscooper
closed
1 month ago
0
Next