issues
search
huggingface
/
lighteval
Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
MIT License
845
stars
100
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
[BUG] Errors when using BLEURT metric
#315
chuandudx
closed
1 month ago
1
[FT] pass trust_remote_code as flag for loading datasets with custom code
#314
chuandudx
opened
2 months ago
1
OALL v2
#313
alielfilali01
closed
2 months ago
0
[FT] Provide an interface for easier edit of parametrizable metrics
#312
clefourrier
opened
2 months ago
1
Allow kwargs for BERTScore compute function and remove unused var
#311
chuandudx
closed
2 months ago
2
[BUG] Errors when using BERTScore for evaluation
#310
chuandudx
closed
1 month ago
4
Fix Metrics import path in community task template file.
#309
chuandudx
closed
1 month ago
7
Selecting tasks using their superset
#308
hynky1999
closed
1 month ago
5
Update README.md to add lighteval pip install steps
#307
clefourrier
closed
2 months ago
1
Allow AdapterModels to have custom tokens
#306
mapmeld
opened
2 months ago
5
[FT] Remove obsolete config properties (frozen, output_regex)
#305
hynky1999
opened
2 months ago
1
Skip tests if no secrets are provided
#304
hynky1999
closed
2 months ago
0
fix doc in readme
#303
NathanHB
closed
2 months ago
0
[BUG] No script named 'run_evals_accelerate.py'
#302
mxjmtxrm
closed
2 months ago
1
[BUG] TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]
#301
alielfilali01
closed
2 months ago
2
fix readme doc about custom tasks
#300
NathanHB
closed
2 months ago
0
[BUG] Error when using TGI endpoint.
#299
Vanessa-Taing
closed
2 months ago
3
Fixes bug: `You can't create a model without either a list of model_args or a model_config_path` when model_config_path was submited.
#298
NathanHB
closed
2 months ago
0
[BUG] ValueError: You can't create a model without either a list of model_args or a model_config_path.
#297
Vanessa-Taing
closed
2 months ago
2
[FT] Any Pypi package for this toolkit?
#296
zhimin-z
closed
2 months ago
3
Adds config tempaltes
#295
hynky1999
closed
2 months ago
1
[FT] Task groupings as separate tasks
#294
hynky1999
closed
1 month ago
0
Support for multilingual generative metrics
#293
hynky1999
closed
2 months ago
2
fix(accelerate): Fix missing model_config_path
#292
srossi93
closed
2 months ago
4
bump dev version to 0.5.0
#291
NathanHB
closed
2 months ago
0
bump nltk version
#290
NathanHB
closed
2 months ago
0
Task config
#289
hynky1999
closed
2 months ago
1
[BUG] Question on batch preparation in MMLU evaluation
#288
JefferyChen453
opened
2 months ago
3
Tokenization-wise encoding
#287
hynky1999
closed
2 months ago
4
[BUG] Nanotron batch detection doesn't work
#286
hynky1999
opened
2 months ago
0
Standalone nanotron config
#285
hynky1999
closed
2 months ago
2
Logging Revamp
#284
hynky1999
closed
2 months ago
1
fix nanotron
#283
NathanHB
closed
2 months ago
2
adding documentation
#282
NathanHB
closed
1 month ago
2
Adding chat completion task to endpoint models
#281
sadra-barikbin
opened
3 months ago
0
Make info loggers dataclass
#280
hynky1999
closed
3 months ago
0
Remove expensive prediction run during test collection
#279
hynky1999
closed
3 months ago
0
[BUG] Can not load `deutsche-telekom/Ger-RAG-eval` dataset.
#278
PhilipMay
opened
3 months ago
2
[BUG] community_tasks not working or example is broken
#277
PhilipMay
closed
2 months ago
2
Probability Metric + New Normalization
#276
hynky1999
closed
2 months ago
4
[BUG] Zero accuracy in Hellaswag for Llama-2-7b (using 8bit quantization)
#275
rankofootball
opened
3 months ago
2
add vlmm backend
#274
NathanHB
closed
2 months ago
0
[FT] Open ai endpoint
#273
Pommel4711
closed
1 month ago
1
Refactoring the few shot management
#272
clefourrier
closed
3 months ago
0
Small file reorg (only renames/moves)
#271
clefourrier
closed
3 months ago
0
Data Loading Issues in Offline Mode
#270
shahad2099
closed
3 months ago
3
Programmatic interface + cleaner management of requests
#269
clefourrier
closed
3 months ago
0
updates ifeval repo
#268
NathanHB
closed
3 months ago
0
fix the location of tasks list in the readme
#267
NathanHB
closed
3 months ago
0
[EVAL] Update IFEval dataset
#266
clefourrier
closed
3 months ago
1
Previous
Next