issues
search
bigcode-project
/
bigcode-evaluation-harness
A framework for the evaluation of autoregressive code generation language models.
Apache License 2.0
710
stars
183
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
error: list index out of range, when testing in multi-gpu?
#105
wwngh1233
closed
3 weeks ago
6
Support Seq2SeqLM model class (to facilitate the CodeT5+ models)
#104
keyboardAnt
opened
1 year ago
0
how to use --instruction_tokens?
#103
wwngh1233
closed
11 months ago
1
Support `Salesforce/codet5p-220m` and other `T5ForConditionalGeneration` models
#102
keyboardAnt
closed
3 weeks ago
1
resolve #100
#101
IQ179
closed
1 year ago
0
What does 'bs' in LANGUAGES list mean?
#100
IQ179
closed
1 year ago
0
Add missing typescript import from MultiPL-E evaluation
#99
loubnabnl
closed
1 year ago
0
failed evaluation on GSM8K
#98
tangzhy
closed
1 year ago
4
[WIP] adding ShaderEval tasks
#97
Vipitis
closed
9 months ago
2
Update MultiPL-E prompts
#96
arjunguha
closed
1 year ago
0
Support 8bit and 4bit inference
#95
loubnabnl
closed
1 year ago
0
Getting Zeros for StarCoder on multiple-js
#94
amitbcp
closed
1 year ago
5
fix apps prompt
#93
loubnabnl
closed
1 year ago
0
APPS dataset prompting seems wrong
#92
hongcheki
closed
1 year ago
1
8-bit models unsupported
#91
cassanof
closed
1 year ago
4
Update README to use prebuilt docker images
#90
arjunguha
closed
1 year ago
0
Support `transformers.pipeline(model=...)` models like `HuggingFaceH4/starchat-beta`
#89
keyboardAnt
closed
1 year ago
2
Support StudentEval benchmark
#88
arjunguha
closed
6 months ago
0
Publish the Docker images to ghcr.io?
#87
arjunguha
closed
1 year ago
3
Attempt to make MultiPl-E's evaluation parallelization over all completions at once rather than just over each problem.
#86
esslushy
opened
1 year ago
15
Add instruction-tuning tasks mode
#85
loubnabnl
closed
1 year ago
1
adding a new task
#84
ArmelRandy
closed
1 year ago
0
santacoder fp16 causes NaN on humaneval?
#83
ywen666
closed
1 year ago
2
Reproducing the performance of HumanEval on starcoder
#82
huybery
closed
1 year ago
4
Fix LLaMA Evaluations
#81
sedrickkeh
closed
1 year ago
4
Pin to a particular MultiPL-E revision (SantaCoder)
#80
arjunguha
closed
1 year ago
0
Update requirements.txt
#79
loubnabnl
closed
1 year ago
0
Any plan for attaching release tag?
#78
wavy-jung
closed
1 year ago
4
requirements.txt doesn't support newer models (KeyError)
#77
sedrickkeh
closed
1 year ago
1
Update README.md
#76
loubnabnl
closed
1 year ago
0
Problem launching evaluation
#75
ck-amrahd
closed
1 year ago
0
Llama 7B fails for Human Eval
#74
mnoukhov
closed
1 year ago
2
Cannot run eval with local model directory
#73
luquitared
closed
1 year ago
4
fix typo in README
#72
andre15silva
closed
1 year ago
0
add trust_remote_code to tokenizer loading
#71
loubnabnl
closed
1 year ago
0
prepend and parse prefix cli arg correctly when doing FIM
#70
benlipkin
closed
1 year ago
0
Add SantaCoder FIM task
#69
loubnabnl
closed
8 months ago
3
investigate discrepancy in odex implementation
#68
loubnabnl
opened
1 year ago
0
Update readme & docs
#67
loubnabnl
closed
1 year ago
0
added FIM tokens for bigcode/large-model
#66
benlipkin
closed
1 year ago
0
Add: learning performance-improving code edits 🥧
#65
SwayamInSync
opened
1 year ago
4
Program repair
#64
keyboardAnt
opened
1 year ago
3
update humaneval postprocessing
#63
loubnabnl
closed
1 year ago
1
Rename path arguments
#62
loubnabnl
closed
1 year ago
0
remove model from accelerate prepare and add precision argument
#61
loubnabnl
closed
1 year ago
0
Add file name support for our method.
#60
SivilTaram
closed
1 year ago
0
[WIS] Program repair
#59
keyboardAnt
closed
1 year ago
0
Would be better to save generations on the fly
#58
Muennighoff
opened
1 year ago
0
chat bug fixer for humaneval-x-bugs
#57
mitya52
closed
1 year ago
3
[WIP] Add the BugRepair Task evaluation
#56
keyboardAnt
closed
1 year ago
0
Previous
Next