search

nuprl / MultiPL-E

A multi-programming language benchmark for LLMs
https://nuprl.github.io/MultiPL-E/
Other
200 stars 38 forks source link

Language Names in MBPP Prompts #158

Open timothytmzhou opened 1 month ago

timothytmzhou commented 1 month ago

Hi, I was looking at the MBPP prompts and they seem to name languages wrong. For example, looking at mbpp-js many of the prompts start with "write a jsthon function to...". Or for ml, they begin with "write a mlthon function to...". I assume these should say Javascript/OCaml/etc. instead.

arjunguha commented 1 month ago

Yeah, this a case of text substitution gone wrong. We could fix them. I'm not sure it will really make any difference. The function signature is typically enough of a signal about what the programming language should be.