nuprl / MultiPL-E

A multi-programming language benchmark for LLMs
https://nuprl.github.io/MultiPL-E/
Other
200 stars 38 forks source link

All non-multiline commented prompts currently broken #114

Closed cassanof closed 1 year ago

cassanof commented 1 year ago

The issue in #111 is not isolated to R, but to most if not all languages that have non-multiline commented prompts. The problem lies in the regex in the translator. Example from CPP:

#include<assert.h>
#include<bits/stdc++.h>
// Given a string s, count the number of uppercase vowels in even indices.
For example:
count_upper('aBCdEf') returns 1
count_upper('abcdefg') returns 0
count_upper('dBBE') returns 0
long count_upper(std::string s) {