nuprl / MultiPL-E

A multi-programming language benchmark for LLMs
https://nuprl.github.io/MultiPL-E/
Other
201 stars 38 forks source link

Strange Scala Unit test Translation for output Tuple Type w/ extra parenthesis #65

Open PootieT opened 1 year ago

PootieT commented 1 year ago

Example: HumanEval_155_even_odd_count

import scala.math._
import scala.collection.mutable._
object Problem {
    def evenOddCount(num : Long) : Tuple2[Long, Long] = {
        // Some program here
    } 
    def main(args: Array[String]) = {
    assert(evenOddCount((7l)).equals(((0l, 1l))));
    assert(evenOddCount((-78l)).equals(((1l, 1l))));
    assert(evenOddCount((3452l)).equals(((2l, 2l))));
    assert(evenOddCount((346211l)).equals(((3l, 3l))));
    assert(evenOddCount((-345821l)).equals(((3l, 3l))));
    assert(evenOddCount((-2l)).equals(((1l, 0l))));
    assert(evenOddCount((-45347l)).equals(((2l, 3l))));
    assert(evenOddCount((0l)).equals(((1l, 0l))));
    }

}

There is an extra level of parenthesis around theses expected tuple values that doesn't match up with what the programs output. At first I thought this is mistranslated for all programs with output type of Tuple2, but seems like it only applies to this one program..

arjunguha commented 1 year ago

Does this cause errors?

PootieT commented 1 year ago

Yeah, I tried removing the extra parenthesis and the same program starts passing tests.

abhijangda commented 1 year ago

Which scala version are you using? I compiled it with scala 2.13.11 and it passed tests. Here is the code I used:

import scala.math._
import scala.collection.mutable._
object Problem {
    def evenOddCount(num : Long) : Tuple2[Long, Long] = {
        return (0l, 1l); // Some program here
    } 
    def main(args: Array[String]) = {
    assert(evenOddCount((7l)).equals(((0l, 1l))));
    }
}
arjunguha commented 1 year ago

@PootieT :)