symflower / eval-dev-quality

DevQualityEval: An evaluation benchmark 📈 and framework to compare and evolve the quality of code generation of LLMs.
https://symflower.com/en/company/blog/2024/dev-quality-eval-v0.4.0-is-llama-3-better-than-gpt-4-for-generating-tests/
MIT License
57 stars 3 forks source link

New task to check for Go and Java compilation errors #160

Closed ruiAzevedo19 closed 2 weeks ago

ruiAzevedo19 commented 3 weeks ago

Tasks

Examples

Go

File: functionWithoutOpeningBracket.go

package somepackage

functionWithoutOpeningBracket(x int) int 
     return 123
}
$ go build
./functionWithoutOpeningBracket.go:4:2: syntax error: non-declaration statement outside function body

Java

File: MethodWithoutOpeningBracket.java

package com.symflower;

public class MethodWithoutOpeningBracket {
    public static int methodWithoutOpeningBracket(int x)
        return 123;
    }
}
`mvn clean install` output ``` [INFO] Scanning for projects... [INFO] [INFO] ----------------------< com.symflower:playground >---------------------- [INFO] Building playground 1.0-SNAPSHOT [INFO] from pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- clean:3.2.0:clean (default-clean) @ playground --- [INFO] Deleting /home/rui/dev/java/playground/target [INFO] [INFO] --- resources:3.3.1:resources (default-resources) @ playground --- [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent! [INFO] Copying 0 resource from src/main/resources to target/classes [INFO] [INFO] --- compiler:3.11.0:compile (default-compile) @ playground --- [INFO] Changes detected - recompiling the module! :source [WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent! [INFO] Compiling 1 source file with javac [debug target 17] to target/classes [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /home/rui/dev/java/playground/src/main/java/com/symflower/MethodWithoutOpeningBracket.java:[4,61] ';' expected [ERROR] /home/rui/dev/java/playground/src/main/java/com/symflower/MethodWithoutOpeningBracket.java:[7,1] class, interface, enum, or record expected [INFO] 2 errors [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 0.891 s [INFO] Finished at: 2024-06-05T10:18:11+01:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.11.0:compile (default-compile) on project playground: Compilation failure: Compilation failure: [ERROR] /home/rui/dev/java/playground/src/main/java/com/symflower/MethodWithoutOpeningBracket.java:[4,61] ';' expected [ERROR] /home/rui/dev/java/playground/src/main/java/com/symflower/MethodWithoutOpeningBracket.java:[7,1] class, interface, enum, or record expected [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException ```