Given source code with compilation errors, the model needs to repair the code such that the source code compiles. The response is validated by executing predefined tests making sure that the implementation itself is not altered.
[x] For symflower model define this task as unsupported because we always generate deterministic tests
[x] For LLM models
[x] Define the new task as supported
[x] Create an interface for tasks
[x] Interface: Task
[x] Methods
[x] Run(repository) (assessment, err): run the task for the given repository and return the assessments
[x] Identifier: returns the task identifier
[x] Define tasks
[x] TaskWriteTests
[x] The Run method is basically what we already have in evaluate/repository.go:Evaluate
[x] Remove evaluate/repository.go:Evaluate since is now part of the task
[x] TaskCodeRepair
[x] The Run method is responsible to only run the task for source code files (filter out test files and other files)
[x] The method must range over the sub-directories in mistakes testdata and and run the code repair task for each sub-directory
[x] Add two methods to the language interface
[x] DefaultFileExtension returns the language file extension
[x] DefaultTestFileSuffix returns the language test file suffix, i.e., _test.go for Go and Test.java for Java
Note: this will be used to easily filter out files
[x] Calling the Run method
[x] replace the call temporaryRepository.Evaluate(...) in evaluate/evaluate.go:Evaluate with the task Run method
[x] We are ranging over temporaryRepository.Tasks so we need a function TaskForIdentifier(taskIdentifer) that, given a task identifier, return the task struct
Goal
Given source code with compilation errors, the model needs to repair the code such that the source code compiles. The response is validated by executing predefined tests making sure that the implementation itself is not altered.
PRs
Follow-up
TODOs
symflower unit-testscode-repairsymflowermodel define this task as unsupported because we always generate deterministic testsTaskRun(repository) (assessment, err): run the task for the given repository and return the assessmentsIdentifier: returns the task identifierTaskWriteTestsRunmethod is basically what we already have inevaluate/repository.go:Evaluateevaluate/repository.go:Evaluatesince is now part of the taskTaskCodeRepairRunmethod is responsible to only run the task for source code files (filter out test files and other files)mistakestestdata and and run the code repair task for each sub-directoryDefaultFileExtensionreturns the language file extensionDefaultTestFileSuffixreturns the language test file suffix, i.e.,_test.gofor Go andTest.javafor JavaRunmethodtemporaryRepository.Evaluate(...)inevaluate/evaluate.go:Evaluatewith the taskRunmethodtemporaryRepository.Tasksso we need a functionTaskForIdentifier(taskIdentifer)that, given a task identifier, return the task struct