symflower eval-dev-quality issues

symflower / eval-dev-quality

DevQualityEval: An evaluation benchmark 📈 and framework to compare and evolve the quality of code generation of LLMs.

https://symflower.com/en/company/blog/2024/dev-quality-eval-v0.4.0-is-llama-3-better-than-gpt-4-for-generating-tests/

MIT License

137 stars 5 forks source link

issues

Newest

Newest Most commented Recently updated Oldest Least commented Least recently updated

Remove scoring from within the evaluation and remove and lint unused functions

#369 bauersimon opened 4 weeks ago
0
Cleanup Docker volume for SIGTERM

#368 bauersimon opened 4 weeks ago
0
Support Spring

#367 bauersimon opened 4 weeks ago
0
Remove Docker evaluation volume if evaluation is aborted

#366 bauersimon opened 1 month ago
0
Spring Boot "write-test"

#365 bauersimon opened 1 month ago
1
Customizable configuration for test framework and execution validation

#364 bauersimon closed 4 weeks ago
1
Spring boot plain write tests

#363 bauersimon closed 4 weeks ago
2
Enable Symflower linter by default in VSC

#362 ruiAzevedo19 closed 1 month ago
0
VSCode recommended extensions and debugging configuration

#361 bauersimon closed 1 month ago
0
fix, Solving Plain once (with or without template) is enough to not get disqualified

#360 bauersimon closed 1 month ago
0
fix, Use all the rules of Go linter "revive" but exclude some that do not make sense and exit "1" on errors, always

#359 zimmski closed 1 month ago
0
"Write-test" task with Symflower Smart Test Template as base

#358 bauersimon closed 1 month ago
1
Remove categorization and total score of reporting, and the "report" command

#357 zimmski closed 1 month ago
1
Remove category SVG from the report, because we are moving to a leaderboard

#356 zimmski closed 1 month ago
1
Install and run some Go linters

#355 zimmski closed 1 month ago
1
Always apply "symflower fix" and refactor "write test" task and LLM prompting to prepare for templates

#354 bauersimon closed 1 month ago
0
Simplify assessment cleanup for tests

#353 bauersimon closed 1 month ago
0
Clarify wording, logs, debugging regarding symflower fix

#352 bauersimon closed 1 month ago
0
"Write-test" task with Symflower Smart Test Template as base

#351 bauersimon closed 1 month ago
0
"Write-test" task with Symflower Smart Test Template as base

#350 bauersimon closed 1 month ago
0
Script to convert all SVG files to PNG with Inkscape

#349 bauersimon closed 1 month ago
0
Fix typos in Ruby README paths and update to new deep dive

#348 bauersimon closed 1 month ago
0
Track query tokens and save them to CSV

#347 bauersimon opened 1 month ago
0
fix, Input form template does not allow special rendering

#346 bauersimon closed 1 month ago
0
Overhaul bug and feature request issue templates

#345 bauersimon closed 1 month ago
0
Automate revision string of the binary

#344 bauersimon opened 1 month ago
0
Improve wording of templates and documentation

#343 zimmski closed 1 month ago
0
Issue templates for bugs and feature requests

#342 bauersimon closed 1 month ago
0
Different behavior depending on Go version

#341 bauersimon opened 2 months ago
0
Bump golang.org/x/image from 0.11.0 to 0.18.0

#340 dependabot[bot] closed 2 months ago
0
Update symflower.com with DevQualityEval entries

#339 zimmski closed 2 months ago
0
Add issue templates for bug and feature requests

#338 zimmski closed 1 month ago
0
Write run count to CSV report

#337 bauersimon closed 1 month ago
0
eval-dev-quality: command not found

#336 Hambaobao closed 2 months ago
3
Fixes and readme

#335 bauersimon closed 2 months ago
1
Enable Ruby tests in Windows CI

#334 bauersimon opened 2 months ago
0
Set up ruby inside the Github Actions Workflow

#333 Munsio closed 2 months ago
0
fix, Use another example for the missing import package, since the current one does not work because Ruby auto-loads the JSON module

#332 ruiAzevedo19 closed 2 months ago
2
Throw an error when trying to use a configuration file with containerized runtimes

#331 Munsio closed 2 months ago
1
Add new subcommand `verify` that just does repository verification and run that in CI

#330 bauersimon opened 2 months ago
0
Kubernetes job watching does not work when erroring

#329 Munsio opened 2 months ago
0
Update Symflower to support Ruby in symflower test and add remaining task unit tests

#328 bauersimon closed 2 months ago
1
Finalize Ruby support

#327 ruiAzevedo19 closed 2 months ago
1
Define the mistakes logic for Ruby

#326 ruiAzevedo19 closed 2 months ago
0
Convert duplicated testdata files into symlinks

#325 bauersimon closed 2 months ago
1
v0.6 results

#324 Munsio closed 2 months ago
0
Use new Symflower version which reduces error output of the "fix" command

#323 bauersimon closed 3 months ago
3
fix, Prefix openrouter model with provider ID only once

#322 bauersimon closed 3 months ago
0
fix, Score with passing tests in code-repair task cause coverage can be cheated

#321 bauersimon closed 3 months ago
1
Code repair should only consider #(passing tests) and never coverage

#320 bauersimon closed 3 months ago
0