symflower / eval-dev-quality

DevQualityEval: An evaluation benchmark 📈 and framework to compare and evolve the quality of code generation of LLMs.
https://symflower.com/en/company/blog/2024/dev-quality-eval-v0.4.0-is-llama-3-better-than-gpt-4-for-generating-tests/
MIT License
57 stars 3 forks source link

The prompt uses different paths depending on the OS #152

Open Munsio opened 3 weeks ago

Munsio commented 3 weeks ago

Running on Windows we do give the prompt the windows style paths instead of the linux one which leads to an different prompt (not sure who it may impact).

  Given the following Java code file "src\main\java\com\eval\Plain.java" with package "com.eval", provide a test file for this code with JUnit 5 as a test framework.
  The tests should produce 100 percent code coverage and must compile.
  The response must contain only the test code and nothing else.

  ```java
  package com.eval;

  class Plain {
      static void plain() {
      }
  }