symflower / eval-dev-quality

DevQualityEval: An evaluation benchmark 📈 and framework to compare and evolve the quality of code generation of LLMs.
https://symflower.com/en/company/blog/2024/dev-quality-eval-v0.4.0-is-llama-3-better-than-gpt-4-for-generating-tests/
MIT License
133 stars 5 forks source link

Reporting tool does not work with `/**/` directory globing #304

Open Munsio opened 3 months ago

Munsio commented 3 months ago

I tried to combine all the reports of the last run into one csv with the following command:

eval-dev-quality report --evaluation-path ./**/evaluation.csv

The problem is that the /**/ globing seems to match only the first directory instead of all directories in the current directory.

bauersimon commented 3 months ago

find . -name "*.csv" | xargs eval-dev-quality report --evaluation-path for now?

bauersimon commented 3 months ago

Is the globing resolved by the shell or is that something we need to explicitly handle?

Munsio commented 3 months ago

The reporting command should already handle globing

ruiAzevedo19 commented 3 months ago

The report command uses filepath.Glob to get all filepaths given a glob pattern. There's a chance it does not support the /**/ pattern, I'll take a look

ruiAzevedo19 commented 3 months ago

The builtin glob does not support the globstar /**/. If we want that functionality we need to implement it or use a third-party library.

This library allows us to use the double star pattern: doublestar

Here is an example comparing the Go's builtin glob and the doublestar library:

func TestGlob(t *testing.T) {
    temporaryDirectory := t.TempDir()

    dirA := filepath.Join(temporaryDirectory, "a")
    dirB := filepath.Join(temporaryDirectory, "b")
    dirC := filepath.Join(temporaryDirectory, "c", "a")

    require.NoError(t, os.MkdirAll(dirA, 0700))
    require.NoError(t, os.MkdirAll(dirB, 0700))
    require.NoError(t, os.MkdirAll(dirC, 0700))

    require.NoError(t, os.WriteFile(filepath.Join(dirA, "file.csv"), []byte("content"), 0700))
    require.NoError(t, os.WriteFile(filepath.Join(dirB, "file.csv"), []byte("content"), 0700))
    require.NoError(t, os.WriteFile(filepath.Join(dirC, "file.csv"), []byte("content"), 0700))

    globPattern := filepath.Join(temporaryDirectory, "**", "file.csv")
    // Go's built-in glob.
    goGlob, err := filepath.Glob(globPattern)
    require.NoError(t, err)
    // doublestar third party.
    doublestarGlob, err := doublestar.FilepathGlob(globPattern)
    require.NoError(t, err)

    fmt.Printf("Go: %+v\n", goGlob)
    fmt.Printf("Doublestar: %+v\n", doublestarGlob)
}

The output of the test if the following:

Go: [/tmp/TestGlob4236981812/001/a/file.csv /tmp/TestGlob4236981812/001/b/file.csv]
Doublestar: [/tmp/TestGlob4236981812/001/a/file.csv /tmp/TestGlob4236981812/001/b/file.csv /tmp/TestGlob4236981812/001/c/a/file.csv]