symflower / eval-dev-quality

DevQualityEval: An evaluation benchmark 📈 and framework to compare and evolve the quality of code generation of LLMs.
https://symflower.com/en/company/blog/2024/dev-quality-eval-v0.4.0-is-llama-3-better-than-gpt-4-for-generating-tests/
MIT License
57 stars 3 forks source link

Collect Go coverage if tests trigger panic #175

Closed bauersimon closed 2 weeks ago

bauersimon commented 2 weeks ago
package light

func typeArrayAccess(x []int) int {
    if x[0] == 123 {
        return x[0]
    }

    return 3
}
///////////////////////////////////////////////////////////////////////////////
package light

import "testing"

func TestTypeArrayAccess(t *testing.T) {
    tests := []struct {
        name     string
        input    []int
        expected int
    }{
        {"Test with a value", []int{123}, 123},
        {"Test with another value", []int{456}, 3},
        {"Test with an empty slice", []int{}, 3},
        {"Test with nil", nil, 3},
    }

    for _, test := range tests {
        t.Run(test.name, func(t *testing.T) {
            result := typeArrayAccess(test.input)
            if result != test.expected {
                t.Errorf("Expected %d, got %d", test.expected, result)
            }
        })
    }
}

coverage

Executes tests with 0 coverage objects
bauersimon commented 2 weeks ago

As discussed, there is no way around that. If a model triggers a panic without recover, there is nothing we can do because go test will exit 1 without writing coverage.