ozontech / allure-go

Complete Allure provider in Go which doesn't overload the interface usage
https://t.me/allure_go_chat
Apache License 2.0
306 stars 33 forks source link

Parameterized test may drop some parameters silently #99

Open andrey-kuznetsov opened 1 month ago

andrey-kuznetsov commented 1 month ago

Describe the bug Suite runner uses literal parameter value to create test name: https://github.com/ozontech/allure-go/blob/master/pkg/framework/runner/suite_runner.go#L230-L234 Also, test name is truncated to 150 characters, and this truncated string is used as a key in "name->method" mapping. Once one have long parameter strings or structures with long common prefix, they surprisingly lose tests for some of the parameters.

To Reproduce

import (
    "github.com/ozontech/allure-go/pkg/framework/provider"
    "github.com/ozontech/allure-go/pkg/framework/suite"
    "github.com/stretchr/testify/assert"
    "strings"
    "testing"
)

type SomeSuite struct {
    suite.Suite
    ParamSomething []string
}

var count int

func (s *SomeSuite) BeforeAll(t provider.T) {
    count = 0
    commonPrefix := strings.Repeat("*", 150)
    s.ParamSomething = []string{commonPrefix + "a", commonPrefix + "b"}
}

func (s *SomeSuite) TableTestSomething(t provider.T, param string) {
    count++
}

func TestIt(t *testing.T) {
    count = 0
    suite.RunSuite(t, new(SomeSuite))
    assert.Equal(t, 2, count)
}

Additional context Name length limitation in order to fix unrelated TempDir issue looks fragile. Please consider UUID, parameter value hash or even require Stringer conformance from a parameter. I tried UUID -- it worked fine for me.