search

apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.05k stars 167 forks source link

[BUG] make test panic with the error `fatal error: concurrent map read and map write` #2713

Closed iziang closed 1 year ago

iziang commented 1 year ago

make test panic with the following error

/Users/ziang/git/kubeblocks/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./apis/...;./externalapis/..."
gofmt -l -w -s $(git ls-files --exclude-standard | grep "\.go$")
GOOS=linux go vet -mod=mod ./...
/Users/ziang/go/bin/goimports -local github.com/apecloud/kubeblocks -w $(git ls-files|grep "\.go$")
golangci-lint is already installed
/Users/ziang/go/bin/staticcheck ./...
/Users/ziang/go/bin/golangci-lint run ./...
go generate -x ./internal/testutil/k8s/mocks/...
go run github.com/golang/mock/mockgen -copyright_file ../../../../hack/boilerplate.go.txt -package mocks -destination k8sclient_mocks.go sigs.k8s.io/controller-runtime/pkg/client Client
go generate -x ./internal/configuration/container/mocks/...
go run github.com/golang/mock/mockgen -copyright_file ../../../../hack/boilerplate.go.txt -package mocks -destination dockerclient_mocks.go github.com/docker/docker/client ContainerAPIClient
go run github.com/golang/mock/mockgen -copyright_file ../../../../hack/boilerplate.go.txt -package mocks -destination criclient_mocks.go k8s.io/cri-api/pkg/apis/runtime/v1 RuntimeServiceClient
go generate -x ./internal/configuration/proto/mocks/...
go run github.com/golang/mock/mockgen -copyright_file ../../../../hack/boilerplate.go.txt -package mocks -destination reconfigure_client_mocks.go github.com/apecloud/kubeblocks/internal/configuration/proto ReconfigureClient
/Users/ziang/git/kubeblocks/bin/controller-gen rbac:roleName=manager-role crd:generateEmbeddedObjectMeta=true webhook paths="./cmd/manager/...;./apis/...;./controllers/...;./internal/..." output:crd:artifacts:config=config/crd/bases
KUBEBUILDER_ASSETS="/Users/ziang/Library/Application Support/io.kubebuilder.envtest/k8s/1.25.0-darwin-arm64" go test -short -coverprofile cover.out ./internal/... ./apis/... ./controllers/... ./cmd/...
ok      github.com/apecloud/kubeblocks/internal/class   0.539s  coverage: 51.9% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/cloudprovider   1.055s  coverage: 40.9% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/cluster 1.271s  coverage: 62.8% of statements
?       github.com/apecloud/kubeblocks/internal/cli/cmd [no test files]
ok      github.com/apecloud/kubeblocks/internal/cli/cmd/accounts    2.695s  coverage: 51.9% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/cmd/addon   1.954s  coverage: 9.8% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/cmd/alert   2.244s  coverage: 74.2% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/cmd/bench   0.233s  coverage: 70.4% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/cmd/class   0.604s  coverage: 81.0% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/cmd/cluster 3.522s  coverage: 57.2% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/cmd/clusterdefinition   1.625s  coverage: 70.0% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/cmd/clusterversion  2.828s  coverage: 70.0% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/cmd/dashboard   2.762s  coverage: 66.0% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/cmd/kubeblocks  1.966s  coverage: 52.5% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/cmd/migration   0.636s  coverage: 34.8% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/cmd/options 2.218s  coverage: 100.0% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/cmd/playground  49.668s coverage: 27.2% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/cmd/version 0.733s  coverage: 52.4% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/create  0.701s  coverage: 68.4% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/delete  1.192s  coverage: 85.9% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/edit    0.766s  coverage: 100.0% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/exec    0.975s  coverage: 71.4% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/list    1.043s  coverage: 75.6% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/patch   0.628s  coverage: 28.0% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/printer 1.559s  coverage: 49.3% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/testing 1.791s  coverage: 62.3% of statements
?       github.com/apecloud/kubeblocks/internal/cli/types   [no test files]
?       github.com/apecloud/kubeblocks/internal/cli/types/migrationapi  [no test files]
ok      github.com/apecloud/kubeblocks/internal/cli/util    25.893s coverage: 51.6% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/util/helm   0.684s  coverage: 58.6% of statements
ok      github.com/apecloud/kubeblocks/internal/cli/util/prompt 0.866s  coverage: 100.0% of statements
ok      github.com/apecloud/kubeblocks/internal/configuration   2.134s  coverage: 82.4% of statements
ok      github.com/apecloud/kubeblocks/internal/configuration/config_manager    5.489s  coverage: 67.3% of statements
ok      github.com/apecloud/kubeblocks/internal/configuration/container 1.682s  coverage: 72.1% of statements
?       github.com/apecloud/kubeblocks/internal/configuration/container/mocks   [no test files]
?       github.com/apecloud/kubeblocks/internal/configuration/proto [no test files]
?       github.com/apecloud/kubeblocks/internal/configuration/proto/mocks   [no test files]
ok      github.com/apecloud/kubeblocks/internal/configuration/util  1.104s  coverage: 57.4% of statements
?       github.com/apecloud/kubeblocks/internal/constant    [no test files]
ok      github.com/apecloud/kubeblocks/internal/controller/builder  7.472s  coverage: 76.9% of statements
?       github.com/apecloud/kubeblocks/internal/controller/client   [no test files]
ok      github.com/apecloud/kubeblocks/internal/controller/component    11.034s coverage: 80.5% of statements
ok      github.com/apecloud/kubeblocks/internal/controller/graph    1.054s  coverage: 68.0% of statements
ok      github.com/apecloud/kubeblocks/internal/controller/lifecycle    0.426s  coverage: 1.4% of statements
ok      github.com/apecloud/kubeblocks/internal/controller/plan 10.980s coverage: 63.7% of statements
?       github.com/apecloud/kubeblocks/internal/controller/types    [no test files]
ok      github.com/apecloud/kubeblocks/internal/controllerutil  6.849s  coverage: 59.2% of statements
?       github.com/apecloud/kubeblocks/internal/generics    [no test files]
ok      github.com/apecloud/kubeblocks/internal/gotemplate  0.478s  coverage: 77.5% of statements
ok      github.com/apecloud/kubeblocks/internal/preflight   32.267s coverage: 78.4% of statements
ok      github.com/apecloud/kubeblocks/internal/preflight/analyzer  0.919s  coverage: 87.2% of statements
ok      github.com/apecloud/kubeblocks/internal/preflight/collector 0.613s  coverage: 79.1% of statements
?       github.com/apecloud/kubeblocks/internal/preflight/interactive   [no test files]
ok      github.com/apecloud/kubeblocks/internal/preflight/testing   0.337s  coverage: 92.3% of statements
ok      github.com/apecloud/kubeblocks/internal/preflight/util  0.471s  coverage: 100.0% of statements
ok      github.com/apecloud/kubeblocks/internal/sqlchannel  1.036s  coverage: 65.9% of statements
ok      github.com/apecloud/kubeblocks/internal/sqlchannel/engine   0.187s  coverage: 60.0% of statements
?       github.com/apecloud/kubeblocks/internal/testutil    [no test files]
?       github.com/apecloud/kubeblocks/internal/testutil/apps   [no test files]
?       github.com/apecloud/kubeblocks/internal/testutil/k8s    [no test files]
?       github.com/apecloud/kubeblocks/internal/testutil/k8s/mocks  [no test files]
ok      github.com/apecloud/kubeblocks/internal/unstructured    0.463s  coverage: 79.2% of statements
ok      github.com/apecloud/kubeblocks/internal/unstructured/redis  0.106s  coverage: 45.9% of statements
?       github.com/apecloud/kubeblocks/internal/webhook [no test files]
ok      github.com/apecloud/kubeblocks/apis/apps/v1alpha1   16.066s coverage: 28.7% of statements
ok      github.com/apecloud/kubeblocks/apis/dataprotection/v1alpha1 0.198s  coverage: 2.5% of statements
ok      github.com/apecloud/kubeblocks/apis/extensions/v1alpha1 0.757s  coverage: 23.4% of statements
ok      github.com/apecloud/kubeblocks/controllers/apps 46.536s coverage: 78.9% of statements
ok      github.com/apecloud/kubeblocks/controllers/apps/components  20.157s coverage: 67.6% of statements
ok      github.com/apecloud/kubeblocks/controllers/apps/components/consensus    12.912s coverage: 27.4% of statements
ok      github.com/apecloud/kubeblocks/controllers/apps/components/replication  16.584s coverage: 69.5% of statements
ok      github.com/apecloud/kubeblocks/controllers/apps/components/stateful 15.143s coverage: 75.0% of statements
ok      github.com/apecloud/kubeblocks/controllers/apps/components/stateless    11.872s coverage: 78.3% of statements
?       github.com/apecloud/kubeblocks/controllers/apps/components/types    [no test files]
ok      github.com/apecloud/kubeblocks/controllers/apps/components/util 14.957s coverage: 76.4% of statements
ok      github.com/apecloud/kubeblocks/controllers/apps/configuration   16.160s coverage: 73.7% of statements
ok      github.com/apecloud/kubeblocks/controllers/apps/operations  14.991s coverage: 82.4% of statements
ok      github.com/apecloud/kubeblocks/controllers/apps/operations/util 12.671s coverage: 84.4% of statements
Running Suite: Data Protection Controller Suite - /Users/ziang/git/kubeblocks/controllers/dataprotection
========================================================================================================
Random Seed: 1681874075

Will run 19 of 19 specs
config settings: map[clearresourcepollinginterval:1ms clearresourcetimeout:10s cm_namespace: consistentlyduration:3s consistentlypollinginterval:1ms eventuallypollinginterval:1ms eventuallytimeout:10s kill_container_signal:SIGKILL kubeblocks_tools_image:apecloud/kubeblocks:latest maxconcurrentreconciles_dataprotection:20]
•••••••••••fatal error: concurrent map read and map write

goroutine 927 [running]:
github.com/spf13/viper.(*Viper).searchMap(0x140000fd6c0?, 0x105b52d40?, {0x14000901d20, 0x1, 0x1})
    /Users/ziang/go/pkg/mod/github.com/spf13/viper@v1.15.0/viper.go:659 +0x48
github.com/spf13/viper.(*Viper).find(0x140000fd6c0, {0x14001811650, 0x18}, 0x1)
    /Users/ziang/go/pkg/mod/github.com/spf13/viper@v1.15.0/viper.go:1329 +0x810
github.com/spf13/viper.(*Viper).Get(0x140000fd6c0, {0x1056704b8?, 0x105678801?})
    /Users/ziang/go/pkg/mod/github.com/spf13/viper@v1.15.0/viper.go:892 +0x48
github.com/spf13/viper.(*Viper).GetString(0x1059cd32a?, {0x1056704b8?, 0x10567882c?})
    /Users/ziang/go/pkg/mod/github.com/spf13/viper@v1.15.0/viper.go:959 +0x20
github.com/spf13/viper.GetString(...)
    /Users/ziang/go/pkg/mod/github.com/spf13/viper@v1.15.0/viper.go:956
iziang commented 1 year ago

Viper is not concurrent safe, the related discussion can be found here: https://github.com/spf13/viper/issues/268.

iziang commented 1 year ago

https://github.com/spf13/viper#is-it-safe-to-concurrently-read-and-write-to-a-viper

iziang commented 1 year ago

we can enable the go data race detector to find race condition bugs like this. I have opened another improvement issue https://github.com/apecloud/kubeblocks/issues/2760.

iziang commented 1 year ago

can't reproduce it now, just close this issue, we can reopen it in the future if necessary.