Grafana: Fix data races

What happened?

Found several data races.

Data race 1: In package github.com/grafana/grafana/pkg/infra/usagestats/service

``` Read at 0x00c00179c6f8 by goroutine 892: github.com/grafana/grafana/pkg/infra/usagestats/service.(*UsageStats).gatherMetrics.func1() /go/src/github.com/grafana/grafana/pkg/infra/usagestats/service/usage_stats.go:90 +0x214 github.com/grafana/grafana/pkg/infra/usagestats/service.(*UsageStats).gatherMetrics.gowrap2() /go/src/github.com/grafana/grafana/pkg/infra/usagestats/service/usage_stats.go:99 +0x41 Previous write at 0x00c00179c6f8 by goroutine 891: github.com/grafana/grafana/pkg/infra/usagestats/service.(*UsageStats).gatherMetrics.func1() /go/src/github.com/grafana/grafana/pkg/infra/usagestats/service/usage_stats.go:90 +0x229 github.com/grafana/grafana/pkg/infra/usagestats/service.(*UsageStats).gatherMetrics.gowrap2() /go/src/github.com/grafana/grafana/pkg/infra/usagestats/service/usage_stats.go:99 +0x41 ```

Data race 2: In package github.com/grafana/grafana/pkg/apiserver/endpoints/responsewriter

``` Write at 0x00c0000da250 by goroutine 11: github.com/grafana/grafana/pkg/apiserver/endpoints/responsewriter.(*ResponseAdapter).WriteHeader() /go/src/github.com/grafana/grafana/pkg/apiserver/endpoints/responsewriter/responsewriter.go:81 +0xb6 github.com/grafana/grafana/pkg/apiserver/endpoints/responsewriter_test.asyncHandler() /go/src/github.com/grafana/grafana/pkg/apiserver/endpoints/responsewriter/responsewriter_test.go:98 +0x3a net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/apiserver/endpoints/responsewriter_test.TestResponseAdapter.func2.WrapHandler.2.1() /go/src/github.com/grafana/grafana/pkg/apiserver/endpoints/responsewriter/responsewriter.go:25 +0x79 Previous read at 0x00c0000da250 by goroutine 10: github.com/grafana/grafana/pkg/apiserver/endpoints/responsewriter_test.TestResponseAdapter.func2() /go/src/github.com/grafana/grafana/pkg/apiserver/endpoints/responsewriter/responsewriter_test.go:67 +0x3d4 testing.tRunner() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/testing/testing.go:1689 +0x21e testing.(*T).Run.gowrap1() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/testing/testing.go:1742 +0x44 ```

Data race 3: In package github.com/grafana/grafana/pkg/apiserver/rest

``` Write at 0x000006abf250 by goroutine 41: k8s.io/apimachinery/pkg/apis/meta/v1.(*ObjectMeta).SetLabels() /go/pkg/mod/k8s.io/apimachinery@v0.29.3/pkg/apis/meta/v1/meta.go:164 +0x48 k8s.io/apiserver/pkg/apis/example.(*Pod).SetLabels() :1 +0x1f github.com/grafana/grafana/pkg/apiserver/rest.enrichLegacyObject() /go/src/github.com/grafana/grafana/pkg/apiserver/rest/dualwriter_mode2.go:360 +0x18a github.com/grafana/grafana/pkg/apiserver/rest.(*DualWriterMode1).Create.func1() /go/src/github.com/grafana/grafana/pkg/apiserver/rest/dualwriter_mode1.go:54 +0x164 Previous read at 0x000006abf250 by goroutine 40: reflect.typedmemmove() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/mbarrier.go:203 +0x0 reflect.packEface() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/reflect/value.go:135 +0xc5 reflect.valueInterface() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/reflect/value.go:1526 +0x179 reflect.Value.Interface() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/reflect/value.go:1496 +0xb4 fmt.(*pp).printValue() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/fmt/print.go:769 +0xc5 fmt.(*pp).printValue() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/fmt/print.go:853 +0x1d3e fmt.(*pp).printValue() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/fmt/print.go:921 +0x132a fmt.(*pp).printArg() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/fmt/print.go:759 +0xb84 fmt.(*pp).doPrintf() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/fmt/print.go:1174 +0x10ce fmt.Sprintf() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/fmt/print.go:239 +0x5c github.com/stretchr/testify/mock.Arguments.Diff() /go/pkg/mod/github.com/stretchr/testify@v1.9.0/mock/mock.go:947 +0x2f5 fmt.Sprintf() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/fmt/print.go:239 +0x5c github.com/stretchr/testify/mock.Arguments.Diff() /go/pkg/mod/github.com/stretchr/testify@v1.9.0/mock/mock.go:939 +0x1b2 github.com/stretchr/testify/mock.(*Mock).methodWasCalled() /go/pkg/mod/github.com/stretchr/testify@v1.9.0/mock/mock.go:732 +0x1d7 github.com/stretchr/testify/mock.(*Mock).AssertNotCalled() /go/pkg/mod/github.com/stretchr/testify@v1.9.0/mock/mock.go:682 +0x15c github.com/grafana/grafana/pkg/apiserver/rest.TestMode1_Create.func4() /go/src/github.com/grafana/grafana/pkg/apiserver/rest/dualwriter_mode1_test.go:78 +0x564 github.com/grafana/grafana/pkg/apiserver/rest.legacyStoreMock.Create() /go/src/github.com/grafana/grafana/pkg/apiserver/rest/storage_mocks_test.go:33 +0x224 github.com/grafana/grafana/pkg/apiserver/rest.(*legacyStoreMock).Create() :1 +0xba github.com/grafana/grafana/pkg/apiserver/rest.(*DualWriterMode1).Create() /go/src/github.com/grafana/grafana/pkg/apiserver/rest/dualwriter_mode1.go:44 +0x364 github.com/grafana/grafana/pkg/apiserver/rest.TestMode1_Create.func4() /go/src/github.com/grafana/grafana/pkg/apiserver/rest/dualwriter_mode1_test.go:71 +0x2e7 testing.tRunner() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/testing/testing.go:1689 +0x21e testing.(*T).Run.gowrap1() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/testing/testing.go:1742 +0x44 ```

Data race 4: Found at runtime, in package github.com/grafana/grafana/pkg/web

``` Write at 0x00c001fac9a0 by goroutine 790: github.com/grafana/grafana/pkg/web.Renderer.func1.1() /go/src/github.com/grafana/grafana/pkg/web/render.go:39 +0x446 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.mwFromHandler.func1.1() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:137 +0x319 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.mwFromHandler.func1.1() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:137 +0x319 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.mwFromHandler.func1.1() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:137 +0x319 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.mwFromHandler.func1.1() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:137 +0x319 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/middleware/csrf.(*CSRF).Middleware.func1.1() /go/src/github.com/grafana/grafana/pkg/middleware/csrf/csrf.go:66 +0x395 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/api.(*HTTPServer).addMiddlewaresAndStaticRoutes.Recovery.func4.1() /go/src/github.com/grafana/grafana/pkg/middleware/recovery.go:180 +0x17b net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/middleware/loggermw.(*loggerImpl).Middleware.func1.1() /go/src/github.com/grafana/grafana/pkg/middleware/loggermw/logger.go:72 +0x834 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/middleware.RequestMetrics.func1.1() /go/src/github.com/grafana/grafana/pkg/middleware/request_metrics.go:75 +0x332 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/api.(*HTTPServer).addMiddlewaresAndStaticRoutes.RequestTracing.func2.1() /go/src/github.com/grafana/grafana/pkg/middleware/request_tracing.go:88 +0xf0c net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/api.(*HTTPServer).addMiddlewaresAndStaticRoutes.SetupRequestMetadata.func1.1() /go/src/github.com/grafana/grafana/pkg/middleware/requestmeta/request_metadata.go:66 +0x1da net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.(*Context).run() /go/src/github.com/grafana/grafana/pkg/web/context.go:52 +0x1cb github.com/grafana/grafana/pkg/web.(*Router).Handle.func1() /go/src/github.com/grafana/grafana/pkg/web/router.go:155 +0x21d github.com/grafana/grafana/pkg/web.(*Router).ServeHTTP() /go/src/github.com/grafana/grafana/pkg/web/router.go:211 +0x396 github.com/grafana/grafana/pkg/web.(*Macaron).ServeHTTP() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:163 +0x1a4 net/http.serverHandler.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:3137 +0x2a1 net/http.(*conn).serve() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2039 +0x13c4 net/http.(*Server).Serve.gowrap3() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:3285 +0x4f Previous write at 0x00c001fac9a0 by goroutine 1481: github.com/grafana/grafana/pkg/web.Renderer.func1.1() /go/src/github.com/grafana/grafana/pkg/web/render.go:39 +0x446 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.mwFromHandler.func1.1() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:137 +0x319 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.mwFromHandler.func1.1() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:137 +0x319 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.mwFromHandler.func1.1() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:137 +0x319 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.mwFromHandler.func1.1() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:137 +0x319 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/middleware/csrf.(*CSRF).Middleware.func1.1() /go/src/github.com/grafana/grafana/pkg/middleware/csrf/csrf.go:66 +0x395 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/api.(*HTTPServer).addMiddlewaresAndStaticRoutes.Recovery.func4.1() /go/src/github.com/grafana/grafana/pkg/middleware/recovery.go:180 +0x17b net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/middleware/loggermw.(*loggerImpl).Middleware.func1.1() /go/src/github.com/grafana/grafana/pkg/middleware/loggermw/logger.go:72 +0x834 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/middleware.RequestMetrics.func1.1() /go/src/github.com/grafana/grafana/pkg/middleware/request_metrics.go:75 +0x332 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/api.(*HTTPServer).addMiddlewaresAndStaticRoutes.RequestTracing.func2.1() /go/src/github.com/grafana/grafana/pkg/middleware/request_tracing.go:88 +0xf0c net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/api.(*HTTPServer).addMiddlewaresAndStaticRoutes.SetupRequestMetadata.func1.1() /go/src/github.com/grafana/grafana/pkg/middleware/requestmeta/request_metadata.go:66 +0x1da net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.(*Context).run() /go/src/github.com/grafana/grafana/pkg/web/context.go:52 +0x1cb github.com/grafana/grafana/pkg/web.(*Router).Handle.func1() /go/src/github.com/grafana/grafana/pkg/web/router.go:155 +0x21d github.com/grafana/grafana/pkg/web.(*Router).ServeHTTP() /go/src/github.com/grafana/grafana/pkg/web/router.go:221 +0x34c github.com/grafana/grafana/pkg/web.(*Macaron).ServeHTTP() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:163 +0x1a4 net/http.serverHandler.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:3137 +0x2a1 net/http.(*conn).serve() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2039 +0x13c4 net/http.(*Server).Serve.gowrap3() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:3285 +0x4f ```

Data race 5: In tests of package github.com/grafana/grafana/pkg/services/apiserver/storage/entity/test

``` Write at 0x00c001d72000 by goroutine 190: github.com/grafana/grafana/pkg/services/store/entity/sqlstash.(*broadcaster[go.shape.*uint8]).stream() /go/src/github.com/grafana/grafana/pkg/services/store/entity/sqlstash/broadcaster.go:88 +0x684 github.com/grafana/grafana/pkg/services/store/entity/sqlstash.(*broadcaster[go.shape.*uint8]).start.gowrap1() /go/src/github.com/grafana/grafana/pkg/services/store/entity/sqlstash/broadcaster.go:72 +0x4f Previous write at 0x00c001d72000 by goroutine 187: github.com/grafana/grafana/pkg/services/store/entity/sqlstash.(*broadcaster[go.shape.*uint8]).start() /go/src/github.com/grafana/grafana/pkg/services/store/entity/sqlstash/broadcaster.go:74 +0x424 github.com/grafana/grafana/pkg/services/store/entity/sqlstash.NewBroadcaster[go.shape.*uint8]() /go/src/github.com/grafana/grafana/pkg/services/store/entity/sqlstash/broadcaster.go:17 +0xa9 github.com/grafana/grafana/pkg/services/store/entity/sqlstash.(*sqlEntityServer).init() /go/src/github.com/grafana/grafana/pkg/services/store/entity/sqlstash/sql_storage_server.go:142 +0x5ec github.com/grafana/grafana/pkg/services/store/entity/sqlstash.(*sqlEntityServer).Init.func1() /go/src/github.com/grafana/grafana/pkg/services/store/entity/sqlstash/sql_storage_server.go:90 +0x2e sync.(*Once).doSlow() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/sync/once.go:74 +0xf0 sync.(*Once).Do() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/sync/once.go:65 +0x44 github.com/grafana/grafana/pkg/services/store/entity/sqlstash.(*sqlEntityServer).Init() /go/src/github.com/grafana/grafana/pkg/services/store/entity/sqlstash/sql_storage_server.go:89 +0x67 github.com/grafana/grafana/pkg/services/store/entity/sqlstash.(*sqlEntityServer).Watch() /go/src/github.com/grafana/grafana/pkg/services/store/entity/sqlstash/sql_storage_server.go:800 +0x164 github.com/grafana/grafana/pkg/services/store/entity._EntityStore_Watch_Handler() /go/src/github.com/grafana/grafana/pkg/services/store/entity/entity_grpc.pb.go:311 +0xe5 github.com/grafana/grafana/pkg/services/store/entity.NewEntityStoreClientLocal.StreamServerInterceptor.func2() /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/auth/auth.go:68 +0x1a1 github.com/fullstorydev/grpchan.InterceptServer.func2() /go/pkg/mod/github.com/fullstorydev/grpchan@v1.1.1/intercept.go:151 +0x7c github.com/fullstorydev/grpchan/inprocgrpc.(*Channel).NewStream.func1() /go/pkg/mod/github.com/fullstorydev/grpchan@v1.1.1/inprocgrpc/in_process.go:370 +0x60b ```

Data race 6: In tests of package github.com/grafana/grafana/pkg/tests/api/alerting

``` Write at 0x00c013abf5d8 by goroutine 43903: github.com/grafana/grafana/pkg/expr.(*DataPipeline).execute() /go/src/github.com/grafana/grafana/pkg/expr/graph.go:80 +0x34b github.com/grafana/grafana/pkg/expr.(*Service).ExecutePipeline() /go/src/github.com/grafana/grafana/pkg/expr/service.go:111 +0x20a github.com/grafana/grafana/pkg/services/ngalert/eval.(*conditionEvaluator).EvaluateRaw() /go/src/github.com/grafana/grafana/pkg/services/ngalert/eval/eval.go:77 +0x46b github.com/grafana/grafana/pkg/services/ngalert/eval.(*conditionEvaluator).Evaluate() /go/src/github.com/grafana/grafana/pkg/services/ngalert/eval/eval.go:82 +0x87 github.com/grafana/grafana/pkg/services/ngalert/schedule.(*alertRule).evaluate() /go/src/github.com/grafana/grafana/pkg/services/ngalert/schedule/alert_rule.go:354 +0x116e github.com/grafana/grafana/pkg/services/ngalert/schedule.(*alertRule).Run.func1() /go/src/github.com/grafana/grafana/pkg/services/ngalert/schedule/alert_rule.go:299 +0x1269 github.com/grafana/grafana/pkg/services/ngalert/schedule.(*alertRule).Run() /go/src/github.com/grafana/grafana/pkg/services/ngalert/schedule/alert_rule.go:316 +0x41c github.com/grafana/grafana/pkg/services/ngalert/schedule.(*schedule).processTick.func1() /go/src/github.com/grafana/grafana/pkg/services/ngalert/schedule/schedule.go:281 +0x6a golang.org/x/sync/errgroup.(*Group).Go.func1() /go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x91 Previous write at 0x00c013abf5d8 by goroutine 43910: github.com/grafana/grafana/pkg/expr.(*DataPipeline).execute() /go/src/github.com/grafana/grafana/pkg/expr/graph.go:80 +0x34b github.com/grafana/grafana/pkg/expr.(*Service).ExecutePipeline() /go/src/github.com/grafana/grafana/pkg/expr/service.go:111 +0x20a github.com/grafana/grafana/pkg/services/ngalert/eval.(*conditionEvaluator).EvaluateRaw() /go/src/github.com/grafana/grafana/pkg/services/ngalert/eval/eval.go:77 +0x46b github.com/grafana/grafana/pkg/services/ngalert/eval.(*conditionEvaluator).Evaluate() /go/src/github.com/grafana/grafana/pkg/services/ngalert/eval/eval.go:82 +0x87 github.com/grafana/grafana/pkg/services/ngalert/schedule.(*alertRule).evaluate() /go/src/github.com/grafana/grafana/pkg/services/ngalert/schedule/alert_rule.go:354 +0x116e github.com/grafana/grafana/pkg/services/ngalert/schedule.(*alertRule).Run.func1() /go/src/github.com/grafana/grafana/pkg/services/ngalert/schedule/alert_rule.go:299 +0x1269 github.com/grafana/grafana/pkg/services/ngalert/schedule.(*alertRule).Run() /go/src/github.com/grafana/grafana/pkg/services/ngalert/schedule/alert_rule.go:316 +0x41c github.com/grafana/grafana/pkg/services/ngalert/schedule.(*schedule).processTick.func1() /go/src/github.com/grafana/grafana/pkg/services/ngalert/schedule/schedule.go:281 +0x6a golang.org/x/sync/errgroup.(*Group).Go.func1() /go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x91 ```

Data race 7: In tests of package github.com/grafana/grafana/pkg/tests/api/plugins/backendplugin

``` Write at 0x000010512960 by goroutine 141: github.com/grafana/grafana/pkg/infra/log.RegisterContextualLogProvider() /go/src/github.com/grafana/grafana/pkg/infra/log/log.go:292 +0xe9 github.com/grafana/grafana/pkg/infra/tracing.ProvideService() /go/src/github.com/grafana/grafana/pkg/infra/tracing/tracing.go:83 +0x3d github.com/grafana/grafana/pkg/server.InitializeForTest() /go/src/github.com/grafana/grafana/pkg/server/wire_gen.go:667 +0x232 github.com/grafana/grafana/pkg/tests/testinfra.StartGrafanaEnv() /go/src/github.com/grafana/grafana/pkg/tests/testinfra/testinfra.go:54 +0x234 github.com/grafana/grafana/pkg/tests/api/plugins/backendplugin.newTestScenario() /go/src/github.com/grafana/grafana/pkg/tests/api/plugins/backendplugin/backendplugin_test.go:477 +0x144 github.com/grafana/grafana/pkg/tests/api/plugins/backendplugin.TestIntegrationBackendPlugins() /go/src/github.com/grafana/grafana/pkg/tests/api/plugins/backendplugin/backendplugin_test.go:100 +0x9a4 testing.tRunner() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/testing/testing.go:1689 +0x21e testing.(*T).Run.gowrap1() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/testing/testing.go:1742 +0x44 Previous read at 0x000010512960 by goroutine 1700: github.com/grafana/grafana/pkg/infra/log.FromContext() /go/src/github.com/grafana/grafana/pkg/infra/log/log.go:217 +0x34 github.com/grafana/grafana/pkg/infra/log/slogadapter.(*slogHandler).Handle() /go/src/github.com/grafana/grafana/pkg/infra/log/slogadapter/adapter.go:42 +0x159 github.com/go-logr/logr.(*slogSink).log() /go/pkg/mod/github.com/go-logr/logr@v1.4.1/slogsink.go:94 +0x442 github.com/go-logr/logr.(*slogSink).Info() /go/pkg/mod/github.com/go-logr/logr@v1.4.1/slogsink.go:74 +0x79 github.com/go-logr/logr.Logger.Info() /go/pkg/mod/github.com/go-logr/logr@v1.4.1/logr.go:280 +0x107 k8s.io/klog/v2.(*loggingT).output() /go/pkg/mod/k8s.io/klog/v2@v2.120.1/klog.go:914 +0x3f6 k8s.io/klog/v2.(*loggingT).printWithInfos() /go/pkg/mod/k8s.io/klog/v2@v2.120.1/klog.go:740 +0x31c k8s.io/klog/v2.(*loggingT).printDepth() /go/pkg/mod/k8s.io/klog/v2@v2.120.1/klog.go:722 +0xe9 k8s.io/klog/v2.(*loggingT).print() /go/pkg/mod/k8s.io/klog/v2@v2.120.1/klog.go:713 +0xff1 k8s.io/klog/v2.Verbose.Info() /go/pkg/mod/k8s.io/klog/v2@v2.120.1/klog.go:1417 +0xfae k8s.io/apiserver/pkg/server.preparedGenericAPIServer.Run() /go/pkg/mod/k8s.io/apiserver@v0.29.2/pkg/server/genericapiserver.go:671 +0xf55 github.com/grafana/grafana/pkg/services/apiserver.(*service).startCoreServer.func1() /go/src/github.com/grafana/grafana/pkg/services/apiserver/service.go:367 +0x5c ```

Data race 8: In tests of package github.com/grafana/grafana/pkg/tests/apis/playlist

There seems to be a race trying to mangle HTTP headers here. This appears to generate varying traces and may be confusing. ``` Read at 0x00c003e78720 by goroutine 7500: runtime.mapiterinit() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/map.go:816 +0x0 k8s.io/client-go/transport.(*debuggingRoundTripper).RoundTrip() /go/pkg/mod/k8s.io/client-go@v0.29.3/transport/round_trippers.go:578 +0x24c8 k8s.io/client-go/transport.(*userAgentRoundTripper).RoundTrip() /go/pkg/mod/k8s.io/client-go@v0.29.3/transport/round_trippers.go:168 +0x4f6 net/http.send() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/client.go:259 +0x9a6 net/http.(*Client).send() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/client.go:180 +0x14e net/http.(*Client).do() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/client.go:724 +0x11c8 net/http.(*Client).Do() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/client.go:590 +0x6a6 k8s.io/client-go/rest.(*Request).request() /go/pkg/mod/k8s.io/client-go@v0.29.3/rest/request.go:1023 +0x3ac k8s.io/client-go/rest.(*Request).Do() /go/pkg/mod/k8s.io/client-go@v0.29.3/rest/request.go:1063 +0x119 k8s.io/client-go/dynamic.(*dynamicResourceClient).Create() /go/pkg/mod/k8s.io/client-go@v0.29.3/dynamic/simple.go:138 +0xabb github.com/grafana/grafana/pkg/api.(*playlistK8sHandler).createPlaylist() /go/src/github.com/grafana/grafana/pkg/api/playlist.go:439 +0x2d7 github.com/grafana/grafana/pkg/api.(*playlistK8sHandler).createPlaylist-fm() :1 +0x3d github.com/grafana/grafana/pkg/api/response.wrap_handler.func3() /go/src/github.com/grafana/grafana/pkg/api/response/web_hack.go:40 +0x4e net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x1f1 github.com/grafana/grafana/pkg/web.mwFromHandler.func1.1() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:131 +0x1bd net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.mwFromHandler.func1.1() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:137 +0x319 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.mwFromHandler.func1.1() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:137 +0x319 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/tests/testinfra.StartGrafanaEnv.func1.1() /go/src/github.com/grafana/grafana/pkg/tests/testinfra/testinfra.go:70 +0xd0 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.mwFromHandler.func1.1() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:137 +0x319 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.mwFromHandler.func1.1() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:137 +0x319 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/services/contexthandler.(*ContextHandler).Middleware-fm.(*ContextHandler).Middleware.func1() /go/src/github.com/grafana/grafana/pkg/services/contexthandler/contexthandler.go:141 +0x14f4 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.mwFromHandler.func1.1() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:137 +0x319 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.mwFromHandler.func1.1() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:137 +0x319 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.mwFromHandler.func1.1() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:137 +0x319 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.mwFromHandler.func1.1() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:137 +0x319 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.mwFromHandler.func1.1() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:137 +0x319 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.Renderer.func1.1() /go/src/github.com/grafana/grafana/pkg/web/render.go:44 +0x583 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.mwFromHandler.func1.1() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:137 +0x319 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.mwFromHandler.func1.1() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:137 +0x319 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.mwFromHandler.func1.1() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:137 +0x319 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.mwFromHandler.func1.1() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:137 +0x319 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/middleware/csrf.(*CSRF).Middleware.func1.1() /go/src/github.com/grafana/grafana/pkg/middleware/csrf/csrf.go:66 +0x395 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/api.(*HTTPServer).addMiddlewaresAndStaticRoutes.Recovery.func4.1() /go/src/github.com/grafana/grafana/pkg/middleware/recovery.go:180 +0x17b net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/middleware/loggermw.(*loggerImpl).Middleware.func1.1() /go/src/github.com/grafana/grafana/pkg/middleware/loggermw/logger.go:72 +0x834 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/middleware.RequestMetrics.func1.1() /go/src/github.com/grafana/grafana/pkg/middleware/request_metrics.go:75 +0x332 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/api.(*HTTPServer).addMiddlewaresAndStaticRoutes.RequestTracing.func2.1() /go/src/github.com/grafana/grafana/pkg/middleware/request_tracing.go:88 +0xf0c net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/api.(*HTTPServer).addMiddlewaresAndStaticRoutes.SetupRequestMetadata.func1.1() /go/src/github.com/grafana/grafana/pkg/middleware/requestmeta/request_metadata.go:66 +0x1da net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/web.(*Context).run() /go/src/github.com/grafana/grafana/pkg/web/context.go:52 +0x1cb github.com/grafana/grafana/pkg/web.(*Router).Handle.func1() /go/src/github.com/grafana/grafana/pkg/web/router.go:155 +0x21d github.com/grafana/grafana/pkg/web.(*Router).ServeHTTP() /go/src/github.com/grafana/grafana/pkg/web/router.go:221 +0x34c github.com/grafana/grafana/pkg/web.(*Macaron).ServeHTTP() /go/src/github.com/grafana/grafana/pkg/web/macaron.go:163 +0x1a4 net/http.serverHandler.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:3137 +0x2a1 net/http.(*conn).serve() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2039 +0x13c4 net/http.(*Server).Serve.gowrap3() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:3285 +0x4f Previous write at 0x00c003e78720 by goroutine 7648: runtime.mapassign_faststr() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/map_faststr.go:203 +0x0 net/textproto.MIMEHeader.Set() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/textproto/header.go:22 +0xb4 net/http.Header.Set() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/header.go:40 +0x35 k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.WithAuditInit.withAuditInit.func33() /go/pkg/mod/k8s.io/apiserver@v0.29.2/pkg/endpoints/filters/audit_init.go:60 +0x12c net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/apiserver/builder.SetupConfig.func2.WithAcceptHeader.2() /go/src/github.com/grafana/grafana/pkg/apiserver/endpoints/filters/accept.go:13 +0xc4 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 github.com/grafana/grafana/pkg/apiserver/builder.SetupConfig.func2.WithPathRewriters.3() /go/src/github.com/grafana/grafana/pkg/apiserver/endpoints/filters/path_rewriter.go:29 +0x1b9 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 k8s.io/component-base/tracing.WithTracing.func1() /go/pkg/mod/k8s.io/component-base@v0.0.0-20240417101527-62c04b35eff6/tracing/utils.go:105 +0x28c net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP() /go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.51.0/handler.go:212 +0x18bb go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1() /go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.51.0/handler.go:73 +0x67 net/http.HandlerFunc.ServeHTTP() /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/http/server.go:2166 +0x47 k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP() /go/pkg/mod/k8s.io/apiserver@v0.29.2/pkg/server/handler.go:189 +0x5b github.com/grafana/grafana/pkg/services/apiserver.(*service).GetDirectRestConfig.func1.WrapHandler.1.1() /go/src/github.com/grafana/grafana/pkg/apiserver/endpoints/responsewriter/responsewriter.go:25 +0x68 ```

What did you expect to happen?

To find none.

Did this work before?

No.

How do we reproduce it?

Run make enable-go-race (see https://github.com/grafana/grafana/pull/88380)
Run make run
Run make test
Run make test-go-integration

Is the bug inside a dashboard panel?

No.

Environment (with versions)?

All.

Grafana platform?

None

Datasource(s)?

No.

grafana / grafana