fatal error: concurrent map iteration and map write

[hansongChina]

What happened: fatal error: concurrent map iteration and map write

What you expected to happen: Running normally

How to reproduce it (as minimally and precisely as possible): Run 24 hours to find once, do not know how to reproduce Below is the detailed error message

/root/workspace/karmada/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:100 +0x1dd created by k8s.io/apiserver/pkg/server/filters.(timeoutHandler).ServeHTTP /root/go/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0654d7fe8 sp=0xc0654d7fe0 pc=0x46ad01 runtime.goexit() /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:114 +0x70 fp=0xc0654d7fe0 sp=0xc0654d7f90 pc=0x18ebad0 k8s.io/apiserver/pkg/server/filters.(timeoutHandler).ServeHTTP.func1() /root/go/go/src/net/http/server.go:2047 +0x2f fp=0xc0654d7f90 sp=0xc0654d7f68 pc=0x6aff6f net/http.HandlerFunc.ServeHTTP(0xc01a45ea80, {0x2796f98, 0xc064e9b4c0}, 0x8e56b2) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:89 +0x46b fp=0xc0654d7f68 sp=0xc0654d7ea0 pc=0x170a78b k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x2796f98, 0xc064e9b4c0}, 0xc064ec5200) /root/go/go/src/net/http/server.go:2047 +0x2f fp=0xc0654d7ea0 sp=0xc0654d7e78 pc=0x6aff6f net/http.HandlerFunc.ServeHTTP(0x279ebc0, {0x2796f98, 0xc064e9b4c0}, 0x2745430) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:80 +0x839 fp=0xc0654d7e78 sp=0xc0654d7c58 pc=0x1712d19 k8s.io/apiserver/pkg/endpoints/filters.withAuthentication.func1({0x2796f98, 0xc064e9b4c0}, 0xc064ec5400) /root/go/go/src/net/http/server.go:2047 +0x2f fp=0xc0654d7c58 sp=0xc0654d7c30 pc=0x6aff6f net/http.HandlerFunc.ServeHTTP(0x279ebf8, {0x2796f98, 0xc064e9b4c0}, 0x2509ea8) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:104 +0x1a5 fp=0xc0654d7c30 sp=0xc0654d7b70 pc=0x170aaa5 k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x2796f98, 0xc064e9b4c0}, 0xc064ec5400) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:98 +0x37 fp=0xc0654d7b70 sp=0xc0654d7b40 pc=0x170ab37 k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1·dwrap·1() /root/go/go/src/net/http/server.go:2047 +0x2f fp=0xc0654d7b40 sp=0xc0654d7b18 pc=0x6aff6f net/http.HandlerFunc.ServeHTTP(0xc064ecc630, {0x2796f98, 0xc064e9b4c0}, 0x99bb6a) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:80 +0x178 fp=0xc0654d7b18 sp=0xc0654d7a50 pc=0x170a498 k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x2796f98, 0xc064e9b4c0}, 0xc064ec5400) /root/go/go/src/net/http/server.go:2047 +0x2f fp=0xc0654d7a50 sp=0xc0654d7a28 pc=0x6aff6f net/http.HandlerFunc.ServeHTTP(0x3be3930, {0x2796f98, 0xc064e9b4c0}, 0xa) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:104 +0x1a5 fp=0xc0654d7a28 sp=0xc0654d7968 pc=0x170aaa5 k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x2796f98, 0xc064e9b4c0}, 0xc064ec5400) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:98 +0x37 fp=0xc0654d7968 sp=0xc0654d7938 pc=0x170ab37 k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1·dwrap·1() /root/go/go/src/net/http/server.go:2047 +0x2f fp=0xc0654d7938 sp=0xc0654d7910 pc=0x6aff6f net/http.HandlerFunc.ServeHTTP(0xc064ecc630, {0x2796f98, 0xc064e9b4c0}, 0x99bb6a) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:80 +0x178 fp=0xc0654d7910 sp=0xc0654d7848 pc=0x170a498 k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x2796f98, 0xc064e9b4c0}, 0xc064ec5400) /root/go/go/src/net/http/server.go:2047 +0x2f fp=0xc0654d7848 sp=0xc0654d7820 pc=0x6aff6f net/http.HandlerFunc.ServeHTTP(0x3be3930, {0x2796f98, 0xc064e9b4c0}, 0xa) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x21c fp=0xc0654d7820 sp=0xc0654d7190 pc=0x1714ddc k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1({0x2796f98, 0xc064e9b4c0}, 0xc064ec5400) /root/go/go/src/net/http/server.go:2047 +0x2f fp=0xc0654d7190 sp=0xc0654d7168 pc=0x6aff6f net/http.HandlerFunc.ServeHTTP(0x1, {0x2796f98, 0xc064e9b4c0}, 0xc0638d7220) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:104 +0x1a5 fp=0xc0654d7168 sp=0xc0654d70a8 pc=0x170aaa5 k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x2796f98, 0xc064e9b4c0}, 0xc064ec5400) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:98 +0x37 fp=0xc0654d70a8 sp=0xc0654d7078 pc=0x170ab37 k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1·dwrap·1() /root/go/go/src/net/http/server.go:2047 +0x2f fp=0xc0654d7078 sp=0xc0654d7050 pc=0x6aff6f net/http.HandlerFunc.ServeHTTP(0xc064ecc630, {0x2796f98, 0xc064e9b4c0}, 0x99bb6a) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:187 +0x26b fp=0xc0654d7050 sp=0xc0654d6fa8 pc=0x18e896b k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1({0x2796f98, 0xc064e9b4c0}, 0xc064ec5400) /root/go/go/src/net/http/server.go:2047 +0x2f fp=0xc0654d6fa8 sp=0xc0654d6f80 pc=0x6aff6f net/http.HandlerFunc.ServeHTTP(0x3b918e0, {0x2796f98, 0xc064e9b4c0}, 0x3) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:80 +0x178 fp=0xc0654d6f80 sp=0xc0654d6eb8 pc=0x170a498 k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x2796f98, 0xc064e9b4c0}, 0xc064ec5400) /root/go/go/src/net/http/server.go:2047 +0x2f fp=0xc0654d6eb8 sp=0xc0654d6e90 pc=0x6aff6f net/http.HandlerFunc.ServeHTTP(0x3be3930, {0x2796f98, 0xc064e9b4c0}, 0x3f647ae147ae147b) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x4d0 fp=0xc0654d6e90 sp=0xc0654d6d60 pc=0x1714590 k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1({0x2796f98, 0xc064e9b4c0}, 0xc064ec5400) /root/go/go/src/net/http/server.go:2047 +0x2f fp=0xc0654d6d60 sp=0xc0654d6d38 pc=0x6aff6f net/http.HandlerFunc.ServeHTTP(0x279ebf8, {0x2796f98, 0xc064e9b4c0}, 0x4) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:104 +0x1a5 fp=0xc0654d6d38 sp=0xc0654d6c78 pc=0x170aaa5 k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x2796f98, 0xc064e9b4c0}, 0xc064ec5400) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:98 +0x37 fp=0xc0654d6c78 sp=0xc0654d6c48 pc=0x170ab37 k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1·dwrap·1()

:1 +0x66 fp=0xc0654d6c48 sp=0xc0654d6c00 pc=0x19d7e66 k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc000427660, {0x2796f98, 0xc064e9b4c0}, 0x99bb6a) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/server/handler.go:146 +0x5c6 fp=0xc0654d6c00 sp=0xc0654d6ae8 pc=0x19cf886 k8s.io/apiserver/pkg/server.director.ServeHTTP({{0x23fc8d1, 0x170abdf}, 0xc000b1f710, 0xc00013ba40}, {0x2796f98, 0xc064e9b4c0}, 0xc064ec5400) /root/workspace/karmada/vendor/github.com/emicklei/go-restful/container.go:199 github.com/emicklei/go-restful.(*Container).Dispatch(...) /root/workspace/karmada/vendor/github.com/emicklei/go-restful/container.go:288 +0x9d1 fp=0xc0654d6ae8 sp=0xc0654d6968 pc=0x133c4d1 github.com/emicklei/go-restful.(*Container).dispatch(0xc000b1f710, {0x2796f98, 0xc064e9b4c0}, 0xc064ec5400) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:555 +0x223 fp=0xc0654d6968 sp=0xc0654d67f0 pc=0x16f29e3 k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc064ecc7b0, 0xc05bfba930) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1228 +0x6f fp=0xc0654d67f0 sp=0xc0654d67a0 pc=0x198f6af k8s.io/apiserver/pkg/endpoints.restfulConnectResource.func1(0xc064ecc7b0, 0xc05bfba930) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:220 +0xc7b fp=0xc0654d67a0 sp=0xc0654d6428 pc=0x19723bb k8s.io/apiserver/pkg/endpoints/handlers.ConnectResource.func1({0x2796f98, 0xc064e9b520}, 0xc064ec5400) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:499 +0xb18 fp=0xc0654d6428 sp=0xc0654d61b0 pc=0x16f1178 k8s.io/apiserver/pkg/endpoints/metrics.RecordLongRunning(0xc064ec5400, 0x1f99b60, {0x23f4cf0, 0x9}, 0xc060ac2728) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:226 +0x246 fp=0xc0654d61b0 sp=0xc0654d60b8 pc=0x1972a46 k8s.io/apiserver/pkg/endpoints/handlers.ConnectResource.func1.1() /root/go/go/src/net/http/server.go:2047 +0x2f fp=0xc0654d60b8 sp=0xc0654d6090 pc=0x6aff6f net/http.HandlerFunc.ServeHTTP(0xc00052e490, {0x2796f98, 0xc064e9b520}, 0xc0007ea7e0) /root/workspace/karmada/pkg/search/proxy/controller.go:168 +0x3e2 fp=0xc0654d6090 sp=0xc0654d5fb8 pc=0x1bfbba2 github.com/karmada-io/karmada/pkg/search/proxy.(*Controller).Connect.func1({0x2796f98, 0xc064e9b520}, 0x1) /root/go/go/src/net/http/server.go:2047 +0x2f fp=0xc0654d5fb8 sp=0xc0654d5f90 pc=0x6aff6f net/http.HandlerFunc.ServeHTTP(0x279ebf8, {0x2796f98, 0xc064e9b520}, 0x2745430) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/handlers/get.go:285 +0x12d6 fp=0xc0654d5f90 sp=0xc0654d5790 pc=0x1965956 k8s.io/apiserver/pkg/endpoints/handlers.ListResource.func1({0x2796f98, 0xc064e9b520}, 0xc064ec5700) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/handlers/response.go:150 +0x691 fp=0xc0654d5790 sp=0xc0654d5478 pc=0x196f431 k8s.io/apiserver/pkg/endpoints/handlers.transformResponseObject({0x279ebf8, 0xc064ecd8c0}, 0xc02eb60b60, 0x4, _, {_, _}, _, {0x0, {0x0, ...}, ...}, ...) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:275 +0x65f fp=0xc0654d5478 sp=0xc0654d5148 pc=0x1703c3f k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated({0x2796db8, 0xc064ed60c0}, {0x2795048, 0xc02eb60b60}, {{0x0, 0x23ff8aa}, {0xc049cce24c, 0x23ff8aa}}, {0x2796f98, 0xc064e9b520}, ...) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/request/webhook_duration.go:216 +0x5e fp=0xc0654d5148 sp=0xc0654d5120 pc=0x99d85e k8s.io/apiserver/pkg/endpoints/request.TrackSerializeResponseObjectLatency({0x279ebf8, 0xc064eccae0}, 0xc0647455c0) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/request/webhook_duration.go:75 +0x9c fp=0xc0654d5120 sp=0xc0654d50d0 pc=0x99cebc k8s.io/apiserver/pkg/endpoints/request.(*durationTracker).Track(0xc064ecc270, 0xc0647455c0) /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:276 +0x5f fp=0xc0654d50d0 sp=0xc0654d5070 pc=0x1703d1f k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated.func2() /root/workspace/karmada/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:107 +0x4a9 fp=0xc0654d5070 sp=0xc0654d4f58 pc=0x17026a9 k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.SerializeObject({0x23ff8aa, 0x10}, {0x27863f0, 0xc0654bd3e0}, {0x2796f98, 0xc064e9b520}, 0xc064ec5700, 0xc8, {0x27819b8, 0xc064e9b640}) :1 +0xbd fp=0xc0654d4f58 sp=0xc0654d4ec8 pc=0x89853d k8s.io/apimachinery/pkg/runtime.(*WithVersionEncoder).Encode(0xc0654bd3e0, {0x27819b8, 0xc064e9b640}, {0x275ab60, 0xc064fb3200}) /root/workspace/karmada/vendor/k8s.io/apimachinery/pkg/runtime/helper.go:240 +0x2ee fp=0xc0654d4ec8 sp=0xc0654d4d58 pc=0x88c4ee k8s.io/apimachinery/pkg/runtime.WithVersionEncoder.Encode({{0x2786490, 0xc064fcb440}, {0x2781b20, 0xc00019a050}, {0x2781a58, 0xc0003bc070}}, {0x27819b8, 0xc064e9b640}, {0x275ab60, 0xc064fb3200}) /root/workspace/karmada/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:220 +0xfc fp=0xc0654d4d58 sp=0xc0654d4cf8 pc=0x8a1b7c k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Encode(0xc00019a050, {0x27819b8, 0xc064e9b640}, {0x275ab60, 0xc064fb3200}) /root/workspace/karmada/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:246 +0x19a fp=0xc0654d4cf8 sp=0xc0654d4c68 pc=0x8a1d7a k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).doEncode(0x23ee314, {0x27819b8, 0xc064e9b640}, {0x275ab60, 0xc064fb3200}) /root/go/go/src/encoding/json/stream.go:206 +0x7e fp=0xc0654d4c68 sp=0xc0654d4be8 pc=0x6efb5e encoding/json.(*Encoder).Encode(0xc0654d4c98, {0x2377ec0, 0xc064e9b640}) /root/go/go/src/encoding/json/encode.go:332 +0xfa fp=0xc0654d4be8 sp=0xc0654d4b70 pc=0x6e3b1a encoding/json.(*encodeState).marshal(0x208ca20, {0x2377ec0, 0xc064e9b640}, {0x4, 0x0}) /root/go/go/src/encoding/json/encode.go:360 +0x78 fp=0xc0654d4b70 sp=0xc0654d4b10 pc=0x6e3ef8 encoding/json.(*encodeState).reflectValue(0xc060ac0ba8, {0x2377ec0, 0xc064e9b640, 0x2}, {0x20, 0x39}) /root/go/go/src/encoding/json/encode.go:479 +0xbe fp=0xc0654d4b10 sp=0xc0654d4a88 pc=0x6e485e encoding/json.marshalerEncoder(0xc05f604280, {0x2377ec0, 0xc064e9b640, 0x0}, {0x0, 0x0}) /root/workspace/karmada/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured/unstructured_list.go:128 +0x5e fp=0xc0654d4a88 sp=0xc0654d4a40 pc=0x97685e k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.(*UnstructuredList).MarshalJSON(0x2377ec0) :1 +0x4c fp=0xc0654d4a40 sp=0xc0654d4a10 pc=0x977aec k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.(*unstructuredJSONScheme).Encode(0x3be3500, {0x27819b8, 0xc064e9b640}, {0x2756380, 0xc0654bd440}) /root/workspace/karmada/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured/helpers.go:352 +0xbe fp=0xc0654d4a10 sp=0xc0654d49c0 pc=0x970e3e k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.unstructuredJSONScheme.Encode({}, {0x27819b8, 0xc064e9b640}, {0x2756380, 0xc0654bd440}) /root/workspace/karmada/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured/helpers.go:369 +0x498 fp=0xc0654d49c0 sp=0xc0654d47b8 pc=0x971318 k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.unstructuredJSONScheme.doEncode({}, {0x27819b8, 0xc064e9b640}, {0x2756380, 0xc0654bd440}) /root/go/go/src/encoding/json/stream.go:206 +0x7e fp=0xc0654d47b8 sp=0xc0654d4738 pc=0x6efb5e encoding/json.(*Encoder).Encode(0xc0654d48b0, {0x208ca20, 0xc0654bd470}) /root/go/go/src/encoding/json/encode.go:332 +0xfa fp=0xc0654d4738 sp=0xc0654d46c0 pc=0x6e3b1a encoding/json.(*encodeState).marshal(0x7f4c9a4ba008, {0x208ca20, 0xc0654bd470}, {0xc8, 0x3}) /root/go/go/src/encoding/json/encode.go:360 +0x78 fp=0xc0654d46c0 sp=0xc0654d4660 pc=0x6e3ef8 encoding/json.(*encodeState).reflectValue(0xc060ac06f8, {0x208ca20, 0xc0654bd470, 0x120}, {0x20, 0x39}) /root/go/go/src/encoding/json/encode.go:779 +0x45 fp=0xc0654d4660 sp=0xc0654d4620 pc=0x6f11c5 encoding/json.mapEncoder.encode-fm(0x208ca20, {0x208ca20, 0xc0654bd470, 0x203019}, {0x19, 0x30}) /root/go/go/src/encoding/json/encode.go:814 +0x5c3 fp=0xc0654d4620 sp=0xc0654d4460 pc=0x6e6bc3 encoding/json.mapEncoder.encode({0x0}, 0xc0628a6180, {0x208ca20, 0xc0654bd470, 0x208ca20}, {0x3, 0x0}) /root/go/go/src/encoding/json/encode.go:716 +0xc8 fp=0xc0654d4460 sp=0xc0654d4418 pc=0x6e5fa8 encoding/json.interfaceEncoder(0xc0628a6180, {0x204f300, 0xc0654b5700, 0x1}, {0xe9, 0x37}) /root/go/go/src/encoding/json/encode.go:360 +0x78 fp=0xc0654d4418 sp=0xc0654d43b8 pc=0x6e3ef8 encoding/json.(*encodeState).reflectValue(0x204f300, {0x1f2d4c0, 0xc05dd82528, 0xc0654bd4a0}, {0x40, 0x25}) /root/go/go/src/encoding/json/encode.go:869 +0x45 fp=0xc0654d43b8 sp=0xc0654d4378 pc=0x6f1285 encoding/json.sliceEncoder.encode-fm(0x1f2d4c0, {0x1f2d4c0, 0xc05dd82528, 0x4d1389}, {0x4, 0x0}) /root/go/go/src/encoding/json/encode.go:889 +0x2e6 fp=0xc0654d4378 sp=0xc0654d4270 pc=0x6e77c6 encoding/json.sliceEncoder.encode({0x1f9af20}, 0xc0628a6180, {0x1f2d4c0, 0xc05dd82528, 0x1f2d4c0}, {0x0, 0x0}) /root/go/go/src/encoding/json/encode.go:909 +0x45 fp=0xc0654d4270 sp=0xc0654d4230 pc=0x6f1345 encoding/json.arrayEncoder.encode-fm(0x5652455059482e64, {0x1f2d4c0, 0xc05dd82528, 0x40e650}, {0x28, 0x0}) /root/go/go/src/encoding/json/encode.go:916 +0xb5 fp=0xc0654d4230 sp=0xc0654d41c0 pc=0x6e7c95 encoding/json.arrayEncoder.encode({0x0}, 0xc0628a6180, {0x1f2d4c0, 0xc05dd82528, 0x4035b2}, {0x80, 0x42}) /root/go/go/src/encoding/json/encode.go:716 +0xc8 fp=0xc0654d41c0 sp=0xc0654d4178 pc=0x6e5fa8 encoding/json.interfaceEncoder(0xc0628a6180, {0x204f300, 0xc0654c8cf0, 0x4037e9}, {0xa0, 0xd4}) /root/go/go/src/encoding/json/encode.go:360 +0x78 fp=0xc0654d4178 sp=0xc0654d4118 pc=0x6e3ef8 encoding/json.(*encodeState).reflectValue(0x204f300, {0x208ca20, 0xc001dcb4d0, 0x5856412e64697570}, {0x35, 0x31}) /root/go/go/src/encoding/json/encode.go:779 +0x45 fp=0xc0654d4118 sp=0xc0654d40d8 pc=0x6f11c5 encoding/json.mapEncoder.encode-fm(0x208ca20, {0x208ca20, 0xc001dcb4d0, 0x4d1389}, {0x22, 0x66}) /root/go/go/src/encoding/json/encode.go:814 +0x5c3 fp=0xc0654d40d8 sp=0xc0654d3f18 pc=0x6e6bc3 encoding/json.mapEncoder.encode({0x6f692e736574656e}, 0xc0628a6180, {0x208ca20, 0xc001dcb4d0, 0x208ca20}, {0x65, 0x61}) /root/go/go/src/encoding/json/encode.go:716 +0xc8 fp=0xc0654d3f18 sp=0xc0654d3ed0 pc=0x6e5fa8 encoding/json.interfaceEncoder(0xc0628a6180, {0x204f300, 0xc06634aa20, 0x412e646975706301}, {0xe9, 0x37}) /root/go/go/src/encoding/json/encode.go:360 +0x78 fp=0xc0654d3ed0 sp=0xc0654d3e70 pc=0x6e3ef8 encoding/json.(*encodeState).reflectValue(0x204f300, {0x208ca20, 0xc001dcb5f0, 0xc066342750}, {0x0, 0x78}) /root/go/go/src/encoding/json/encode.go:779 +0x45 fp=0xc0654d3e70 sp=0xc0654d3e30 pc=0x6f11c5 encoding/json.mapEncoder.encode-fm(0x208ca20, {0x208ca20, 0xc001dcb5f0, 0x4d1389}, {0x5, 0x0}) /root/go/go/src/encoding/json/encode.go:799 +0x305 fp=0xc0654d3e30 sp=0xc0654d3c70 pc=0x6e6905 encoding/json.mapEncoder.encode({0x1f9af20}, 0xc0628a6180, {0x208ca20, 0xc001dcb5f0, 0x208ca20}, {0x0, 0x0}) /root/go/go/src/reflect/value.go:1618 +0x99 fp=0xc0654d3c70 sp=0xc0654d3c50 pc=0x4d30b9 reflect.(*MapIter).Next(0xc06633f560) /root/go/go/src/runtime/map.go:1346 +0x19 fp=0xc0654d3c50 sp=0xc0654d3c38 pc=0x464919 reflect.mapiternext(0x15) /root/go/go/src/runtime/map.go:858 +0x4eb fp=0xc0654d3c38 sp=0xc0654d3bc8 pc=0x40f98b runtime.mapiternext(0x14) /root/go/go/src/runtime/panic.go:1198 +0x71 fp=0xc0654d3bc8 sp=0xc0654d3b98 pc=0x436e91 fatal error: concurrent map iteration and map write **Anything else we need to know?**: karmada 1.3 version and The search module **Environment**: - Karmada version: 1.3 - kubectl-karmada or karmadactl version (the result of `kubectl-karmada version` or `karmadactl version`): - Others:

[XiShanYongYe-Chang]

Hi @ikaven1024, can you help take a look?

[ikaven1024]

/assign

[ikaven1024]

The root cause is response objects are shared in cache. https://github.com/karmada-io/karmada/pull/2483 will deep copy objects from cache, this may help. @hansongChina you can pick this PR and retry.

[XiShanYongYe-Chang]

@ikaven1024, maybe we need to cherry-pick #2483 to branch release1.3.

[ikaven1024]

@ikaven1024, maybe we need to cherry-pick #2483 to branch release1.3.

Agree, I will do it.

[XiShanYongYe-Chang]

Agree, I will do it.

Thanks! You can cherry-pick refer to https://karmada.io/docs/contributor/cherry-picks.

[hansongChina]

I've done that in the list get Watch method deep copy objects, Can't seem to solve the problem。 @ikaven1024

[XiShanYongYe-Chang]

I've done that in the list get Watch method deep copy objects, Can't seem to solve the problem。 @ikaven1024

Hi @hansongChina, do you mean you've cherry-picked the related code before you run into this problem?

[RainbowMango]

Karmada version: 1.3

Are you using v1.3.0?

[hansongChina]

Karmada version: 1.3

Are you using v1.3.0?

yes ！

[hansongChina]

I've done that in the list get Watch method deep copy objects, Can't seem to solve the problem。 @ikaven1024

Hi @hansongChina, do you mean you've cherry-picked the related code before you run into this problem?

We added，not cherry-picked the related code， not deep copy can cause an invalid resource version bug。

[ikaven1024]

not deep copy can cause an invalid resource version bug。

Do you mean updating object cause invalid resource version bug? This is fixed in https://github.com/karmada-io/karmada/pull/2469 https://github.com/karmada-io/karmada/blob/a7a944a1a5a7de07291363ead0f1a0015819c4bd/pkg/search/proxy/cluster_proxy.go#L116

[ikaven1024]

@hansongChina could you show your modifications?

[hansongChina]

not deep copy can cause an invalid resource version bug。

Do you mean updating object cause invalid resource version bug? This is fixed in #2469

https://github.com/karmada-io/karmada/blob/a7a944a1a5a7de07291363ead0f1a0015819c4bd/pkg/search/proxy/cluster_proxy.go#L116

yes This is also a bug ，but this not updating, get method， If you don't add deep copy ，Using the same pointer in cache and changing the annotation value will affect cache。

[ikaven1024]

The code you show is watch method. Objects in event are deep copied in https://github.com/karmada-io/karmada/blob/614e28508336d6c03a938ce1bf0678dafef034f0/vendor/k8s.io/apiserver/pkg/storage/cacher/cacher.go#L1289-L1303.

For get method, object is deep copied: https://github.com/karmada-io/karmada/blob/21282fd1832b225892130654c4715de452de3e28/pkg/search/proxy/store/multi_cluster_cache.go#L139-L150

If you don't add deep copy ，Using the same pointer in cache and changing the annotation value will affect cache。

Could you describe how to trigger this matter?

[hansongChina]

Ok, I got it. I'll be running around watching your code when it merges into the 1.3.0 BRANCH

[ikaven1024]

@hansongChina could you show the running goroutines below the panic in log. I want to know who is writing the map.

[ikaven1024]

Ok, I got it. I'll be running around watching your code when it merges into the 1.3.0 BRANCH

Sorry i forgot to tell you these commits had been merged into 1.3.0 branch.

[XiShanYongYe-Chang]

Hi @hansongChina, How's it going? Do you have any problems using version 1.3?