search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
123.7k stars 17.62k forks source link

reflect: unexpected fault address 0x0 #68355

Closed shaj13 closed 2 months ago

shaj13 commented 3 months ago

Go version

go version go1.22.1 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/go/pkg/mod'
GONOPROXY='<internal>'
GONOSUMDB='<internal>'
GOOS='linux'
GOPATH='/go'
GOPRIVATE='<internal>'
GOPROXY='<internal>'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.1'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1448936908=/tmp/go-build -gno-record-gcc-switches'

What did you do?

Two months ago, we upgraded from 1.21 to 1.22.1. Today, we received an incident report about a panic. Upon reviewing the logs, we found that it originated from the reflect package due to a fault address. The issue occurs when a Kubernetes object is passed to the apimachinery unstructured, which uses reflection behind the scenes.

import (
    v1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
    "k8s.io/apimachinery/pkg/runtime"
    v1beta1 "k8s.io/metrics/pkg/apis/metrics/v1beta1"
)

    if err := runtime.
        DefaultUnstructuredConverter.
        FromUnstructured(
            u.UnstructuredContent(),
            v,
        ); err != nil {
        return nil, err
    }

The UnstructuredContent() never returns nil

image

What did you see happen?

2024-07-08T14:12:49.881927297Z unexpected fault address 0x0
2024-07-08T14:12:49.881958704Z fatal error: fault
2024-07-08T14:12:49.886359489Z [signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x4c7ec5]
2024-07-08T14:12:49.886370829Z 
2024-07-08T14:12:49.886377957Z goroutine 130 gp=0xc000254fc0 m=6 mp=0xc00049a008 [running]:
2024-07-08T14:12:49.886383321Z runtime.throw({0x1c1dcd6?, 0xc003082750?})
2024-07-08T14:12:49.886388154Z  /usr/local/go/src/runtime/panic.go:1023 +0x5c fp=0xc002af8978 sp=0xc002af8948 pc=0x43d39c
2024-07-08T14:12:49.886392280Z runtime.sigpanic()
2024-07-08T14:12:49.886395705Z  /usr/local/go/src/runtime/signal_unix.go:895 +0x285 fp=0xc002af89d8 sp=0xc002af8978 pc=0x455f25
2024-07-08T14:12:49.886398845Z internal/abi.(*Type).HasName(...)
2024-07-08T14:12:49.886402324Z  /usr/local/go/src/internal/abi/type.go:172
2024-07-08T14:12:49.886405359Z reflect.directlyAssignable(0x10?, 0x1ed8308?)
2024-07-08T14:12:49.886409097Z  /usr/local/go/src/reflect/type.go:1386 +0x25 fp=0xc002af8a00 sp=0xc002af89d8 pc=0x4c7ec5
2024-07-08T14:12:49.886413876Z reflect.Value.assignTo({0x6ff0040c6013c00?, 0x1e0a1d001e0a8165?, 0xc00375b2d0?}, {0x1c3588f, 0x16}, 0x18988c0, 0x0)
2024-07-08T14:12:49.886417160Z  /usr/local/go/src/reflect/value.go:3329 +0x98 fp=0xc002af8aa8 sp=0xc002af8a00 pc=0x4dccd8
2024-07-08T14:12:49.886420354Z reflect.Value.MapIndex({0x197b5e0?, 0xc003ec9860?, 0x13?}, {0x6ff0040c6013c00, 0x1e0a1d001e0a8165, 0xf745010013801c00})
2024-07-08T14:12:49.886423642Z  /usr/local/go/src/reflect/value.go:1821 +0xe8 fp=0xc002af8b18 sp=0xc002af8aa8 pc=0x4d7728
2024-07-08T14:12:49.886428433Z k8s.io/apimachinery/pkg/runtime.structFromUnstructured({0x197b5e0?, 0xc003ec9860?, 0x7f1b79762d98?}, {0x1bd1820?, 0xc003082600?, 0x41a2f8?}, 0xc002af9778)
2024-07-08T14:12:49.886442321Z  vendor/k8s.io/apimachinery/pkg/runtime/converter.go:546 +0x49c fp=0xc002af8c80 sp=0xc002af8b18 pc=0xc7c5bc
2024-07-08T14:12:49.886447122Z k8s.io/apimachinery/pkg/runtime.fromUnstructured({0x1951ce0?, 0xc0047ccfb0?, 0x1?}, {0x1bd1820?, 0xc003082600?, 0xc66e9a?}, 0xc002af9778)
2024-07-08T14:12:49.886450668Z  vendor/k8s.io/apimachinery/pkg/runtime/converter.go:359 +0x3cf fp=0xc002af8de8 sp=0xc002af8c80 pc=0xc79daf
2024-07-08T14:12:49.886454051Z k8s.io/apimachinery/pkg/runtime.sliceFromUnstructured({0x186eec0?, 0xc0035f4b40?, 0x411c7b?}, {0x18755e0?, 0xc004318140?, 0xc002de2e10?}, 0xc002af9778)
2024-07-08T14:12:49.886457313Z  vendor/k8s.io/apimachinery/pkg/runtime/converter.go:488 +0x8c5 fp=0xc002af8f68 sp=0xc002af8de8 pc=0xc7bc05
2024-07-08T14:12:49.886460485Z k8s.io/apimachinery/pkg/runtime.fromUnstructured({0x186eec0?, 0xc0035f4b40?, 0x2?}, {0x18755e0?, 0xc004318140?, 0x98?}, 0xc002af9778)
2024-07-08T14:12:49.886463787Z  vendor/k8s.io/apimachinery/pkg/runtime/converter.go:355 +0x412 fp=0xc002af90d0 sp=0xc002af8f68 pc=0xc79df2
2024-07-08T14:12:49.886466748Z k8s.io/apimachinery/pkg/runtime.structFromUnstructured({0x197b5e0?, 0xc003ec9710?, 0x411c7b?}, {0x1c03220?, 0xc004318110?, 0xc002de2e10?}, 0xc002af9778)
2024-07-08T14:12:49.886469993Z  vendor/k8s.io/apimachinery/pkg/runtime/converter.go:550 +0x816 fp=0xc002af9238 sp=0xc002af90d0 pc=0xc7c936
2024-07-08T14:12:49.886473076Z k8s.io/apimachinery/pkg/runtime.fromUnstructured({0x197b5e0?, 0xc003ec9710?, 0x2?}, {0x1c03220?, 0xc004318110?, 0x98?}, 0xc002af9778)
2024-07-08T14:12:49.886476340Z  vendor/k8s.io/apimachinery/pkg/runtime/converter.go:359 +0x3cf fp=0xc002af93a0 sp=0xc002af9238 pc=0xc79daf
2024-07-08T14:12:49.886479386Z k8s.io/apimachinery/pkg/runtime.structFromUnstructured({0x197b5e0?, 0xc003ec2270?, 0xc002af9590?}, {0x1ad80a0?, 0xc004318008?, 0xc002af9670?}, 0xc002af9778)
2024-07-08T14:12:49.886482403Z  vendor/k8s.io/apimachinery/pkg/runtime/converter.go:550 +0x816 fp=0xc002af9508 sp=0xc002af93a0 pc=0xc7c936
2024-07-08T14:12:49.886508041Z k8s.io/apimachinery/pkg/runtime.fromUnstructured({0x197b5e0?, 0xc003ec2270?, 0x0?}, {0x1ad80a0?, 0xc004318008?, 0x630b4?}, 0xc002af9778)
2024-07-08T14:12:49.886513850Z  vendor/k8s.io/apimachinery/pkg/runtime/converter.go:359 +0x3cf fp=0xc002af9670 sp=0xc002af9508 pc=0xc79daf
2024-07-08T14:12:49.886517216Z k8s.io/apimachinery/pkg/runtime.(*unstructuredConverter).FromUnstructuredWithValidation(0x2ce5e60, 0xc003ec2270, {0x1be1140, 0xc004318008}, 0x0)
2024-07-08T14:12:49.886520437Z  vendor/k8s.io/apimachinery/pkg/runtime/converter.go:247 +0x290 fp=0xc002af9800 sp=0xc002af9670 pc=0xc793b0
2024-07-08T14:12:49.886523599Z k8s.io/apimachinery/pkg/runtime.(*unstructuredConverter).FromUnstructured(...)
2024-07-08T14:12:49.886528439Z  vendor/k8s.io/apimachinery/pkg/runtime/converter.go:274

What did you expect to see?

reflect package does not panic.

gabyhelp commented 3 months ago

Related Issues and Documentation

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

seankhliao commented 3 months ago

Is this reproducible? or memory corruption on your machine?

shaj13 commented 3 months ago

@seankhliao

What do you mean by memory corruption on your machine? I haven't been able to reproduce it, but the panic indicates it's coming from the reflect package. FYI This is happening on VMware ESXi.

ianlancetaylor commented 3 months ago

The question is whether this could be due to memory corruption on the machine on which the program was running.

shaj13 commented 3 months ago

The question is whether this could be due to memory corruption on the machine on which the program was running.

I don't believe there is memory corruption (but not certain).

FYI, Env is a Kubernetes cluster running on VMware ESXi, with most pods running Go 1.21. We upgraded this specific program/pod to Go 1.22.1, and that’s when we observed this issue and https://github.com/golang/go/issues/64030#issuecomment-2186446853. I got the core file Let me know if you need anything specific.

(dlv) bt
 0  0x0000000000479501 in runtime.raise
    at /usr/local/go/src/runtime/sys_linux_amd64.s:154
 1  0x00000000004560e5 in runtime.dieFromSignal
    at /usr/local/go/src/runtime/signal_unix.go:923
 2  0x00000000004567e6 in runtime.sigfwdgo
    at /usr/local/go/src/runtime/signal_unix.go:1128
 3  0x0000000000454d05 in runtime.sigtrampgo
    at /usr/local/go/src/runtime/signal_unix.go:432
 4  0x0000000000479501 in runtime.raise
    at /usr/local/go/src/runtime/sys_linux_amd64.s:153
 5  0x00000000004560e5 in runtime.dieFromSignal
    at /usr/local/go/src/runtime/signal_unix.go:923
 6  0x000000000043d89a in runtime.crash
    at /usr/local/go/src/runtime/signal_unix.go:1005
 7  0x000000000043d89a in runtime.fatalthrow.func1
    at /usr/local/go/src/runtime/panic.go:1203
 8  0x0000000000475e4a in runtime.systemstack
    at /usr/local/go/src/runtime/asm_amd64.s:509
 9  0x0000000000475de8 in runtime.systemstack_switch
    at /usr/local/go/src/runtime/asm_amd64.s:474
10  0x000000000043d7e5 in runtime.fatalthrow
    at /usr/local/go/src/runtime/panic.go:1192
11  0x000000000043d39c in runtime.throw
    at /usr/local/go/src/runtime/panic.go:1023
12  0x0000000000455f25 in runtime.sigpanic
    at /usr/local/go/src/runtime/signal_unix.go:895
13  0x00000000004c7ec5 in reflect.directlyAssignable
    at /usr/local/go/src/reflect/type.go:1386
Sending output to pager...
 0  0x0000000000479501 in runtime.raise
    at /usr/local/go/src/runtime/sys_linux_amd64.s:154
 1  0x00000000004560e5 in runtime.dieFromSignal
    at /usr/local/go/src/runtime/signal_unix.go:923
 2  0x00000000004567e6 in runtime.sigfwdgo
    at /usr/local/go/src/runtime/signal_unix.go:1128
 3  0x0000000000454d05 in runtime.sigtrampgo
    at /usr/local/go/src/runtime/signal_unix.go:432
 4  0x0000000000479501 in runtime.raise
    at /usr/local/go/src/runtime/sys_linux_amd64.s:153
 5  0x00000000004560e5 in runtime.dieFromSignal
    at /usr/local/go/src/runtime/signal_unix.go:923
 6  0x000000000043d89a in runtime.crash
    at /usr/local/go/src/runtime/signal_unix.go:1005
 7  0x000000000043d89a in runtime.fatalthrow.func1
    at /usr/local/go/src/runtime/panic.go:1203
 8  0x0000000000475e4a in runtime.systemstack
    at /usr/local/go/src/runtime/asm_amd64.s:509
 9  0x0000000000475de8 in runtime.systemstack_switch
    at /usr/local/go/src/runtime/asm_amd64.s:474
10  0x000000000043d7e5 in runtime.fatalthrow
    at /usr/local/go/src/runtime/panic.go:1192
11  0x000000000043d39c in runtime.throw
    at /usr/local/go/src/runtime/panic.go:1023
12  0x0000000000455f25 in runtime.sigpanic
    at /usr/local/go/src/runtime/signal_unix.go:895
13  0x00000000004c7ec5 in reflect.directlyAssignable
    at /usr/local/go/src/reflect/type.go:1386
14  0x00000000004dccd8 in reflect.Value.assignTo
    at /usr/local/go/src/reflect/value.go:3329
15  0x00000000004d7728 in reflect.Value.MapIndex
    at /usr/local/go/src/reflect/value.go:1821
16  0x0000000000c7c5bc in k8s.io/apimachinery/pkg/runtime.structFromUnstructured
:546
17  0x0000000000c79daf in k8s.io/apimachinery/pkg/runtime.fromUnstructured
    at vendor/k8s.io/apimachinery/pkg/runtime/converter.go
:359
18  0x0000000000c7bc05 in k8s.io/apimachinery/pkg/runtime.sliceFromUnstructured
    at vendor/k8s.io/apimachinery/pkg/runtime/converter.go
:488
19  0x0000000000c79df2 in k8s.io/apimachinery/pkg/runtime.fromUnstructured
    at vendor/k8s.io/apimachinery/pkg/runtime/converter.go
:355
20  0x0000000000c7c936 in k8s.io/apimachinery/pkg/runtime.structFromUnstructured
    at vendor/k8s.io/apimachinery/pkg/runtime/converter.go
:550
21  0x0000000000c79daf in k8s.io/apimachinery/pkg/runtime.fromUnstructured
    at vendor/k8s.io/apimachinery/pkg/runtime/converter.go
:359
22  0x0000000000c7c936 in k8s.io/apimachinery/pkg/runtime.structFromUnstructured
    at vendor/k8s.io/apimachinery/pkg/runtime/converter.go
:550
23  0x0000000000c79daf in k8s.io/apimachinery/pkg/runtime.fromUnstructured
    at vendor/k8s.io/apimachinery/pkg/runtime/converter.go
:359
24  0x0000000000c793b0 in k8s.io/apimachinery/pkg/runtime.(*unstructuredConverter).FromUnstructuredWithValidat
ion
    at vendor/k8s.io/apimachinery/pkg/runtime/converter.go
:247
25  0x000000000179b635 in k8s.io/apimachinery/pkg/runtime.(*unstructuredConverter).FromUnstructured
    at vendor/k8s.io/apimachinery/pkg/runtime/converter.go
:274

(dlv) frame 24
> runtime.raise() /usr/local/go/src/runtime/sys_linux_amd64.s:154 (PC: 0x479501)
Warning: debugging optimized function
Frame 24: vendor/k8s.io/apimachinery/pkg/runtime/converter.go:247 (PC: c793b0)
(dlv) 
> runtime.raise() /usr/local/go/src/runtime/sys_linux_amd64.s:154 (PC: 0x479501)
Warning: debugging optimized function
Frame 24: vendor/k8s.io/apimachinery/pkg/runtime/converter.go:247 (PC: c793b0)

(dlv) locals
~r0typ_ = (unreadable empty OP stack)
t = reflect.Type(*reflect.rtype) 0xbeef000000000808
value = reflect.Value {typ_: ("*internal/abi.Type")(0x1be1140), ptr: unsafe.Pointer(0xc004318008), flag: 22}
fromUnstructuredContext = (unreadable empty OP stack)

(dlv) args
c = ("*k8s.io/apimachinery/pkg/runtime.unstructuredConverter")(0x2ce5e60)
u = map[string]interface {} [...]
obj = interface {}(*k8s.io/api/core/v1.Pod) 0xbeef000000000608
returnUnknownFields = false
~r0 = (unreadable empty OP stack)

(dlv) print u 
map[string]interface {} [
    "metadata": map[string]interface {} [
        "generateName": *(*interface {})(0xc0037f5dd0), 
        "uid": *(*interface {})(0xc0037f5de0), 
        "labels": *(*interface {})(0xc0037f5df0), 
        "annotations": *(*interface {})(0xc0037f5e00), 
        "managedFields": *(*interface {})(0xc0037f5e10), 
        "name": *(*interface {})(0xc0037f5ee0), 
        "namespace": *(*interface {})(0xc0037f5ef0), 
        "resourceVersion": *(*interface {})(0xc0037f5f00), 
        "creationTimestamp": *(*interface {})(0xc0037f5f10), 
        "ownerReferences": *(*interface {})(0xc0037f5f20), 
    ], 
    "spec": map[string]interface {} [
        "volumes": *(*interface {})(0xc003ed2090), 
        "securityContext": *(*interface {})(0xc003ed20a0), 
        "containers": *(*interface {})(0xc003ed21a0), 
        "automountServiceAccountToken": *(*interface {})(0xc003ed21b0), 
        "schedulerName": *(*interface {})(0xc003ed21c0), 
        "tolerations": *(*interface {})(0xc003ed21d0), 
        "restartPolicy": *(*interface {})(0xc003ed22b0), 
        "terminationGracePeriodSeconds": *(*interface {})(0xc003ed22c0), 
        "dnsPolicy": *(*interface {})(0xc003ed22d0), 
        "serviceAccountName": *(*interface {})(0xc003ed22e0), 
        "serviceAccount": *(*interface {})(0xc003ed22f0), 
        "enableServiceLinks": *(*interface {})(0xc003ed2300), 
        "nodeName": *(*interface {})(0xc003ed23c0), 
        "priority": *(*interface {})(0xc003ed23d0), 
        "preemptionPolicy": *(*interface {})(0xc003ed23e0), 
    ], 
    "status": map[string]interface {} [
        "conditions": *(*interface {})(0xc00375b950), 
        "startTime": *(*interface {})(0xc00375b960), 
        "containerStatuses": *(*interface {})(0xc00375b970), 
        "qosClass": *(*interface {})(0xc00375b980), 
        "phase": *(*interface {})(0xc00375ba60), 
        "hostIP": *(*interface {})(0xc00375ba70), 
        "hostIPs": *(*interface {})(0xc00375ba80), 
        "podIP": *(*interface {})(0xc00375ba90), 
        "podIPs": *(*interface {})(0xc00375baa0), 
    ], 
    "kind": "Pod", 
    "apiVersion": "v1", 
]

(dlv) print obj 
interface {}(*k8s.io/api/core/v1.Pod) *{
    TypeMeta: k8s.io/apimachinery/pkg/apis/meta/v1.TypeMeta {Kind: "Pod", APIVersion: "v1"},
    ObjectMeta: k8s.io/apimachinery/pkg/apis/meta/v1.ObjectMeta {
        Name: "data-store-98975978-5qm6w",
        GenerateName: "data-store-98975978-",
        Namespace: "default",
        SelfLink: "",
        UID: "4f3fa736-728c-4b48-80d8-7d53e04b26b6",
        ResourceVersion: "7662",
        Generation: 0,
        CreationTimestamp: (*"k8s.io/apimachinery/pkg/apis/meta/v1.Time")(0xc004318090),
        DeletionTimestamp: *k8s.io/apimachinery/pkg/apis/meta/v1.Time nil,
        DeletionGracePeriodSeconds: *int64 nil,
        Labels: map[string]string [...],
        Annotations: map[string]string [...],
        OwnerReferences: []k8s.io/apimachinery/pkg/apis/meta/v1.OwnerReference len: 1, cap: 1, [
            (*"k8s.io/apimachinery/pkg/apis/meta/v1.OwnerReference")(0xc00259cb40),
        ],
        Finalizers: []string len: 0, cap: 0, nil,
        ManagedFields: []k8s.io/apimachinery/pkg/apis/meta/v1.ManagedFieldsEntry len: 3, cap: 4, [
            (*"k8s.io/apimachinery/pkg/apis/meta/v1.ManagedFieldsEntry")(0xc003082180),
            (*"k8s.io/apimachinery/pkg/apis/meta/v1.ManagedFieldsEntry")(0xc0030821e0),
            (*"k8s.io/apimachinery/pkg/apis/meta/v1.ManagedFieldsEntry")(0xc003082240),
        ],},
    Spec: k8s.io/api/core/v1.PodSpec {
        Volumes: []k8s.io/api/core/v1.Volume len: 3, cap: 4, [
            (*"k8s.io/api/core/v1.Volume")(0xc001527408),
            (*"k8s.io/api/core/v1.Volume")(0xc001527500),
            (*"k8s.io/api/core/v1.Volume")(0xc0015275f8),
        ],
        InitContainers: []k8s.io/api/core/v1.Container len: 0, cap: 0, nil,
        Containers: []k8s.io/api/core/v1.Container len: 1, cap: 1, [
            (*"k8s.io/api/core/v1.Container")(0xc003082600),
        ],
        EphemeralContainers: []k8s.io/api/core/v1.EphemeralContainer len: 0, cap: 0, nil,
        RestartPolicy: "",
        TerminationGracePeriodSeconds: *int64 nil,
        ActiveDeadlineSeconds: *int64 nil,
        DNSPolicy: "",
        NodeSelector: map[string]string nil,
        ServiceAccountName: "",
        DeprecatedServiceAccount: "",
        AutomountServiceAccountToken: *bool nil,
        NodeName: "",
        HostNetwork: false,
        HostPID: false,
        HostIPC: false,
        ShareProcessNamespace: *bool nil,
        SecurityContext: *k8s.io/api/core/v1.PodSecurityContext nil,
        ImagePullSecrets: []k8s.io/api/core/v1.LocalObjectReference len: 0, cap: 0, nil,
        Hostname: "",
        Subdomain: "",
        Affinity: *k8s.io/api/core/v1.Affinity nil,
        SchedulerName: "",
        Tolerations: []k8s.io/api/core/v1.Toleration len: 0, cap: 0, nil,
        HostAliases: []k8s.io/api/core/v1.HostAlias len: 0, cap: 0, nil,
        PriorityClassName: "",
        Priority: *int32 nil,
        DNSConfig: *k8s.io/api/core/v1.PodDNSConfig nil,
        ReadinessGates: []k8s.io/api/core/v1.PodReadinessGate len: 0, cap: 0, nil,
        RuntimeClassName: *string nil,
        EnableServiceLinks: *bool nil,
        PreemptionPolicy: *k8s.io/api/core/v1.PreemptionPolicy nil,
        Overhead: k8s.io/api/core/v1.ResourceList nil,
        TopologySpreadConstraints: []k8s.io/api/core/v1.TopologySpreadConstraint len: 0, cap: 0, nil,
        SetHostnameAsFQDN: *bool nil,
        OS: *k8s.io/api/core/v1.PodOS nil,
        HostUsers: *bool nil,
        SchedulingGates: []k8s.io/api/core/v1.PodSchedulingGate len: 0, cap: 0, nil,
        ResourceClaims: []k8s.io/api/core/v1.PodResourceClaim len: 0, cap: 0, nil,},
    Status: k8s.io/api/core/v1.PodStatus {
        Phase: "",
        Conditions: []k8s.io/api/core/v1.PodCondition len: 0, cap: 0, nil,
        Message: "",
        Reason: "",
        NominatedNodeName: "",
        HostIP: "",
        HostIPs: []k8s.io/api/core/v1.HostIP len: 0, cap: 0, nil,
        PodIP: "",
        PodIPs: []k8s.io/api/core/v1.PodIP len: 0, cap: 0, nil,
        StartTime: *k8s.io/apimachinery/pkg/apis/meta/v1.Time nil,
        InitContainerStatuses: []k8s.io/api/core/v1.ContainerStatus len: 0, cap: 0, nil,
        ContainerStatuses: []k8s.io/api/core/v1.ContainerStatus len: 0, cap: 0, nil,
        QOSClass: "",
        EphemeralContainerStatuses: []k8s.io/api/core/v1.ContainerStatus len: 0, cap: 0, nil,
        Resize: "",
        ResourceClaimStatuses: []k8s.io/api/core/v1.PodResourceClaimStatus len: 0, cap: 0, nil,},}

(dlv) frame 16
(dlv) frame 16
> runtime.raise() /usr/local/go/src/runtime/sys_linux_amd64.s:154 (PC: 0x479501)
Warning: debugging optimized function
Frame 16: vendor/k8s.io/apimachinery/pkg/runtime/converter.go:546 (PC: c7c5bc)

(dlv) args
sv = reflect.Value {typ_: ("*internal/abi.Type")(0x197b5e0), ptr: unsafe.Pointer(0xc003ec9860), flag: 21}
dv = reflect.Value {typ_: ("*internal/abi.Type")(0x1bd1820), ptr: unsafe.Pointer(0xc003082600), flag: 409}
ctx = ("*k8s.io/apimachinery/pkg/runtime.fromUnstructuredContext")(0xc002af9778)
(~r0) = (unreadable empty OP stack)
~r0 = (unreadable empty OP stack)
(dlv) p *(*"k8s.io/api/core/v1.Container")(0xc003ec9860)
k8s.io/api/core/v1.Container {
    Name: (unreadable invalid length: -3442974251240390400),
    Image: "",
    Command: []string len: 0, cap: 2, [],
    Args: []string len: 824699571008, cap: 0, [(unreadable could not read string pointer error while reading spliced memory at 0x832f3f0f00000000: EOF),(unreadable could not read string pointer error while reading spliced memory at 0x832f3f0f00000000: EOF),(unreadable could not read string pointer error while reading spliced memory at 0x832f3f0f00000000: EOF),(unreadable could not read string pointer error while reading spliced memory at 0x832f3f0f00000000: EOF),...+824699571004 more],
    WorkingDir: "",
    Ports: []k8s.io/api/core/v1.ContainerPort len: 2350715723715706880, cap: 824699571296, [
        (*"k8s.io/api/core/v1.ContainerPort")(0x3),
        (*"k8s.io/api/core/v1.ContainerPort")(0x3b),
        (*"k8s.io/api/core/v1.ContainerPort")(0x73),
        (*"k8s.io/api/core/v1.ContainerPort")(0xab),
        (*"k8s.io/api/core/v1.ContainerPort")(0xe3),
        (*"k8s.io/api/core/v1.ContainerPort")(0x11b),
        (*"k8s.io/api/core/v1.ContainerPort")(0x153),
        (*"k8s.io/api/core/v1.ContainerPort")(0x18b),
        (*"k8s.io/api/core/v1.ContainerPort")(0x1c3),
        (*"k8s.io/api/core/v1.ContainerPort")(0x1fb),
        (*"k8s.io/api/core/v1.ContainerPort")(0x233),
        (*"k8s.io/api/core/v1.ContainerPort")(0x26b),
        (*"k8s.io/api/core/v1.ContainerPort")(0x2a3),
        (*"k8s.io/api/core/v1.ContainerPort")(0x2db),
        (*"k8s.io/api/core/v1.ContainerPort")(0x313),
        (*"k8s.io/api/core/v1.ContainerPort")(0x34b),
        (*"k8s.io/api/core/v1.ContainerPort")(0x383),
        (*"k8s.io/api/core/v1.ContainerPort")(0x3bb),
        (*"k8s.io/api/core/v1.ContainerPort")(0x3f3),
        (*"k8s.io/api/core/v1.ContainerPort")(0x42b),
        (*"k8s.io/api/core/v1.ContainerPort")(0x463),
        (*"k8s.io/api/core/v1.ContainerPort")(0x49b),
        (*"k8s.io/api/core/v1.ContainerPort")(0x4d3),
        (*"k8s.io/api/core/v1.ContainerPort")(0x50b),
        (*"k8s.io/api/core/v1.ContainerPort")(0x543),
        (*"k8s.io/api/core/v1.ContainerPort")(0x57b),
        (*"k8s.io/api/core/v1.ContainerPort")(0x5b3),
        (*"k8s.io/api/core/v1.ContainerPort")(0x5eb),
        (*"k8s.io/api/core/v1.ContainerPort")(0x623),
        (*"k8s.io/api/core/v1.ContainerPort")(0x65b),
        (*"k8s.io/api/core/v1.ContainerPort")(0x693),
        (*"k8s.io/api/core/v1.ContainerPort")(0x6cb),
        (*"k8s.io/api/core/v1.ContainerPort")(0x703),
        (*"k8s.io/api/core/v1.ContainerPort")(0x73b),
        (*"k8s.io/api/core/v1.ContainerPort")(0x773),
        (*"k8s.io/api/core/v1.ContainerPort")(0x7ab),
        (*"k8s.io/api/core/v1.ContainerPort")(0x7e3),
        (*"k8s.io/api/core/v1.ContainerPort")(0x81b),
        (*"k8s.io/api/core/v1.ContainerPort")(0x853),
        (*"k8s.io/api/core/v1.ContainerPort")(0x88b),
        (*"k8s.io/api/core/v1.ContainerPort")(0x8c3),
        (*"k8s.io/api/core/v1.ContainerPort")(0x8fb),
        (*"k8s.io/api/core/v1.ContainerPort")(0x933),
        (*"k8s.io/api/core/v1.ContainerPort")(0x96b),
        (*"k8s.io/api/core/v1.ContainerPort")(0x9a3),
        (*"k8s.io/api/core/v1.ContainerPort")(0x9db),
        (*"k8s.io/api/core/v1.ContainerPort")(0xa13),
        (*"k8s.io/api/core/v1.ContainerPort")(0xa4b),
        (*"k8s.io/api/core/v1.ContainerPort")(0xa83),
        (*"k8s.io/api/core/v1.ContainerPort")(0xabb),
        (*"k8s.io/api/core/v1.ContainerPort")(0xaf3),
        (*"k8s.io/api/core/v1.ContainerPort")(0xb2b),
        (*"k8s.io/api/core/v1.ContainerPort")(0xb63),
        (*"k8s.io/api/core/v1.ContainerPort")(0xb9b),
        (*"k8s.io/api/core/v1.ContainerPort")(0xbd3),
        (*"k8s.io/api/core/v1.ContainerPort")(0xc0b),
        (*"k8s.io/api/core/v1.ContainerPort")(0xc43),
        (*"k8s.io/api/core/v1.ContainerPort")(0xc7b),
        (*"k8s.io/api/core/v1.ContainerPort")(0xcb3),
        (*"k8s.io/api/core/v1.ContainerPort")(0xceb),
        (*"k8s.io/api/core/v1.ContainerPort")(0xd23),
        (*"k8s.io/api/core/v1.ContainerPort")(0xd5b),
        (*"k8s.io/api/core/v1.ContainerPort")(0xd93),
        (*"k8s.io/api/core/v1.ContainerPort")(0xdcb),
        ...+2350715723715706816 more
    ],
    EnvFrom: []k8s.io/api/core/v1.EnvFromSource len: 0, cap: 0, nil,
    Env: (unreadable Negative array length),
    Resources: k8s.io/api/core/v1.ResourceRequirements {
        Limits: k8s.io/api/core/v1.ResourceList nil,
        Requests: k8s.io/api/core/v1.ResourceList nil,
        Claims: (unreadable non-zero length array with nil base),},
    ResizePolicy: []k8s.io/api/core/v1.ContainerResizePolicy len: 0, cap: 0, [],
    RestartPolicy: *k8s.io/api/core/v1.ContainerRestartPolicy nil,
    VolumeMounts: (unreadable Negative array length),
    VolumeDevices: []k8s.io/api/core/v1.VolumeDevice len: 0, cap: 0, nil,
    LivenessProbe: *k8s.io/api/core/v1.Probe {
        ProbeHandler: (*"k8s.io/api/core/v1.ProbeHandler")(0x1),
        InitialDelaySeconds: (unreadable error while reading spliced memory at 0x1: EOF),
        TimeoutSeconds: (unreadable error while reading spliced memory at 0x1: EOF),
        PeriodSeconds: (unreadable error while reading spliced memory at 0x1: EOF),
        SuccessThreshold: (unreadable error while reading spliced memory at 0x1: EOF),
        FailureThreshold: (unreadable error while reading spliced memory at 0x1: EOF),
        TerminationGracePeriodSeconds: (unreadable error while reading spliced memory at 0x1: EOF),},
    ReadinessProbe: *k8s.io/api/core/v1.Probe {
        ProbeHandler: (*"k8s.io/api/core/v1.ProbeHandler")(0xa4c9ba4900000000),
        InitialDelaySeconds: (unreadable error while reading spliced memory at 0xa4c9ba4900000000: EOF),
        TimeoutSeconds: (unreadable error while reading spliced memory at 0xa4c9ba4900000000: EOF),
        PeriodSeconds: (unreadable error while reading spliced memory at 0xa4c9ba4900000000: EOF),
        SuccessThreshold: (unreadable error while reading spliced memory at 0xa4c9ba4900000000: EOF),
        FailureThreshold: (unreadable error while reading spliced memory at 0xa4c9ba4900000000: EOF),
        TerminationGracePeriodSeconds: (unreadable error while reading spliced memory at 0xa4c9ba4900000000: EOF),},
    StartupProbe: *k8s.io/api/core/v1.Probe {
        ProbeHandler: (*"k8s.io/api/core/v1.ProbeHandler")(0xc003eccfc0),
        InitialDelaySeconds: 0,
        TimeoutSeconds: 0,
        PeriodSeconds: 0,
        SuccessThreshold: 0,
        FailureThreshold: 0,
        TerminationGracePeriodSeconds: *int64 nil,},
    Lifecycle: *k8s.io/api/core/v1.Lifecycle nil,
    TerminationMessagePath: "",
    TerminationMessagePolicy: (unreadable invalid length: -960315560233533440),
    ImagePullPolicy: "",
    SecurityContext: *k8s.io/api/core/v1.SecurityContext nil,
    Stdin: false,
    StdinOnce: false,
    TTY: false,}

(dlv) locals 
.autotmp_60 = struct {} {}
.autotmp_51 = struct {} {}
~r0ptr = (unreadable empty OP stack)
~r0flag = (unreadable empty OP stack)
st = reflect.Type(*reflect.rtype) 0xbeef000000000e08
dt = reflect.Type(*reflect.rtype) 0xbeef000000000f08
svInlined = (unreadable could not find loclist entry at 0xb05e76 for address 0xc7c5bc)
i = 19
fieldInfo = ("*k8s.io/apimachinery/pkg/runtime.fieldInfo")(0xc0032240c0)
(dlv) p *("*k8s.io/apimachinery/pkg/runtime.fieldInfo")(0xc0032240c0)
k8s.io/apimachinery/pkg/runtime.fieldInfo {
    name: (unreadable could not read string at 0x5000d33db7565000 due to error while reading spliced memory at 0x5000d33db7565000: EOF),
    nameValue: reflect.Value {
        typ_: *(*"internal/abi.Type")(0x6ff0040c6013c00),
        ptr: unsafe.Pointer(0x1e0a1d001e0a8165),
        flag: 17817648600600026112,},
    omitempty: true,}

Frame 15: /usr/local/go/src/reflect/value.go:1821 (PC: 4d7728)
(dlv) args
v = reflect.Value {typ_: ("*internal/abi.Type")(0x197b5e0), ptr: unsafe.Pointer(0xc003ec9860), flag: 0}
key = (unreadable could not find loclist entry at 0x15c5af for address 0x4d7728)
~r0 = (unreadable empty OP stack)
(dlv) locals
e = (unreadable could not find loclist entry at 0x15c66a for address 0x4d7728)
(dlv)
gopherbot commented 2 months ago

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

lukai249324454 commented 1 month ago

I also encountered this issue in the go 1.23.1 version, It occurs multiple times and takes over 10 minutes to appear. Each time, it occurs during a map of type map [string] any, when i for range a map to copy

image image

ianlancetaylor commented 1 month ago

@lukai249324454 Thanks. I don't see any obvious reason to think that you are seeing the same bug. Please open a new issue.

Please also include plain text as text, not as an image. Images are much harder to read. Thanks.