vmware-tanzu / velero

Backup and migrate Kubernetes applications and their persistent volumes
https://velero.io
Apache License 2.0
8.58k stars 1.39k forks source link

"unexpected fault address" when executing velero backup-location get --default -oyaml #5154

Closed matthiasritter closed 1 year ago

matthiasritter commented 2 years ago

What steps did you take and what happened: I executed the following command velero backup-location get --default -o yaml or I executed the following command velero backup-location get --default -o json The command failed with the error:

unexpected fault address 0x360808085a175a75
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x2 addr=0x360808085a175a75 pc=0x1046ce6c0]

goroutine 1 [running]:
runtime.throw({0x105ac0a97?, 0x0?})
    runtime/panic.go:992 +0x50 fp=0x14000bd7160 sp=0x14000bd7130 pc=0x10469e300
runtime.sigpanic()
    runtime/signal_unix.go:825 +0x1a4 fp=0x14000bd7190 sp=0x14000bd7160 pc=0x1046b53c4
aeshashbody()
    runtime/asm_arm64.s:878 +0x2d0 fp=0x14000bd71a0 sp=0x14000bd71a0 pc=0x1046ce6c0
runtime.mapiternext(0x140001f0200)
    runtime/map.go:934 +0x2c4 fp=0x14000bd7210 sp=0x14000bd71a0 pc=0x104678894
runtime.mapiterinit(0x1400068ea20?, 0x140001f4cc0?, 0x140005d8a80?)
    runtime/map.go:861 +0x2f0 fp=0x14000bd7240 sp=0x14000bd7210 pc=0x104678590
reflect.mapiterinit(0x10019e001?, 0x140005d8a98?, 0x140001f0200?)
    runtime/map.go:1373 +0x20 fp=0x14000bd7270 sp=0x14000bd7240 pc=0x1046caa20
github.com/modern-go/reflect2.(*UnsafeMapType).UnsafeIterate(...)
    github.com/modern-go/reflect2@v1.0.1/unsafe_map.go:112
github.com/json-iterator/go.(*sortKeysMapEncoder).IsEmpty(0x14000bd7378?, 0x104672d54?)
    github.com/json-iterator/go@v1.1.11/reflect_map.go:333 +0x30 fp=0x14000bd72b0 sp=0x14000bd7270 pc=0x104ae7270
github.com/json-iterator/go.(*placeholderEncoder).IsEmpty(0x106068398?, 0x140009361e0?)
    github.com/json-iterator/go@v1.1.11/reflect.go:336 +0x2c fp=0x14000bd72d0 sp=0x14000bd72b0 pc=0x104adf62c
github.com/json-iterator/go.(*structFieldEncoder).IsEmpty(0x140007ea300, 0x10605e9fd?)
    github.com/json-iterator/go@v1.1.11/reflect_struct_encoder.go:118 +0x50 fp=0x14000bd72f0 sp=0x14000bd72d0 pc=0x104af2f00
github.com/json-iterator/go.(*structEncoder).Encode(0x140007ea450, 0x140005d8a98?, 0x140005d8a80)
    github.com/json-iterator/go@v1.1.11/reflect_struct_encoder.go:148 +0x554 fp=0x14000bd73e0 sp=0x14000bd72f0 pc=0x104af3534
github.com/json-iterator/go.(*structFieldEncoder).Encode(0x140004ba840, 0x105eaf4d3?, 0x140005d8a80)
    github.com/json-iterator/go@v1.1.11/reflect_struct_encoder.go:110 +0x58 fp=0x14000bd7460 sp=0x14000bd73e0 pc=0x104af2d68
github.com/json-iterator/go.(*structEncoder).Encode(0x140004ba930, 0x140003ca870?, 0x140005d8a80)
    github.com/json-iterator/go@v1.1.11/reflect_struct_encoder.go:158 +0x708 fp=0x14000bd7550 sp=0x14000bd7460 pc=0x104af36e8
github.com/json-iterator/go.(*OptionalEncoder).Encode(0x1062c8ce0?, 0x14000267860?, 0x0?)
    github.com/json-iterator/go@v1.1.11/reflect_optional.go:70 +0xcc fp=0x14000bd75b0 sp=0x14000bd7550 pc=0x104aeb9dc
github.com/json-iterator/go.(*onePtrEncoder).Encode(0x140007241f0, 0x140009361c0, 0x14000376f00?)
    github.com/json-iterator/go@v1.1.11/reflect.go:219 +0x8c fp=0x14000bd75f0 sp=0x14000bd75b0 pc=0x104adebdc
github.com/json-iterator/go.(*Stream).WriteVal(0x140005d8a80, {0x1062c8ce0, 0x140009361c0})
    github.com/json-iterator/go@v1.1.11/reflect.go:98 +0x174 fp=0x14000bd7670 sp=0x14000bd75f0 pc=0x104addfa4
github.com/json-iterator/go.(*frozenConfig).Marshal(0x14000267860, {0x1062c8ce0, 0x140009361c0})
    github.com/json-iterator/go@v1.1.11/config.go:299 +0x88 fp=0x14000bd7710 sp=0x14000bd7670 pc=0x104ad5358
k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).doEncode(0x105ad1bc7?, {0x10631b310?, 0x140009361c0?}, {0x106313548, 0x140000e0a80})
    k8s.io/apimachinery@v0.22.2/pkg/runtime/serializer/json/json.go:306 +0x54 fp=0x14000bd77b0 sp=0x14000bd7710 pc=0x104efa624
k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Encode(0x140001123c0, {0x10631b310, 0x140009361c0}, {0x106313548, 0x140000e0a80})
    k8s.io/apimachinery@v0.22.2/pkg/runtime/serializer/json/json.go:301 +0xe4 fp=0x14000bd7810 sp=0x14000bd77b0 pc=0x104efa584
github.com/vmware-tanzu/velero/pkg/util/encode.EncodeTo({0x10631b310, 0x140009361c0}, {0x16b7977d7?, 0x10631b338?}, {0x106313548, 0x140000e0a80})
    github.com/vmware-tanzu/velero/pkg/util/encode/encode.go:50 +0x6c fp=0x14000bd7860 sp=0x14000bd7810 pc=0x1056128ec
github.com/vmware-tanzu/velero/pkg/util/encode.Encode({0x10631b310, 0x140009361c0}, {0x16b7977d7, 0x4})
    github.com/vmware-tanzu/velero/pkg/util/encode/encode.go:36 +0x5c fp=0x14000bd78b0 sp=0x14000bd7860 pc=0x1056127dc
github.com/vmware-tanzu/velero/pkg/cmd/util/output.printEncoded({0x10631b338, 0x140002cc070}, {0x16b7977d7, 0x4})
    github.com/vmware-tanzu/velero/pkg/cmd/util/output/output.go:135 +0x74 fp=0x14000bd7900 sp=0x14000bd78b0 pc=0x1059d0814
github.com/vmware-tanzu/velero/pkg/cmd/util/output.PrintWithFormat(0x0?, {0x10631b338, 0x140002cc070})
    github.com/vmware-tanzu/velero/pkg/cmd/util/output/output.go:117 +0x7c fp=0x14000bd7950 sp=0x14000bd7900 pc=0x1059d06bc
github.com/vmware-tanzu/velero/pkg/cmd/cli/backuplocation.NewGetCommand.func1(0x14000894000?, {0x1400068e680, 0x0, 0x2?})
    github.com/vmware-tanzu/velero/pkg/cmd/cli/backuplocation/get.go:87 +0x1cc fp=0x14000bd7da0 sp=0x14000bd7950 pc=0x1059feaac
github.com/spf13/cobra.(*Command).execute(0x14000894000, {0x1400068e660, 0x2, 0x2})
    github.com/spf13/cobra@v1.2.1/command.go:860 +0x4ac fp=0x14000bd7e80 sp=0x14000bd7da0 pc=0x10480bf8c
github.com/spf13/cobra.(*Command).ExecuteC(0x14000139680)
    github.com/spf13/cobra@v1.2.1/command.go:974 +0x354 fp=0x14000bd7f40 sp=0x14000bd7e80 pc=0x10480c594
github.com/spf13/cobra.(*Command).Execute(...)
    github.com/spf13/cobra@v1.2.1/command.go:902
main.main()
    github.com/vmware-tanzu/velero/cmd/velero/velero.go:34 +0x64 fp=0x14000bd7f70 sp=0x14000bd7f40 pc=0x105abe3f4
runtime.main()
    runtime/proc.go:250 +0x250 fp=0x14000bd7fd0 sp=0x14000bd7f70 pc=0x1046a0840
runtime.goexit()
    runtime/asm_arm64.s:1263 +0x4 fp=0x14000bd7fd0 sp=0x14000bd7fd0 pc=0x1046d1754

goroutine 19 [chan receive]:
k8s.io/klog.(*loggingT).flushDaemon(0x0?)
    k8s.io/klog@v1.0.0/klog.go:1010 +0x60
created by k8s.io/klog.init.0
    k8s.io/klog@v1.0.0/klog.go:411 +0x148

goroutine 20 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x0?)
    k8s.io/klog/v2@v2.9.0/klog.go:1169 +0x60
created by k8s.io/klog/v2.init.0
    k8s.io/klog/v2@v2.9.0/klog.go:420 +0x154

goroutine 21 [sleep]:
time.Sleep(0x6fc23ac00)
    runtime/time.go:194 +0x11c
sigs.k8s.io/controller-runtime/pkg/log.init.0.func1()
    sigs.k8s.io/controller-runtime@v0.10.2/pkg/log/log.go:63 +0x34
created by sigs.k8s.io/controller-runtime/pkg/log.init.0
    sigs.k8s.io/controller-runtime@v0.10.2/pkg/log/log.go:62 +0x28

goroutine 69 [IO wait]:
internal/poll.runtime_pollWait(0x12ece42b8, 0x72)
    runtime/netpoll.go:302 +0xa4
internal/poll.(*pollDesc).wait(0x140008cc380?, 0x1400075a000?, 0x0)
    internal/poll/fd_poll_runtime.go:83 +0x2c
internal/poll.(*pollDesc).waitRead(...)
    internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0x140008cc380, {0x1400075a000, 0x7811, 0x7811})
    internal/poll/fd_unix.go:167 +0x1e4
net.(*netFD).Read(0x140008cc380, {0x1400075a000?, 0x14000902198?, 0x8?})
    net/fd_posix.go:55 +0x2c
net.(*conn).Read(0x140001a4b90, {0x1400075a000?, 0x14000543798?, 0x10467fd5c?})
    net/net.go:183 +0x38
crypto/tls.(*atLeastReader).Read(0x140001b4378, {0x1400075a000?, 0x0?, 0x104673b38?})
    crypto/tls/conn.go:785 +0x44
bytes.(*Buffer).ReadFrom(0x14000902278, {0x1063136c8, 0x140001b4378})
    bytes/buffer.go:204 +0x94
crypto/tls.(*Conn).readFromUntil(0x14000902000, {0x12eb86f00?, 0x140001a4b90}, 0x77ca?)
    crypto/tls/conn.go:807 +0xd8
crypto/tls.(*Conn).readRecordOrCCS(0x14000902000, 0x0)
    crypto/tls/conn.go:614 +0xdc
crypto/tls.(*Conn).readRecord(...)
    crypto/tls/conn.go:582
crypto/tls.(*Conn).Read(0x14000902000, {0x14000a47000, 0x1000, 0x14000543ce0?})
    crypto/tls/conn.go:1285 +0x180
bufio.(*Reader).Read(0x14000944720, {0x140009341f8, 0x9, 0x1400019a0f0?})
    bufio/bufio.go:236 +0x1e4
io.ReadAtLeast({0x1063134c8, 0x14000944720}, {0x140009341f8, 0x9, 0x9}, 0x9)
    io/io.go:331 +0xa4
io.ReadFull(...)
    io/io.go:350
golang.org/x/net/http2.readFrameHeader({0x140009341f8?, 0x9?, 0x1400053af39?}, {0x1063134c8?, 0x14000944720?})
    golang.org/x/net@v0.0.0-20210520170846-37e1c6afe023/http2/frame.go:237 +0x5c
golang.org/x/net/http2.(*Framer).ReadFrame(0x140009341c0)
    golang.org/x/net@v0.0.0-20210520170846-37e1c6afe023/http2/frame.go:492 +0x84
golang.org/x/net/http2.(*clientConnReadLoop).run(0x14000543f90)
    golang.org/x/net@v0.0.0-20210520170846-37e1c6afe023/http2/transport.go:1821 +0x134
golang.org/x/net/http2.(*ClientConn).readLoop(0x14000900180)
    golang.org/x/net@v0.0.0-20210520170846-37e1c6afe023/http2/transport.go:1743 +0x68
created by golang.org/x/net/http2.(*Transport).newClientConn
    golang.org/x/net@v0.0.0-20210520170846-37e1c6afe023/http2/transport.go:695 +0x9c4

What did you expect to happen: To see the actuall configuration in yaml

The following information will help us better understand what's going on: We are using:

Environment:

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

qiuming-best commented 2 years ago

@matthiasritter how do you install velero? And whether have you compiled the velero bin by yourself?

matthiasritter commented 2 years ago

Hi @qiuming-best, we install velero via helm-chart (and terraform helm_release). I've got an M1-Mac and installed the velero-bin via "brew".

qiuming-best commented 2 years ago

@matthiasritter is it 100% occur when executing the above command?

I've tried in my own M1 Mac in AKS cluster, All going well.

matthiasritter commented 2 years ago

Hi @qiuming-best ,

yes it occur every time reproducable (same with my colleagues):

kubectl get backupstoragelocations.velero.io
NAME      PHASE       LAST VALIDATED   AGE   DEFAULT
default   Available   19s              12d   true
k get backupstoragelocations.velero.io default -oyaml
apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  annotations:
    helm.sh/hook: post-install,post-upgrade,post-rollback
    helm.sh/hook-delete-policy: before-hook-creation
  creationTimestamp: "2022-07-19T14:35:36Z"
  generation: 95827
  labels:
    app.kubernetes.io/instance: velero
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: velero
    helm.sh/chart: velero-2.30.1
  name: default
  namespace: velero
  resourceVersion: "143367598"
  uid: 12345678900ß
spec:
  accessMode: ReadWrite
  config:
    bucket: velero-backups
    prefix: ""
    resourceGroup: myresourcegroup
    storageAccount: backupstorageaccount
    subscriptionId: 12345678901234567890
  default: true
  objectStorage:
    bucket: velero-backups
  provider: azure
status:
  lastSyncedTime: "2022-08-01T05:16:29Z"
  lastValidationTime: "2022-08-01T05:16:30Z"
  phase: Available
velero backup-location get --default -o yaml
unexpected fault address 0x360808085a175a75
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x2 addr=0x360808085a175a75 pc=0x1007826c0]
....
qiuming-best commented 2 years ago

@matthiasritter could you provide the output of command velero version for me ?

matthiasritter commented 2 years ago

@qiuming-best sure velero version Client: Version: v1.9.0 Git commit: - Server: Version: v1.9.0

qiuming-best commented 2 years ago

@matthiasritter thanks, is there no Git commit ID shown by the output?

matthiasritter commented 2 years ago

@qiuming-best nope, only this:

velero version
Client:
    Version: v1.9.0
    Git commit: -
Server:
    Version: v1.9.0
qiuming-best commented 2 years ago

@matthiasritter could you retry with the link velero bin instead of using the brew installation way?

matthiasritter commented 2 years ago

@qiuming-best thanks, with this bin it work

jbpin commented 2 years ago

I had the same issue with velero backup get -o json it work with the link. Edit: I've install velero with brew as well

Lyndon-Li commented 2 years ago

Looks like the Velero image is from 1.8 code but compiled with go 1.18, so it should be the same with issue #4942:

  1. They have exactly the same stack trace
  2. The stack trace in this issue matches to the code of go 1.18 perfectly, but it doesn't match go 1.17
  3. From the stack trace, github.com/json-iterator/go@v1.1.11 is called, v1.1.11 is used by Velero 1.8 only, Velero 1.9 is using v1.1.12

However, I don't know how the problem connects to brew install.

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 1 year ago

Closing the stale issue.