[Bug] One etcd-druid e2e tests fails due to panic in bolt db

Describe the bug: This etcd-druid e2e tests failed to pass as it leads to the etcd and backup-restore into an unrecoverable state.

Root cause analysis: It has been observed that when db file of etcd data-dir has been corrupted and when backup-restore tries to verify db by opening the boltDB then panic occurs in BoltDB as BoltDB is not expecting a corrupted DB file as it contains the information of pages and other metadata information. All this leads to etcd and backup-restore into an unrecoverable state.

Expected behavior: If db file of etcd data-dir has been corrupted then backup-restore should detect this and should be able to restore the data-dir of etcd in case of cluster size=1 and in case of cluster size>1 then backup-restore should able to trigger the single member restoration scenario.

How To Reproduce (as minimally and precisely as possible):

Run the etcd-druid e2e tests, and check this tests.

Logs: Backup-restore container logs:

ime="2022-12-09T09:07:20Z" level=info msg="Member etcd-aws-2 part of running cluster" actor=member-add
time="2022-12-09T09:07:20Z" level=info msg="Checking for data directory structure validity..."
time="2022-12-09T09:07:20Z" level=info msg="Checking for Etcd revision..."
unexpected fault address 0x7f4e66cc9008
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0x7f4e66cc9008 pc=0xed25be]

goroutine 234 [running]:
runtime.throw({0x1d20cfd?, 0x0?})
    /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0xc000213740 sp=0xc000213710 pc=0x437b5d
runtime.sigpanic()
    /usr/local/go/src/runtime/signal_unix.go:832 +0x125 fp=0xc000213790 sp=0xc000213740 pc=0x44e1c5
go.etcd.io/bbolt.(*Cursor).search(0xc0002138d0, {0xc00021392d, 0x3, 0x3}, 0xc000213870?)
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/bbolt/cursor.go:249 +0x5e fp=0xc000213828 sp=0xc000213790 pc=0xed25be
go.etcd.io/bbolt.(*Cursor).seek(0xc0002138d0, {0xc00021392d?, 0xc000640001?, 0xc00018a4b0?})
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/bbolt/cursor.go:159 +0x48 fp=0xc000213880 sp=0xc000213828 pc=0xed1fe8
go.etcd.io/bbolt.(*Bucket).Bucket(0xc00018a478, {0xc00021392d, 0x3, 0x3})
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/bbolt/bucket.go:105 +0xd1 fp=0xc000213900 sp=0xc000213880 pc=0xece9d1
go.etcd.io/bbolt.(*Tx).Bucket(...)
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/bbolt/tx.go:101
github.com/gardener/etcd-backup-restore/pkg/initializer/validator.getLatestEtcdRevision.func2(0xc000426a20?)
    /go/src/github.com/gardener/backup-restore/pkg/initializer/validator/datavalidator.go:443 +0x45 fp=0xc000213970 sp=0xc000213900 pc=0x18bd445
go.etcd.io/bbolt.(*DB).View(0xc000426a20?, 0xc000213a48)
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/bbolt/db.go:725 +0x82 fp=0xc0002139d8 sp=0xc000213970 pc=0xed5782
github.com/gardener/etcd-backup-restore/pkg/initializer/validator.getLatestEtcdRevision({0xc000426a20, 0x26})
    /go/src/github.com/gardener/backup-restore/pkg/initializer/validator/datavalidator.go:442 +0x194 fp=0xc000213ac8 sp=0xc0002139d8 pc=0x18bd2d4
github.com/gardener/etcd-backup-restore/pkg/initializer/validator.(*DataValidator).sanityCheck(0xc0001318e0, 0x1a08660?)
    /go/src/github.com/gardener/backup-restore/pkg/initializer/validator/datavalidator.go:148 +0x40b fp=0xc000213c90 sp=0xc000213ac8 pc=0x18bb0cb
github.com/gardener/etcd-backup-restore/pkg/initializer/validator.(*DataValidator).Validate(0xc0001318e0, {0x1d1fd6c, 0x4}, 0x1a08660?)
    /go/src/github.com/gardener/backup-restore/pkg/initializer/validator/datavalidator.go:72 +0x46 fp=0xc000213d30 sp=0xc000213c90 pc=0x18baa46
github.com/gardener/etcd-backup-restore/pkg/initializer.(*EtcdInitializer).Initialize(0xc00011ec18, {0x1d1fd6c, 0x4}, 0x4?)
    /go/src/github.com/gardener/backup-restore/pkg/initializer/initializer.go:69 +0x15e fp=0xc000213e80 sp=0xc000213d30 pc=0x18bdabe
github.com/gardener/etcd-backup-restore/pkg/server.(*HTTPHandler).serveInitialize.func1()
    /go/src/github.com/gardener/backup-restore/pkg/server/httpAPI.go:259 +0x438 fp=0xc000213fe0 sp=0xc000213e80 pc=0x18f2478
runtime.goexit()

full logs of etcd-backup-restore:

Logs


time="2022-12-09T09:07:20Z" level=info msg="etcd-backup-restore Version: v0.21.0"
time="2022-12-09T09:07:20Z" level=info msg="Git SHA: 3e175a87"
time="2022-12-09T09:07:20Z" level=info msg="Go Version: go1.19.2"
time="2022-12-09T09:07:20Z" level=info msg="Go OS/Arch: linux/amd64"
time="2022-12-09T09:07:20Z" level=info msg="compressionConfig:\n  enabled: true\n  policy: gzip\ndefragmentationSchedule: '*/1 * * * *'\netcdConnectionConfig:\n  caFile: /var/etcd/ssl/client/ca/ca.crt\n  certFile: /var/etcd/ssl/client/client/tls.crt\n  connectionTimeout: 5m0s\n  defragTimeout: 15m0s\n  endpoints:\n  - https://etcd-aws-local:2379\n  keyFile: /var/etcd/ssl/client/client/tls.key\n  serviceEndpoints:\n  - https://etcd-aws-client:2379\n  snapshotTimeout: 15m0s\netcdProcessName: etcd\nexponentialBackoffConfig:\n  attemptLimit: 6\n  multiplier: 2\n  thresholdTime: 2m8s\nhealthConfig:\n  deltaSnapshotLeaseName: etcd-aws-delta-snap\n  fullSnapshotLeaseName: etcd-aws-full-snap\n  heartbeatDuration: 10s\n  memberGCDuration: 1m0s\n  memberLeaseRenewalEnabled: true\n  snapshotLeaseRenewalEnabled: true\nleaderElectionConfig:\n  etcdConnectionTimeout: 5s\n  reelectionPeriod: 5s\nownerCheckConfig:\n  ownerCheckDNSCacheTTL: 1m0s\n  ownerCheckInterval: 30s\n  ownerCheckTimeout: 2m0s\nrestorationConfig:\n  MaxRequestBytes: 10485760\n  MaxTxnOps: 10240\n  autoCompactionMode: periodic\n  autoCompactionRetention: 2m\n  embeddedEtcdQuotaBytes: 2147483649\n  initialAdvertisePeerURLs:\n  - http://localhost:2380\n  initialCluster: default=http://localhost:2380\n  initialClusterToken: etcd-cluster\n  maxCallSendMsgSize: 10485760\n  maxFetchers: 6\n  name: default\n  restoreDataDir: /var/etcd/data/new.etcd\nserverConfig:\n  port: 8080\n  server-cert: /var/etcd/ssl/client/server/tls.crt\n  server-key: /var/etcd/ssl/client/server/tls.key\nsnapshotterConfig:\n  deltaSnapshotMemoryLimit: 104857600\n  deltaSnapshotPeriod: 1s\n  garbageCollectionPeriod: 5m0s\n  garbageCollectionPolicy: Exponential\n  maxBackups: 7\n  schedule: 0 */1 * * *\nsnapstoreConfig:\n  container: \"\"\n  maxParallelChunkUploads: 5\n  prefix: etcd-main/v2\n  provider: S3\n  tempDir: /var/etcd/data/temp\n"
time="2022-12-09T09:07:20Z" level=info msg="Setting status to : 503" actor=backup-restore-server
time="2022-12-09T09:07:20Z" level=info msg="Registering the http request handlers..." actor=backup-restore-server
time="2022-12-09T09:07:20Z" level=info msg="Starting the http server..." actor=backup-restore-server
time="2022-12-09T09:07:20Z" level=info msg="Attempting to promote member etcd-aws-2" actor=member-add
time="2022-12-09T09:07:20Z" level=info msg="Starting HTTP server at addr: :8080" actor=backup-restore-server
time="2022-12-09T09:07:20Z" level=info msg="TLS enabled. Starting HTTPS server." actor=backup-restore-server
{"level":"warn","ts":"2022-12-09T09:07:20.401Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-38c3c735-d13f-4407-9914-0d1c7c02a345/etcd-aws-client:2379","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: can only promote a learner member"}
time="2022-12-09T09:07:20Z" level=info msg="Member etcd-aws-2 : 15137539689457843444 already a voting member of cluster." actor=member-add
time="2022-12-09T09:07:20Z" level=info msg="Attempting to update the member Info: etcd-aws-2" actor=member-add
time="2022-12-09T09:07:20Z" level=info msg="Responding to status request with: New" actor=backup-restore-server
time="2022-12-09T09:07:20Z" level=info msg="Received start initialization request." actor=backup-restore-server
time="2022-12-09T09:07:20Z" level=info msg="Updating status from New to Progress" actor=backup-restore-server
time="2022-12-09T09:07:20Z" level=info msg="Setting status to : 503" actor=backup-restore-server
time="2022-12-09T09:07:20Z" level=info msg="Validation failBelowRevision: " actor=backup-restore-server
time="2022-12-09T09:07:20Z" level=info msg="Validation mode: full" actor=backup-restore-server
time="2022-12-09T09:07:20Z" level=info msg="Checking if member etcd-aws-2 is part of a running cluster" actor=member-add
time="2022-12-09T09:07:20Z" level=info msg="Member etcd-aws-2 part of running cluster" actor=member-add
time="2022-12-09T09:07:20Z" level=info msg="Checking for data directory structure validity..."
time="2022-12-09T09:07:20Z" level=info msg="Checking for Etcd revision..."
unexpected fault address 0x7f4e66cc9008
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0x7f4e66cc9008 pc=0xed25be]

goroutine 234 [running]:
runtime.throw({0x1d20cfd?, 0x0?})
    /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0xc000213740 sp=0xc000213710 pc=0x437b5d
runtime.sigpanic()
    /usr/local/go/src/runtime/signal_unix.go:832 +0x125 fp=0xc000213790 sp=0xc000213740 pc=0x44e1c5
go.etcd.io/bbolt.(*Cursor).search(0xc0002138d0, {0xc00021392d, 0x3, 0x3}, 0xc000213870?)
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/bbolt/cursor.go:249 +0x5e fp=0xc000213828 sp=0xc000213790 pc=0xed25be
go.etcd.io/bbolt.(*Cursor).seek(0xc0002138d0, {0xc00021392d?, 0xc000640001?, 0xc00018a4b0?})
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/bbolt/cursor.go:159 +0x48 fp=0xc000213880 sp=0xc000213828 pc=0xed1fe8
go.etcd.io/bbolt.(*Bucket).Bucket(0xc00018a478, {0xc00021392d, 0x3, 0x3})
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/bbolt/bucket.go:105 +0xd1 fp=0xc000213900 sp=0xc000213880 pc=0xece9d1
go.etcd.io/bbolt.(*Tx).Bucket(...)
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/bbolt/tx.go:101
github.com/gardener/etcd-backup-restore/pkg/initializer/validator.getLatestEtcdRevision.func2(0xc000426a20?)
    /go/src/github.com/gardener/backup-restore/pkg/initializer/validator/datavalidator.go:443 +0x45 fp=0xc000213970 sp=0xc000213900 pc=0x18bd445
go.etcd.io/bbolt.(*DB).View(0xc000426a20?, 0xc000213a48)
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/bbolt/db.go:725 +0x82 fp=0xc0002139d8 sp=0xc000213970 pc=0xed5782
github.com/gardener/etcd-backup-restore/pkg/initializer/validator.getLatestEtcdRevision({0xc000426a20, 0x26})
    /go/src/github.com/gardener/backup-restore/pkg/initializer/validator/datavalidator.go:442 +0x194 fp=0xc000213ac8 sp=0xc0002139d8 pc=0x18bd2d4
github.com/gardener/etcd-backup-restore/pkg/initializer/validator.(*DataValidator).sanityCheck(0xc0001318e0, 0x1a08660?)
    /go/src/github.com/gardener/backup-restore/pkg/initializer/validator/datavalidator.go:148 +0x40b fp=0xc000213c90 sp=0xc000213ac8 pc=0x18bb0cb
github.com/gardener/etcd-backup-restore/pkg/initializer/validator.(*DataValidator).Validate(0xc0001318e0, {0x1d1fd6c, 0x4}, 0x1a08660?)
    /go/src/github.com/gardener/backup-restore/pkg/initializer/validator/datavalidator.go:72 +0x46 fp=0xc000213d30 sp=0xc000213c90 pc=0x18baa46
github.com/gardener/etcd-backup-restore/pkg/initializer.(*EtcdInitializer).Initialize(0xc00011ec18, {0x1d1fd6c, 0x4}, 0x4?)
    /go/src/github.com/gardener/backup-restore/pkg/initializer/initializer.go:69 +0x15e fp=0xc000213e80 sp=0xc000213d30 pc=0x18bdabe
github.com/gardener/etcd-backup-restore/pkg/server.(*HTTPHandler).serveInitialize.func1()
    /go/src/github.com/gardener/backup-restore/pkg/server/httpAPI.go:259 +0x438 fp=0xc000213fe0 sp=0xc000213e80 pc=0x18f2478
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000213fe8 sp=0xc000213fe0 pc=0x46bac1
created by github.com/gardener/etcd-backup-restore/pkg/server.(*HTTPHandler).serveInitialize
    /go/src/github.com/gardener/backup-restore/pkg/server/httpAPI.go:231 +0x23f

goroutine 1 [select]:
runtime.gopark(0xc000a9ed18?, 0x2?, 0x9d?, 0xac?, 0xc000a9ec64?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000a9ead0 sp=0xc000a9eab0 pc=0x43a776
runtime.selectgo(0xc000a9ed18, 0xc000a9ec60, 0xc0004c4630?, 0x0, 0xc0002080c0?, 0x1)
    /usr/local/go/src/runtime/select.go:328 +0x7bc fp=0xc000a9ec10 sp=0xc000a9ead0 pc=0x44a5fc
google.golang.org/grpc.(*pickerWrapper).pick(0xc000969bc0, {0x20bf6d8, 0xc0002080c0}, 0x0, {{0x1d55fbf?, 0xc000a9ed88?}, {0x20bf6d8?, 0xc0002080c0?}})
    /go/src/github.com/gardener/backup-restore/vendor/google.golang.org/grpc/picker_wrapper.go:152 +0x215 fp=0xc000a9ed48 sp=0xc000a9ec10 pc=0x960b75
google.golang.org/grpc.(*ClientConn).getTransport(0x30?, {0x20bf6d8?, 0xc0002080c0?}, 0x0?, {0x1d55fbf?, 0x7f4e66fca378?})
    /go/src/github.com/gardener/backup-restore/vendor/google.golang.org/grpc/clientconn.go:897 +0x35 fp=0xc000a9ed98 sp=0xc000a9ed48 pc=0x95bab5
google.golang.org/grpc.(*clientStream).newAttemptLocked(0xc000539d40, {0x0?, 0x0}, 0x0)
    /go/src/github.com/gardener/backup-restore/vendor/google.golang.org/grpc/stream.go:351 +0x1b0 fp=0xc000a9ee38 sp=0xc000a9ed98 pc=0x9737f0
google.golang.org/grpc.newClientStream({0x20bf6d8, 0xc000969f50}, 0x2ea2080, 0xc00057a380, {0x1d55fbf, 0x20}, {0xc0005a8b40, 0x3, 0x4})
    /go/src/github.com/gardener/backup-restore/vendor/google.golang.org/grpc/stream.go:284 +0x104c fp=0xc000a9f090 sp=0xc000a9ee38 pc=0x972d6c
google.golang.org/grpc.invoke({0x20bf6d8?, 0xc000969f50?}, {0x1d55fbf?, 0x0?}, {0x1bcd600, 0x2edc018}, {0x1c073c0, 0xc0005e84c0}, 0x1acaae0?, {0xc0005a8b40, ...})
    /go/src/github.com/gardener/backup-restore/vendor/google.golang.org/grpc/call.go:66 +0x7d fp=0xc000a9f0f8 sp=0xc000a9f090 pc=0x9560bd
go.etcd.io/etcd/clientv3.(*Client).unaryClientInterceptor.func1({0x20bf6a0?, 0xc000888720?}, {0x1d55fbf, 0x20}, {0x1bcd600, 0x2edc018}, {0x1c073c0, 0xc0005e84c0}, 0xc00057a380, 0x1e2bf18, ...)
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/etcd/clientv3/retry_interceptor.go:58 +0x437 fp=0xc000a9f478 sp=0xc000a9f0f8 pc=0xa61d77
google.golang.org/grpc.(*ClientConn).Invoke(0x60?, {0x20bf6a0?, 0xc000888720?}, {0x1d55fbf?, 0x6?}, {0x1bcd600?, 0x2edc018?}, {0x1c073c0?, 0xc0005e84c0?}, {0xc000888780, ...})
    /go/src/github.com/gardener/backup-restore/vendor/google.golang.org/grpc/call.go:35 +0x21f fp=0xc000a9f518 sp=0xc000a9f478 pc=0x955f5f
google.golang.org/grpc.Invoke(...)
    /go/src/github.com/gardener/backup-restore/vendor/google.golang.org/grpc/call.go:60
go.etcd.io/etcd/etcdserver/etcdserverpb.(*clusterClient).MemberList(0xc00038c350?, {0x20bf6a0?, 0xc000888720?}, 0x2ea9120?, {0xc000888780?, 0x4?, 0x6?})
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:4063 +0xc9 fp=0xc000a9f598 sp=0xc000a9f518 pc=0x99b9e9
go.etcd.io/etcd/clientv3.(*retryClusterClient).MemberList(0xc0004c43f0, {0x20bf6a0, 0xc000888720}, 0xc000a9f5c8?, {0x2ea8720, 0x3, 0x3})
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/etcd/clientv3/retry.go:167 +0x133 fp=0xc000a9f600 sp=0xc000a9f598 pc=0xa5fb93
go.etcd.io/etcd/clientv3.(*cluster).MemberList(0x6fc23ac00?, {0x20bf6a0, 0xc000888720})
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/etcd/clientv3/cluster.go:127 +0x42 fp=0xc000a9f648 sp=0xc000a9f600 pc=0xa55142
go.etcd.io/etcd/clientv3.(*Client).MemberList(0x20bf630?, {0x20bf6a0?, 0xc000888720?})
    :1 +0x34 fp=0xc000a9f670 sp=0xc000a9f648 pc=0xa6fd34
github.com/gardener/etcd-backup-restore/pkg/member.(*memberControl).UpdateMemberPeerURL(0xc000940000, {0x20bf630, 0xc0005a9080}, {0x20c6b40, 0xc000583d40})
    /go/src/github.com/gardener/backup-restore/pkg/member/member_control.go:251 +0x12b fp=0xc000a9f718 sp=0xc000a9f670 pc=0x18a0f4b
github.com/gardener/etcd-backup-restore/pkg/server.(*BackupRestoreServer).runServer.func1()
    /go/src/github.com/gardener/backup-restore/pkg/server/backuprestoreserver.go:222 +0x129 fp=0xc000a9f8b8 sp=0xc000a9f718 pc=0x18ee4a9
k8s.io/client-go/util/retry.OnError.func1()
    /go/src/github.com/gardener/backup-restore/vendor/k8s.io/client-go/util/retry/util.go:51 +0x33 fp=0xc000a9f8f8 sp=0xc000a9f8b8 pc=0x189bef3
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection(0xc000389490?)
    /go/src/github.com/gardener/backup-restore/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:211 +0x4d fp=0xc000a9f928 sp=0xc000a9f8f8 pc=0x17774ed
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff({0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0}, 0xd18c2e2800?)
    /go/src/github.com/gardener/backup-restore/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:399 +0x5f fp=0xc000a9f948 sp=0xc000a9f928 pc=0x1777a1f
k8s.io/client-go/util/retry.OnError({0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0}, 0xc000216420?, 0xc000942d70?)
    /go/src/github.com/gardener/backup-restore/vendor/k8s.io/client-go/util/retry/util.go:50 +0xa8 fp=0xc000a9f9c8 sp=0xc000a9f948 pc=0x189be28
github.com/gardener/etcd-backup-restore/pkg/server.(*BackupRestoreServer).runServer(0xc000943040, {0x20bf630?, 0xc0005a9080}, 0xc000687320)
    /go/src/github.com/gardener/backup-restore/pkg/server/backuprestoreserver.go:217 +0x311 fp=0xc000a9fb08 sp=0xc000a9f9c8 pc=0x18ece51
github.com/gardener/etcd-backup-restore/pkg/server.(*BackupRestoreServer).Run(0xc000943040, {0x20bf630, 0xc0005a9080})
    /go/src/github.com/gardener/backup-restore/pkg/server/backuprestoreserver.go:141 +0x47f fp=0xc000a9fcf8 sp=0xc000a9fb08 pc=0x18ec5df
github.com/gardener/etcd-backup-restore/cmd.(*serverOptions).run(0xc000438700?, {0x20bf630, 0xc0005a9080})
    /go/src/github.com/gardener/backup-restore/cmd/options.go:93 +0x55 fp=0xc000a9fd20 sp=0xc000a9fcf8 pc=0x18f90b5
github.com/gardener/etcd-backup-restore/cmd.NewServerCommand.func1(0xc0004e1340?, {0x1d1fdcc?, 0x21?, 0x24?})
    /go/src/github.com/gardener/backup-restore/cmd/server.go:54 +0x1c5 fp=0xc000a9fda8 sp=0xc000a9fd20 pc=0x18fa945
github.com/spf13/cobra.(*Command).execute(0xc0004e1340, {0xc00039d440, 0x21, 0x24})
    /go/src/github.com/gardener/backup-restore/vendor/github.com/spf13/cobra/command.go:854 +0x663 fp=0xc000a9fe80 sp=0xc000a9fda8 pc=0x18af6e3
github.com/spf13/cobra.(*Command).ExecuteC(0xc0004e0580)
    /go/src/github.com/gardener/backup-restore/vendor/github.com/spf13/cobra/command.go:958 +0x39d fp=0xc000a9ff38 sp=0xc000a9fe80 pc=0x18afcdd
github.com/spf13/cobra.(*Command).Execute(...)
    /go/src/github.com/gardener/backup-restore/vendor/github.com/spf13/cobra/command.go:895
main.main()
    /go/src/github.com/gardener/backup-restore/main.go:37 +0x4a fp=0xc000a9ff80 sp=0xc000a9ff38 pc=0x18fb64a
runtime.main()
    /usr/local/go/src/runtime/proc.go:250 +0x212 fp=0xc000a9ffe0 sp=0xc000a9ff80 pc=0x43a3b2
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000a9ffe8 sp=0xc000a9ffe0 pc=0x46bac1

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00008efb0 sp=0xc00008ef90 pc=0x43a776
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:369
runtime.forcegchelper()
    /usr/local/go/src/runtime/proc.go:302 +0xad fp=0xc00008efe0 sp=0xc00008efb0 pc=0x43a60d
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00008efe8 sp=0xc00008efe0 pc=0x46bac1
created by runtime.init.6
    /usr/local/go/src/runtime/proc.go:290 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00008f790 sp=0xc00008f770 pc=0x43a776
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:369
runtime.bgsweep(0x0?)
    /usr/local/go/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc00008f7c8 sp=0xc00008f790 pc=0x4251b7
runtime.gcenable.func1()
    /usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc00008f7e0 sp=0xc00008f7c8 pc=0x419e06
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00008f7e8 sp=0xc00008f7e0 pc=0x46bac1
created by runtime.gcenable
    /usr/local/go/src/runtime/mgc.go:178 +0x6b

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc0000ae000?, 0x209e420?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00008ff70 sp=0xc00008ff50 pc=0x43a776
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:369
runtime.(*scavengerState).park(0x2eaabc0)
    /usr/local/go/src/runtime/mgcscavenge.go:389 +0x53 fp=0xc00008ffa0 sp=0xc00008ff70 pc=0x423213
runtime.bgscavenge(0x0?)
    /usr/local/go/src/runtime/mgcscavenge.go:622 +0x65 fp=0xc00008ffc8 sp=0xc00008ffa0 pc=0x423805
runtime.gcenable.func2()
    /usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc00008ffe0 sp=0xc00008ffc8 pc=0x419da6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00008ffe8 sp=0xc00008ffe0 pc=0x46bac1
created by runtime.gcenable
    /usr/local/go/src/runtime/mgc.go:179 +0xaa

goroutine 5 [finalizer wait]:
runtime.gopark(0x0?, 0x1e2da18?, 0x0?, 0xa2?, 0x2000000020?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00008e628 sp=0xc00008e608 pc=0x43a776
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:369
runtime.runfinq()
    /usr/local/go/src/runtime/mfinal.go:180 +0x10f fp=0xc00008e7e0 sp=0xc00008e628 pc=0x418f0f
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00008e7e8 sp=0xc00008e7e0 pc=0x46bac1
created by runtime.createfing
    /usr/local/go/src/runtime/mfinal.go:157 +0x45

goroutine 23 [GC worker (idle)]:
runtime.gopark(0x405a7d?, 0xc0000b3ce0?, 0xa0?, 0x7?, 0xc0000907a8?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000090750 sp=0xc000090730 pc=0x43a776
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0000907e0 sp=0xc000090750 pc=0x41bf51
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000907e8 sp=0xc0000907e0 pc=0x46bac1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 15 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000090f50 sp=0xc000090f30 pc=0x43a776
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000090fe0 sp=0xc000090f50 pc=0x41bf51
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000090fe8 sp=0xc000090fe0 pc=0x46bac1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 39 [GC worker (idle)]:
runtime.gopark(0x31a4e8d4b5bdb?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00008a750 sp=0xc00008a730 pc=0x43a776
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00008a7e0 sp=0xc00008a750 pc=0x41bf51
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00008a7e8 sp=0xc00008a7e0 pc=0x46bac1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 24 [GC worker (idle)]:
runtime.gopark(0x31a4e8d4c0b7a?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000532750 sp=0xc000532730 pc=0x43a776
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0005327e0 sp=0xc000532750 pc=0x41bf51
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0005327e8 sp=0xc0005327e0 pc=0x46bac1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 16 [GC worker (idle)]:
runtime.gopark(0x31a4e8d4c10de?, 0x3?, 0xd5?, 0xe?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000091750 sp=0xc000091730 pc=0x43a776
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0000917e0 sp=0xc000091750 pc=0x41bf51
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000917e8 sp=0xc0000917e0 pc=0x46bac1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 40 [GC worker (idle)]:
runtime.gopark(0x31a4e8d4afa34?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00008af50 sp=0xc00008af30 pc=0x43a776
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00008afe0 sp=0xc00008af50 pc=0x41bf51
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00008afe8 sp=0xc00008afe0 pc=0x46bac1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 25 [GC worker (idle)]:
runtime.gopark(0x2edd880?, 0x1?, 0x5c?, 0x6?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000532f50 sp=0xc000532f30 pc=0x43a776
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000532fe0 sp=0xc000532f50 pc=0x41bf51
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000532fe8 sp=0xc000532fe0 pc=0x46bac1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 49 [GC worker (idle)]:
runtime.gopark(0x31a4e8d4bc033?, 0x3?, 0xb5?, 0x88?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000091f50 sp=0xc000091f30 pc=0x43a776
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000091fe0 sp=0xc000091f50 pc=0x41bf51
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000091fe8 sp=0xc000091fe0 pc=0x46bac1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 41 [GC worker (idle)]:
runtime.gopark(0x31a4e8d4c241e?, 0x3?, 0xe5?, 0x93?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00008b750 sp=0xc00008b730 pc=0x43a776
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00008b7e0 sp=0xc00008b750 pc=0x41bf51
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00008b7e8 sp=0xc00008b7e0 pc=0x46bac1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 26 [GC worker (idle)]:
runtime.gopark(0x31a4e8d4aec59?, 0x1?, 0x92?, 0x3e?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000533750 sp=0xc000533730 pc=0x43a776
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0005337e0 sp=0xc000533750 pc=0x41bf51
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0005337e8 sp=0xc0005337e0 pc=0x46bac1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 50 [GC worker (idle)]:
runtime.gopark(0x31a4e8d4afa36?, 0x1?, 0x10?, 0xc7?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00052e750 sp=0xc00052e730 pc=0x43a776
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00052e7e0 sp=0xc00052e750 pc=0x41bf51
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00052e7e8 sp=0xc00052e7e0 pc=0x46bac1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 42 [GC worker (idle)]:
runtime.gopark(0x31a4e8d4b5945?, 0x3?, 0x3e?, 0x32?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00008bf50 sp=0xc00008bf30 pc=0x43a776
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00008bfe0 sp=0xc00008bf50 pc=0x41bf51
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00008bfe8 sp=0xc00008bfe0 pc=0x46bac1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 27 [GC worker (idle)]:
runtime.gopark(0x31a4e8d4b5368?, 0x1?, 0x8a?, 0x16?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000533f50 sp=0xc000533f30 pc=0x43a776
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000533fe0 sp=0xc000533f50 pc=0x41bf51
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000533fe8 sp=0xc000533fe0 pc=0x46bac1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 43 [GC worker (idle)]:
runtime.gopark(0x31a4e8d4afac8?, 0x1?, 0x93?, 0xa0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00008c750 sp=0xc00008c730 pc=0x43a776
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00008c7e0 sp=0xc00008c750 pc=0x41bf51
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00008c7e8 sp=0xc00008c7e0 pc=0x46bac1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 51 [GC worker (idle)]:
runtime.gopark(0x31a4e8d4aeaac?, 0x1?, 0x4f?, 0x72?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00052ef50 sp=0xc00052ef30 pc=0x43a776
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00052efe0 sp=0xc00052ef50 pc=0x41bf51
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00052efe8 sp=0xc00052efe0 pc=0x46bac1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 28 [GC worker (idle)]:
runtime.gopark(0x31a4e8d4afc05?, 0x3?, 0x7b?, 0x7c?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000534750 sp=0xc000534730 pc=0x43a776
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0005347e0 sp=0xc000534750 pc=0x41bf51
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0005347e8 sp=0xc0005347e0 pc=0x46bac1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 65 [select]:
runtime.gopark(0xc0005a4788?, 0x3?, 0x0?, 0xb9?, 0xc0005a4772?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000c0df8 sp=0xc0000c0dd8 pc=0x43a776
runtime.selectgo(0xc0000c0f88, 0xc0005a476c, 0xc000596050?, 0x0, 0x0?, 0x1)
    /usr/local/go/src/runtime/select.go:328 +0x7bc fp=0xc0000c0f38 sp=0xc0000c0df8 pc=0x44a5fc
go.opencensus.io/stats/view.(*worker).start(0xc000596050)
    /go/src/github.com/gardener/backup-restore/vendor/go.opencensus.io/stats/view/worker.go:154 +0xad fp=0xc0000c0fc8 sp=0xc0000c0f38 pc=0xb8446d
go.opencensus.io/stats/view.init.0.func1()
    /go/src/github.com/gardener/backup-restore/vendor/go.opencensus.io/stats/view/worker.go:32 +0x26 fp=0xc0000c0fe0 sp=0xc0000c0fc8 pc=0xb84086
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000c0fe8 sp=0xc0000c0fe0 pc=0x46bac1
created by go.opencensus.io/stats/view.init.0
    /go/src/github.com/gardener/backup-restore/vendor/go.opencensus.io/stats/view/worker.go:32 +0x85

goroutine 168 [chan receive]:
runtime.gopark(0xc00012b138?, 0x1?, 0x10?, 0x56?, 0x457485?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0005355b8 sp=0xc000535598 pc=0x43a776
runtime.chanrecv(0xc0001156e0, 0xc000535728, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc000535648 sp=0xc0005355b8 pc=0x406cdb
runtime.chanrecv2(0x3b9aca00?, 0x43a860?)
    /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc000535670 sp=0xc000535648 pc=0x406818
go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc000011c80)
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/etcd/pkg/logutil/merge_logger.go:173 +0x8c fp=0xc0005357c8 sp=0xc000535670 pc=0xa478cc
go.etcd.io/etcd/pkg/logutil.NewMergeLogger.func1()
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/etcd/pkg/logutil/merge_logger.go:91 +0x26 fp=0xc0005357e0 sp=0xc0005357c8 pc=0xa47406
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0005357e8 sp=0xc0005357e0 pc=0x46bac1
created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/etcd/pkg/logutil/merge_logger.go:91 +0xb6

goroutine 225 [select]:
runtime.gopark(0xc0003d2f38?, 0x2?, 0xd0?, 0x9b?, 0xc0003d2f04?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0003d2d88 sp=0xc0003d2d68 pc=0x43a776
runtime.selectgo(0xc0003d2f38, 0xc0003d2f00, 0xc0004c43d0?, 0x0, 0xc000535fa8?, 0x1)
    /usr/local/go/src/runtime/select.go:328 +0x7bc fp=0xc0003d2ec8 sp=0xc0003d2d88 pc=0x44a5fc
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0005a8980)
    /go/src/github.com/gardener/backup-restore/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:69 +0x8f fp=0xc0003d2fc8 sp=0xc0003d2ec8 pc=0x9544cf
google.golang.org/grpc.newCCBalancerWrapper.func1()
    /go/src/github.com/gardener/backup-restore/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:60 +0x26 fp=0xc0003d2fe0 sp=0xc0003d2fc8 pc=0x954406
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0003d2fe8 sp=0xc0003d2fe0 pc=0x46bac1
created by google.golang.org/grpc.newCCBalancerWrapper
    /go/src/github.com/gardener/backup-restore/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:60 +0x1ca

goroutine 165 [chan receive]:
runtime.gopark(0x4e8e65f108?, 0x0?, 0x10?, 0xf6?, 0x457485?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00052f5b8 sp=0xc00052f598 pc=0x43a776
runtime.chanrecv(0xc000115620, 0xc00052f728, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc00052f648 sp=0xc00052f5b8 pc=0x406cdb
runtime.chanrecv2(0x3b9aca00?, 0x43a860?)
    /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc00052f670 sp=0xc00052f648 pc=0x406818
go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc000011b48)
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/etcd/pkg/logutil/merge_logger.go:173 +0x8c fp=0xc00052f7c8 sp=0xc00052f670 pc=0xa478cc
go.etcd.io/etcd/pkg/logutil.NewMergeLogger.func1()
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/etcd/pkg/logutil/merge_logger.go:91 +0x26 fp=0xc00052f7e0 sp=0xc00052f7c8 pc=0xa47406
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00052f7e8 sp=0xc00052f7e0 pc=0x46bac1
created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/etcd/pkg/logutil/merge_logger.go:91 +0xb6

goroutine 93 [chan receive]:
runtime.gopark(0xc00052fdc8?, 0x4405b4?, 0x10?, 0xfe?, 0x457485?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00052fdb8 sp=0xc00052fd98 pc=0x43a776
runtime.chanrecv(0xc000592600, 0xc00052ff28, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc00052fe48 sp=0xc00052fdb8 pc=0x406cdb
runtime.chanrecv2(0x3b9aca00?, 0x43a860?)
    /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc00052fe70 sp=0xc00052fe48 pc=0x406818
go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc000382918)
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/etcd/pkg/logutil/merge_logger.go:173 +0x8c fp=0xc00052ffc8 sp=0xc00052fe70 pc=0xa478cc
go.etcd.io/etcd/pkg/logutil.NewMergeLogger.func1()
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/etcd/pkg/logutil/merge_logger.go:91 +0x26 fp=0xc00052ffe0 sp=0xc00052ffc8 pc=0xa47406
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00052ffe8 sp=0xc00052ffe0 pc=0x46bac1
created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
    /go/src/github.com/gardener/backup-restore/vendor/go.etcd.io/etcd/pkg/logutil/merge_logger.go:91 +0xb6

goroutine 144 [chan receive]:
runtime.gopark(0xc00012b138?, 0x2?, 0x20?, 0x7?, 0x457485?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0002406c8 sp=0xc0002406a8 pc=0x43a776
runtime.chanrecv(0xc0003a00c0, 0xc0002407a0, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc000240758 sp=0xc0002406c8 pc=0x406cdb
runtime.chanrecv2(0x12a05f200?, 0x0?)
    /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc000240780 sp=0xc000240758 pc=0x406818
k8s.io/klog/v2.(*loggingT).flushDaemon(0xc000114420?)
    /go/src/github.com/gardener/backup-restore/vendor/k8s.io/klog/v2/klog.go:1169 +0x6a fp=0xc0002407c8 sp=0xc000240780 pc=0x116ef4a
k8s.io/klog/v2.init.0.func1()
    /go/src/github.com/gardener/backup-restore/vendor/k8s.io/klog/v2/klog.go:417 +0x26 fp=0xc0002407e0 sp=0xc0002407c8 pc=0x116c986
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0002407e8 sp=0xc0002407e0 pc=0x46bac1
created by k8s.io/klog/v2.init.0
    /go/src/github.com/gardener/backup-restore/vendor/k8s.io/klog/v2/klog.go:417 +0xf6

goroutine 154 [sleep]:
runtime.gopark(0x31a4e8cb9bfcf?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000530758 sp=0xc000530738 pc=0x43a776
time.Sleep(0x6fc23ac00)
    /usr/local/go/src/runtime/time.go:195 +0x135 fp=0xc000530798 sp=0xc000530758 pc=0x468835
sigs.k8s.io/controller-runtime/pkg/log.init.0.func1()
    /go/src/github.com/gardener/backup-restore/vendor/sigs.k8s.io/controller-runtime/pkg/log/log.go:63 +0x38 fp=0xc0005307e0 sp=0xc000530798 pc=0x1897638
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0005307e8 sp=0xc0005307e0 pc=0x46bac1
created by sigs.k8s.io/controller-runtime/pkg/log.init.0
    /go/src/github.com/gardener/backup-restore/vendor/sigs.k8s.io/controller-runtime/pkg/log/log.go:62 +0x25

goroutine 155 [select, locked to thread]:
runtime.gopark(0xc000530fa8?, 0x2?, 0x2e?, 0x9a?, 0xc000530fa4?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000530e18 sp=0xc000530df8 pc=0x43a776
runtime.selectgo(0xc000530fa8, 0xc000530fa0, 0x0?, 0x0, 0x0?, 0x1)
    /usr/local/go/src/runtime/select.go:328 +0x7bc fp=0xc000530f58 sp=0xc000530e18 pc=0x44a5fc
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal_unix.go:991 +0x1b0 fp=0xc000530fe0 sp=0xc000530f58 pc=0x44e8b0
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000530fe8 sp=0xc000530fe0 pc=0x46bac1
created by runtime.ensureSigM
    /usr/local/go/src/runtime/signal_unix.go:974 +0xbd

goroutine 193 [syscall]:
runtime.notetsleepg(0x0?, 0x0?)
    /usr/local/go/src/runtime/lock_futex.go:236 +0x34 fp=0xc000241fa0 sp=0xc000241f68 pc=0x40c274
os/signal.signal_recv()
    /usr/local/go/src/runtime/sigqueue.go:152 +0x2f fp=0xc000241fc0 sp=0xc000241fa0 pc=0x467fef
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:23 +0x19 fp=0xc000241fe0 sp=0xc000241fc0 pc=0x4f85b9
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000241fe8 sp=0xc000241fe0 pc=0x46bac1
created by os/signal.Notify.func1.1
    /usr/local/go/src/os/signal/signal.go:151 +0x2a

goroutine 194 [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0002426f8 sp=0xc0002426d8 pc=0x43a776
runtime.chanrecv(0xc000888120, 0x0, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc000242788 sp=0xc0002426f8 pc=0x406cdb
runtime.chanrecv1(0x0?, 0x0?)
    /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc0002427b0 sp=0xc000242788 pc=0x4067d8
main.setupSignalHandler.func1()
    /go/src/github.com/gardener/backup-restore/main.go:53 +0x2d fp=0xc0002427e0 sp=0xc0002427b0 pc=0x18fb80d
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0002427e8 sp=0xc0002427e0 pc=0x46bac1
created by main.setupSignalHandler
    /go/src/github.com/gardener/backup-restore/main.go:52 +0xff

goroutine 197 [IO wait]:
runtime.gopark(0x200000003?, 0xc00029fd40?, 0x0?, 0x0?, 0xc000601a08?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000601998 sp=0xc000601978 pc=0x43a776
runtime.netpollblock(0xc0006019e8?, 0x601a60?, 0xc0?)
    /usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0xc0006019d0 sp=0xc000601998 pc=0x433297
internal/poll.runtime_pollWait(0x7f4e67076ef8, 0x72)
    /usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0xc0006019f0 sp=0xc0006019d0 pc=0x465fc9
internal/poll.(*pollDesc).wait(0xc0005ef280?, 0x6?, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000601a18 sp=0xc0006019f0 pc=0x4d46b2
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0005ef280)
    /usr/local/go/src/internal/poll/fd_unix.go:614 +0x234 fp=0xc000601ab0 sp=0xc000601a18 pc=0x4d9934
net.(*netFD).accept(0xc0005ef280)
    /usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc000601b68 sp=0xc000601ab0 pc=0x5713d5
net.(*TCPListener).accept(0xc0003822d0)
    /usr/local/go/src/net/tcpsock_posix.go:142 +0x28 fp=0xc000601b98 sp=0xc000601b68 pc=0x58a328
net.(*TCPListener).Accept(0xc0003822d0)
    /usr/local/go/src/net/tcpsock.go:288 +0x3d fp=0xc000601bc8 sp=0xc000601b98 pc=0x5894fd
crypto/tls.(*listener).Accept(0xc0003823d8)
    /usr/local/go/src/crypto/tls/tls.go:66 +0x2d fp=0xc000601c08 sp=0xc000601bc8 pc=0x6c2fcd
net/http.(*onceCloseListener).Accept(0xc00042e280?)
    :1 +0x2a fp=0xc000601c20 sp=0xc000601c08 pc=0x8ad10a
net/http.(*Server).Serve(0xc00019e4b0, {0x20b14d8, 0xc0003823d8})
    /usr/local/go/src/net/http/server.go:3070 +0x385 fp=0xc000601d50 sp=0xc000601c20 pc=0x886fe5
net/http.(*Server).ServeTLS(0xc00019e4b0, {0x20be2b8?, 0xc0003822d0}, {0x7ffdd774a88e, 0x23}, {0x7ffdd774a8bf, 0x23})
    /usr/local/go/src/net/http/server.go:3142 +0x3fd fp=0xc000601eb8 sp=0xc000601d50 pc=0x8878bd
net/http.(*Server).ListenAndServeTLS(0xc00019e4b0, {0x7ffdd774a88e, 0x23}, {0x7ffdd774a8bf, 0x23})
    /usr/local/go/src/net/http/server.go:3299 +0x12f fp=0xc000601f38 sp=0xc000601eb8 pc=0x88816f
github.com/gardener/etcd-backup-restore/pkg/server.(*HTTPHandler).Start(0xc000552d00)
    /go/src/github.com/gardener/backup-restore/pkg/server/httpAPI.go:193 +0xf2 fp=0xc000601fc8 sp=0xc000601f38 pc=0x18f1a32
github.com/gardener/etcd-backup-restore/pkg/server.(*BackupRestoreServer).startHTTPServer.func1()
    /go/src/github.com/gardener/backup-restore/pkg/server/backuprestoreserver.go:170 +0x26 fp=0xc000601fe0 sp=0xc000601fc8 pc=0x18ecb06
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000601fe8 sp=0xc000601fe0 pc=0x46bac1
created by github.com/gardener/etcd-backup-restore/pkg/server.(*BackupRestoreServer).startHTTPServer
    /go/src/github.com/gardener/backup-restore/pkg/server/backuprestoreserver.go:170 +0x44a

goroutine 226 [select]:
runtime.gopark(0xc000895f08?, 0x3?, 0x0?, 0x30?, 0xc000895ea2?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000895ca0 sp=0xc000895c80 pc=0x43a776
runtime.selectgo(0xc000895f08, 0xc000895e9c, 0x20a8d00?, 0x0, 0x0?, 0x1)
    /usr/local/go/src/runtime/select.go:328 +0x7bc fp=0xc000895de0 sp=0xc000895ca0 pc=0x44a5fc
google.golang.org/grpc.(*addrConn).resetTransport(0xc00087a000)
    /go/src/github.com/gardener/backup-restore/vendor/google.golang.org/grpc/clientconn.go:1149 +0x605 fp=0xc000895fc8 sp=0xc000895de0 pc=0x95cd65
google.golang.org/grpc.(*addrConn).connect.func1()
    /go/src/github.com/gardener/backup-restore/vendor/google.golang.org/grpc/clientconn.go:815 +0x26 fp=0xc000895fe0 sp=0xc000895fc8 pc=0x95b066
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000895fe8 sp=0xc000895fe0 pc=0x46bac1
created by google.golang.org/grpc.(*addrConn).connect
    /go/src/github.com/gardener/backup-restore/vendor/google.golang.org/grpc/clientconn.go:815 +0x136

goroutine 158 [IO wait]:
runtime.gopark(0x40?, 0xb?, 0x0?, 0x0?, 0x9?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000899068 sp=0xc000899048 pc=0x43a776
runtime.netpollblock(0x47fca5?, 0x2692da0?, 0x6b?)
    /usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0xc0008990a0 sp=0xc000899068 pc=0x433297
internal/poll.runtime_pollWait(0x7f4e67076d18, 0x72)
    /usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0xc0008990c0 sp=0xc0008990a0 pc=0x465fc9
internal/poll.(*pollDesc).wait(0xc000394580?, 0xc0004c7200?, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0008990e8 sp=0xc0008990c0 pc=0x4d46b2
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000394580, {0xc0004c7200, 0x240, 0x240})
    /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc000899168 sp=0xc0008990e8 pc=0x4d5a1a
net.(*netFD).Read(0xc000394580, {0xc0004c7200?, 0x53380d134d2c6dfc?, 0x2a8c93f9072d60b4?})
    /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc0008991b0 sp=0xc000899168 pc=0x56f2c9
net.(*conn).Read(0xc000422050, {0xc0004c7200?, 0xb4bd5a6704d544ab?, 0xe4f828e5fcfb22af?})
    /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0008991f8 sp=0xc0008991b0 pc=0x581065
crypto/tls.(*atLeastReader).Read(0xc000594ba0, {0xc0004c7200?, 0x0?, 0x4ed8aa4a391c0cb3?})
    /usr/local/go/src/crypto/tls/conn.go:787 +0x3d fp=0xc000899240 sp=0xc0008991f8 pc=0x68dd9d
bytes.(*Buffer).ReadFrom(0xc000205078, {0x20a2aa0, 0xc000594ba0})
    /usr/local/go/src/bytes/buffer.go:202 +0x98 fp=0xc000899298 sp=0xc000899240 pc=0x5017d8
crypto/tls.(*Conn).readFromUntil(0xc000204e00, {0x20a6ea0?, 0xc000422050}, 0xd5a79147c6e00bf3?)
    /usr/local/go/src/crypto/tls/conn.go:809 +0xe5 fp=0xc0008992d8 sp=0xc000899298 pc=0x68df85
crypto/tls.(*Conn).readRecordOrCCS(0xc000204e00, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:616 +0x116 fp=0xc000899660 sp=0xc0008992d8 pc=0x68b3d6
crypto/tls.(*Conn).readRecord(...)
    /usr/local/go/src/crypto/tls/conn.go:582
crypto/tls.(*Conn).Read(0xc000204e00, {0xc000124000, 0x1000, 0x20?})
    /usr/local/go/src/crypto/tls/conn.go:1287 +0x16f fp=0xc0008996d0 sp=0xc000899660 pc=0x69102f
net/http.(*connReader).Read(0xc00042d020, {0xc000124000, 0x1000, 0x1000})
    /usr/local/go/src/net/http/server.go:786 +0x171 fp=0xc000899720 sp=0xc0008996d0 pc=0x87c3d1
bufio.(*Reader).fill(0xc0004369c0)
    /usr/local/go/src/bufio/bufio.go:106 +0xff fp=0xc000899758 sp=0xc000899720 pc=0x505f9f
bufio.(*Reader).ReadSlice(0xc0004369c0, 0x0?)
    /usr/local/go/src/bufio/bufio.go:372 +0x2f fp=0xc0008997a8 sp=0xc000899758 pc=0x506b8f
bufio.(*Reader).ReadLine(0xc0004369c0)
    /usr/local/go/src/bufio/bufio.go:401 +0x27 fp=0xc0008997f8 sp=0xc0008997a8 pc=0x506de7
net/textproto.(*Reader).readLineSlice(0xc00042d050)
    /usr/local/go/src/net/textproto/reader.go:56 +0x99 fp=0xc0008998e8 sp=0xc0008997f8 pc=0x814019
net/textproto.(*Reader).ReadLine(...)
    /usr/local/go/src/net/textproto/reader.go:37
net/http.readRequest(0x0?)
    /usr/local/go/src/net/http/request.go:1036 +0x79 fp=0xc0008999f0 sp=0xc0008998e8 pc=0x877759
net/http.(*conn).readRequest(0xc00042e280, {0x20bf630, 0xc000432480})
    /usr/local/go/src/net/http/server.go:994 +0x24a fp=0xc000899b78 sp=0xc0008999f0 pc=0x87d96a
net/http.(*conn).serve(0xc00042e280, {0x20bf6d8, 0xc000969920})
    /usr/local/go/src/net/http/server.go:1916 +0x345 fp=0xc000899fb8 sp=0xc000899b78 pc=0x881ea5
net/http.(*Server).Serve.func3()
    /usr/local/go/src/net/http/server.go:3102 +0x2e fp=0xc000899fe0 sp=0xc000899fb8 pc=0x8873ce
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000899fe8 sp=0xc000899fe0 pc=0x46bac1
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:3102 +0x4db

Environment (please complete the following information):

Etcd version/commit ID :
Etcd-druid version/commit ID :
Cloud Provider [All/AWS/GCS/ABS/Swift/OSS]: All

Anything else we need to know?:

This error occurs due to BoltDB says it shouldn't get corrupted.
I have already fixed this kind of issue by this PR: https://github.com/gardener/etcd-backup-restore/pull/521 but now it seems like it occurrs again in druid e2e tests ... we need to take a relook or come-up with better approach to fix panic in boltDB.

gardener / etcd-druid

[Bug] One etcd-druid e2e tests fails due to panic in bolt db #483