search

prysmaticlabs / prysm

Go implementation of Ethereum proof of stake
https://www.offchainlabs.com
GNU General Public License v3.0
3.46k stars 985 forks source link

Beacon node panic crash with segmentation violation #5972

Closed Phistr90 closed 4 years ago

Phistr90 commented 4 years ago

🐞 Bug Report

Description

My beacon node crashed last night.

Has this worked before in a previous version?

Never had that crash before.

🔬 Minimal Reproduction

Dont know how to reproduce.

🔥 Error

Stack trace starts with output below, please find the complete log here: stacktrace.log


fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x2 addr=0xc09d286000 pc=0x48a913]

runtime stack:
runtime.throw(0x179b418, 0x2a)
        GOROOT/src/runtime/panic.go:1116 +0x72
runtime.sigpanic()
        GOROOT/src/runtime/signal_unix.go:679 +0x46a
runtime.memclrNoHeapPointers(0xc09d1ac000, 0x8b8000)
        GOROOT/src/runtime/memclr_amd64.s:76 +0x113
runtime.(*mheap).alloc(0x27f6d80, 0x45c, 0x7f0ce1790101, 0x4493d0)
        GOROOT/src/runtime/mheap.go:876 +0xd1
runtime.largeAlloc(0x8b640c, 0x480101, 0xc09c7e6000)
        GOROOT/src/runtime/malloc.go:1152 +0x92
runtime.mallocgc.func1()
        GOROOT/src/runtime/malloc.go:1047 +0x46
runtime.systemstack(0x0)
        GOROOT/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()

🌍 Your Environment

Operating System:

 
Linux Ubuntu 19 LTS

What version of Prysm are you running? (Which release)

  
Built from commit eb77f56d8a0bd50860b432ceec8cfa659997aee0
Built at: 2020-05-18T20:24:30+02:00
terencechain commented 4 years ago

Looking at the trace log, this looks libp2p swarm related. Tentatively assigning to @nisdas, feel free unassign if it's not appropriate

prestonvanloon commented 4 years ago

@terencechain where do you see this is swarm related?

From the log, I see this failing to allocate memory when saving a state to the database.

goroutine 78175624 [running]:
runtime.systemstack_switch()
    GOROOT/src/runtime/asm_amd64.s:330 fp=0xc03465d468 sp=0xc03465d460 pc=0x4877c0
runtime.mallocgc(0x8b640c, 0x14c4420, 0x1, 0xc03465d580)
    GOROOT/src/runtime/malloc.go:1046 +0x895 fp=0xc03465d508 sp=0xc03465d468 pc=0x42e5b5
runtime.makeslice(0x14c4420, 0x8b640c, 0x8b640c, 0xc09c7e6000)
    GOROOT/src/runtime/slice.go:49 +0x6c fp=0xc03465d538 sp=0xc03465d508 pc=0x46e75c
github.com/golang/snappy.Encode(0x0, 0x0, 0x0, 0xc09c7e6000, 0x777a38, 0x777a38, 0x0, 0x7d352d8c09729ea4, 0x31e02c52451b25ff)
    external/com_github_golang_snappy/encode.go:22 +0x22d fp=0xc03465d590 sp=0xc03465d538 pc=0xd61bdd
github.com/prysmaticlabs/prysm/beacon-chain/db/kv.encode(0x1b28940, 0xc02e462820, 0x176bdcd, 0x12, 0x0, 0x0, 0x0)
    beacon-chain/db/kv/encoding.go:42 +0x17e fp=0xc03465d600 sp=0xc03465d590 pc=0xd714fe
github.com/prysmaticlabs/prysm/beacon-chain/db/kv.(*Store).SaveState(0xc0006e4c40, 0x1b31200, 0xc0691f05a0, 0xc02bcd95e0, 0xc31e455837bee996, 0x19db50c773a6a92, 0x83396c4572934079, 0x5b0d4e9d93ef8e1a, 0x0, 0x0)
    beacon-chain/db/kv/state.go:115 +0x123 fp=0xc03465d6d0 sp=0xc03465d600 pc=0xd77593
github.com/prysmaticlabs/prysm/beacon-chain/blockchain.(*Service).onBlock(0xc00000c3c0, 0x1b31200, 0xc02414bc50, 0xc033ab5e80, 0xc31e455837bee996, 0x19db50c773a6a92, 0x83396c4572934079, 0x5b0d4e9d93ef8e1a, 0x0, 0x0, ...)
    beacon-chain/blockchain/process_block.go:97 +0x1406 fp=0xc03465db28 sp=0xc03465d6d0 pc=0x1194b66
github.com/prysmaticlabs/prysm/beacon-chain/blockchain.(*Service).ReceiveBlockNoPubsub(0xc00000c3c0, 0x1b31200, 0xc02414bc20, 0xc0472af6c0, 0xc31e455837bee996, 0x19db50c773a6a92, 0x83396c4572934079, 0x5b0d4e9d93ef8e1a, 0x0, 0x0)
    beacon-chain/blockchain/receive_block.go:71 +0x141 fp=0xc03465dc60 sp=0xc03465db28 pc=0x119dd91
github.com/prysmaticlabs/prysm/beacon-chain/sync.(*Service).beaconBlockSubscriber(0xc00000c5a0, 0x1b31200, 0xc01cd89320, 0x1b26380, 0xc0472af6c0, 0x0, 0x1b26380)
    beacon-chain/sync/subscriber_beacon_blocks.go:44 +0x23a fp=0xc03465dd20 sp=0xc03465dc60 pc=0x12510ea
github.com/prysmaticlabs/prysm/beacon-chain/sync.(*Service).beaconBlockSubscriber-fm(0x1b31200, 0xc01cd89320, 0x1b26380, 0xc0472af6c0, 0xc0472af6c0, 0x0)
    beacon-chain/sync/subscriber_beacon_blocks.go:16 +0x52 fp=0xc03465dd68 sp=0xc03465dd20 pc=0x1261db2
github.com/prysmaticlabs/prysm/beacon-chain/sync.(*Service).subscribeWithBase.func1(0xc025b672c0)
    beacon-chain/sync/subscriber.go:167 +0x258 fp=0xc03465dfd8 sp=0xc03465dd68 pc=0x125f218
runtime.goexit()
    GOROOT/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc03465dfe0 sp=0xc03465dfd8 pc=0x4898d1
created by github.com/prysmaticlabs/prysm/beacon-chain/sync.(*Service).subscribeWithBase.func2
    beacon-chain/sync/subscriber.go:189 +0xdb

@Phistr90 can you tell us more about your system? Were you running low on RAM, swap, or disk space at the time of the error?

Phistr90 commented 4 years ago

@prestonvanloon Shouldnt have been the case, I ran only the beacon node and validator client on my intel nuc8i7hvk with 16gb and plenty of ssd space.

prestonvanloon commented 4 years ago

@Phistr90 thanks for confirming. It seems unlikely, but wanted to rule that out.

nisdas commented 4 years ago

@Phistr90 Have you encountered this again ?

no-response[bot] commented 4 years ago

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

2naive commented 4 years ago

Had similar errors:

[2020-08-04 05:11:31]  INFO powchain: Processing deposits from Ethereum 1 chain deposits=1024 genesisValidators=1020
unexpected fault address 0x7f2378e5c65d
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x2 addr=0x7f2378e5c65d pc=0x42285f]

goroutine 246 [running]:
runtime.throw(0x1882c18, 0x5)
        GOROOT/src/runtime/panic.go:1116 +0x72 fp=0xc0021a1190 sp=0xc0021a1160 pc=0x4567c2
runtime.sigpanic()
        GOROOT/src/runtime/signal_unix.go:702 +0x3cc fp=0xc0021a11c0 sp=0xc0021a1190 pc=0x46d7ec
cmpbody()
        GOROOT/src/internal/bytealg/compare_amd64.s:84 +0x9f fp=0xc0021a11c8 sp=0xc0021a11c0 pc=0x42285f
bytes.Compare(...)
        GOROOT/src/bytes/bytes.go:27
go.etcd.io/bbolt.(*Cursor).nsearch.func2(0x5575, 0x3e006d)
        external/io_etcd_go_bbolt/cursor.go:328 +0x9d fp=0xc0021a1228 sp=0xc0021a11c8 pc=0x101323d
sort.Search(0xaaeb, 0xc0021a12c8, 0x0)
        GOROOT/src/sort/search.go:66 +0x58 fp=0xc0021a1260 sp=0xc0021a1228 pc=0x4cf9f8
go.etcd.io/bbolt.(*Cursor).nsearch(0xc0021a1490, 0x2897c38, 0xd, 0xd)
        external/io_etcd_go_bbolt/cursor.go:327 +0x104 fp=0xc0021a1310 sp=0xc0021a1260 pc=0x1003144
go.etcd.io/bbolt.(*Cursor).search(0xc0021a1490, 0x2897c38, 0xd, 0xd, 0x933)
        external/io_etcd_go_bbolt/cursor.go:257 +0x1ca fp=0xc0021a13b0 sp=0xc0021a1310 pc=0x1002b8a
go.etcd.io/bbolt.(*Cursor).seek(0xc0021a1490, 0x2897c38, 0xd, 0xd, 0x3e006d, 0xc0013e4000, 0xc003c26000, 0x3e006d, 0x3e006d, 0x0, ...)
        external/io_etcd_go_bbolt/cursor.go:159 +0x7d fp=0xc0021a1420 sp=0xc0021a13b0 pc=0x100232d
go.etcd.io/bbolt.(*Bucket).Put(0xc0020a5440, 0x2897c38, 0xd, 0xd, 0xc003c26000, 0x3e006d, 0x3e006d, 0xc0007c4400, 0xc0001dc000)
        external/io_etcd_go_bbolt/bucket.go:295 +0x103 fp=0xc0021a14c0 sp=0xc0021a1420 pc=0xfff4c3
github.com/prysmaticlabs/prysm/beacon-chain/db/kv.(*Store).SavePowchainData.func1(0xc0001dc000, 0x1, 0xc0001dc000)
        beacon-chain/db/kv/powchain.go:23 +0xf9 fp=0xc0021a1528 sp=0xc0021a14c0 pc=0x102d909
go.etcd.io/bbolt.(*DB).Update(0xc0007c4400, 0xc0021a15e8, 0x0, 0x0)
        external/io_etcd_go_bbolt/db.go:694 +0x8e fp=0xc0021a1590 sp=0xc0021a1528 pc=0x1005a7e
github.com/prysmaticlabs/prysm/beacon-chain/db/kv.(*Store).SavePowchainData(0xc000e34000, 0x1c87a80, 0xc0000c0030, 0xc0013e40c0, 0x0, 0x0)
        beacon-chain/db/kv/powchain.go:17 +0xd4 fp=0xc0021a1618 sp=0xc0021a1590 pc=0x1022d84
github.com/prysmaticlabs/prysm/beacon-chain/powchain.(*Service).ProcessLog(0xc000176360, 0x1c87a80, 0xc0000c0030, 0xba384ade4f9fb307, 0xc587ac6d542b21ce, 0xdc3ffe8d, 0xc003000880, 0x1, 0x4, 0xc003004fc0, ...)
        beacon-chain/powchain/log_processing.go:92 +0x53c fp=0xc0021a18c8 sp=0xc0021a1618 pc=0x125994c
github.com/prysmaticlabs/prysm/beacon-chain/powchain.(*Service).processPastLogs(0xc000176360, 0x1c87a80, 0xc0000c0030, 0x96004a728ebe464a, 0x1b016b1142dcbf23)
        beacon-chain/powchain/log_processing.go:329 +0x307 fp=0xc0021a1cb0 sp=0xc0021a18c8 pc=0x125bb67
github.com/prysmaticlabs/prysm/beacon-chain/powchain.(*Service).initPOWService(0xc000176360)
        beacon-chain/powchain/service.go:638 +0x349 fp=0xc0021a1d80 sp=0xc0021a1cb0 pc=0x12613b9
github.com/prysmaticlabs/prysm/beacon-chain/powchain.(*Service).run(0xc000176360, 0xc0000497a0)
        beacon-chain/powchain/service.go:653 +0x76 fp=0xc0021a1fb8 sp=0xc0021a1d80 pc=0x1261536
github.com/prysmaticlabs/prysm/beacon-chain/powchain.(*Service).Start.func1(0xc000176360)
        beacon-chain/powchain/service.go:222 +0x50 fp=0xc0021a1fd8 sp=0xc0021a1fb8 pc=0x1261a70
runtime.goexit()
        GOROOT/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0021a1fe0 sp=0xc0021a1fd8 pc=0x489ab1
created by github.com/prysmaticlabs/prysm/beacon-chain/powchain.(*Service).Start
        beacon-chain/powchain/service.go:220 +0x3f

goroutine 1 [chan receive, 2 minutes]:
github.com/prysmaticlabs/prysm/beacon-chain/node.(*BeaconNode).Start(0xc0001f0840)
        beacon-chain/node/node.go:259 +0x1dc
main.startNode(0xc0005f4a80, 0xc000615f00, 0x0)
        beacon-chain/main.go:187 +0xc7
github.com/urfave/cli/v2.(*App).RunContext(0xc0004cd080, 0x1c87a80, 0xc0000c0030, 0xc0000a81b0, 0x1, 0x1, 0x0, 0x0)
        external/com_github_urfave_cli_v2/app.go:315 +0x70b
github.com/urfave/cli/v2.(*App).Run(...)
        external/com_github_urfave_cli_v2/app.go:215
main.main()
        beacon-chain/main.go:161 +0x1be

goroutine 19 [select]:
github.com/dgraph-io/ristretto.(*defaultPolicy).processItems(0xc0001d9c50)
        external/com_github_dgraph_io_ristretto/policy.go:96 +0xbe
created by github.com/dgraph-io/ristretto.newDefaultPolicy
        external/com_github_dgraph_io_ristretto/policy.go:80 +0x129

goroutine 20 [select]:
github.com/dgraph-io/ristretto.(*Cache).processItems(0xc0000c6ba0)
        external/com_github_dgraph_io_ristretto/cache.go:314 +0xed
created by github.com/dgraph-io/ristretto.NewCache
        external/com_github_dgraph_io_ristretto/cache.go:162 +0x22d

goroutine 21 [select, 2 minutes]:
github.com/dgraph-io/ristretto.(*defaultPolicy).processItems(0xc000255d70)
        external/com_github_dgraph_io_ristretto/policy.go:96 +0xbe
created by github.com/dgraph-io/ristretto.newDefaultPolicy
        external/com_github_dgraph_io_ristretto/policy.go:80 +0x129

goroutine 22 [select]:
github.com/dgraph-io/ristretto.(*Cache).processItems(0xc0000c6d20)
        external/com_github_dgraph_io_ristretto/cache.go:314 +0xed
created by github.com/dgraph-io/ristretto.NewCache
        external/com_github_dgraph_io_ristretto/cache.go:162 +0x22d

goroutine 23 [select, 2 minutes]:
github.com/dgraph-io/ristretto.(*defaultPolicy).processItems(0xc00027fe90)
        external/com_github_dgraph_io_ristretto/policy.go:96 +0xbe
created by github.com/dgraph-io/ristretto.newDefaultPolicy
        external/com_github_dgraph_io_ristretto/policy.go:80 +0x129

goroutine 24 [select]:
github.com/dgraph-io/ristretto.(*Cache).processItems(0xc0000c6ea0)
        external/com_github_dgraph_io_ristretto/cache.go:314 +0xed
created by github.com/dgraph-io/ristretto.NewCache
        external/com_github_dgraph_io_ristretto/cache.go:162 +0x22d

goroutine 25 [select, 2 minutes]:
github.com/dgraph-io/ristretto.(*defaultPolicy).processItems(0xc0003ac090)
        external/com_github_dgraph_io_ristretto/policy.go:96 +0xbe
created by github.com/dgraph-io/ristretto.newDefaultPolicy
        external/com_github_dgraph_io_ristretto/policy.go:80 +0x129

goroutine 26 [select]:
github.com/dgraph-io/ristretto.(*Cache).processItems(0xc0000c7020)
        external/com_github_dgraph_io_ristretto/cache.go:314 +0xed
created by github.com/dgraph-io/ristretto.NewCache
        external/com_github_dgraph_io_ristretto/cache.go:162 +0x22d

goroutine 5 [select]:
github.com/dgraph-io/ristretto.(*defaultPolicy).processItems(0xc0000ab770)
        external/com_github_dgraph_io_ristretto/policy.go:96 +0xbe
created by github.com/dgraph-io/ristretto.newDefaultPolicy
        external/com_github_dgraph_io_ristretto/policy.go:80 +0x129

goroutine 6 [select]:
github.com/dgraph-io/ristretto.(*Cache).processItems(0xc00007e660)
        external/com_github_dgraph_io_ristretto/cache.go:314 +0xed
created by github.com/dgraph-io/ristretto.NewCache
        external/com_github_dgraph_io_ristretto/cache.go:162 +0x22d

goroutine 60 [select, 2 minutes]:
github.com/prysmaticlabs/prysm/shared/runutil.RunEvery.func1(0xc0000d6910, 0x2287890, 0x44, 0x1a548c8, 0x1c87a80, 0xc0000c0030)
        shared/runutil/every.go:20 +0x187
created by github.com/prysmaticlabs/prysm/shared/runutil.RunEvery
        shared/runutil/every.go:18 +0x136

goroutine 128 [select]:
github.com/prysmaticlabs/prysm/shared/runutil.RunEvery.func1(0xc0000d6370, 0x2794ed0, 0x5b, 0xc000610420, 0x1c87a40, 0xc00007df00)
        shared/runutil/every.go:20 +0x187
created by github.com/prysmaticlabs/prysm/shared/runutil.RunEvery
        shared/runutil/every.go:18 +0x136

goroutine 61 [chan receive]:
k8s.io/klog.(*loggingT).flushDaemon(0x29a2260)
        external/io_k8s_klog/klog.go:1010 +0x8b
created by k8s.io/klog.init.0
        external/io_k8s_klog/klog.go:411 +0xd6

goroutine 40 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x7f22d57e12d0, 0x72, 0x0)
        GOROOT/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000753118, 0x72, 0x0, 0x0, 0x188540d)
        GOROOT/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        GOROOT/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000753100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        GOROOT/src/internal/poll/fd_unix.go:384 +0x1d4
net.(*netFD).accept(0xc000753100, 0xc000955938, 0xc000090700, 0x7f22fd491560)
        GOROOT/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc0010186a0, 0xc000955978, 0x42ea18, 0x30)
        GOROOT/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc0010186a0, 0x17706a0, 0xc00101a7e0, 0x1630de0, 0x285d4a0)
        GOROOT/src/net/tcpsock.go:261 +0x64
net/http.(*Server).Serve(0xc0001dc9a0, 0x1c80780, 0xc0010186a0, 0x0, 0x0)
        GOROOT/src/net/http/server.go:2901 +0x25d
net/http.(*Server).ListenAndServe(0xc0001dc9a0, 0xc000000005, 0xc000955bb8)
        GOROOT/src/net/http/server.go:2830 +0xb7
github.com/prysmaticlabs/prysm/shared/prometheus.(*Service).Start.func1(0xc0005fed00)
        shared/prometheus/service.go:134 +0x205
created by github.com/prysmaticlabs/prysm/shared/prometheus.(*Service).Start
        shared/prometheus/service.go:122 +0x3f

goroutine 62 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x29a2340)
        external/io_k8s_klog_v2/klog.go:1107 +0x8b
created by k8s.io/klog/v2.init.0
        external/io_k8s_klog_v2/klog.go:416 +0xd6

goroutine 74 [select, 2 minutes]:
github.com/patrickmn/go-cache.(*janitor).Run(0xc0004fd530, 0xc0008080c0)
        external/com_github_patrickmn_go_cache/cache.go:1079 +0xd5
created by github.com/patrickmn/go-cache.runJanitor
        external/com_github_patrickmn_go_cache/cache.go:1099 +0xa9

goroutine 98 [syscall, 2 minutes]:
os/signal.signal_recv(0x0)
        GOROOT/src/runtime/sigqueue.go:147 +0x9c
os/signal.loop()
        GOROOT/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.Notify.func1
        GOROOT/src/os/signal/signal.go:127 +0x44

goroutine 84 [chan receive]:
github.com/ethereum/go-ethereum/metrics.(*meterArbiter).tick(0x299ec80)
        external/com_github_ethereum_go_ethereum/metrics/meter.go:289 +0x78
created by github.com/ethereum/go-ethereum/metrics.NewMeterForced
        external/com_github_ethereum_go_ethereum/metrics/meter.go:70 +0x105

goroutine 85 [select]:
github.com/ethereum/go-ethereum/consensus/ethash.(*remoteSealer).loop(0xc000258690)
        external/com_github_ethereum_go_ethereum/consensus/ethash/sealer.go:278 +0x258
created by github.com/ethereum/go-ethereum/consensus/ethash.startRemoteSealer
        external/com_github_ethereum_go_ethereum/consensus/ethash/sealer.go:262 +0x2ae

goroutine 86 [chan receive, 2 minutes]:
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0xc0005873f0)
        external/com_github_ethereum_go_ethereum/core/tx_cacher.go:63 +0x97
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher
        external/com_github_ethereum_go_ethereum/core/tx_cacher.go:55 +0x9f

goroutine 87 [chan receive, 2 minutes]:
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0xc0005873f0)
        external/com_github_ethereum_go_ethereum/core/tx_cacher.go:63 +0x97
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher
        external/com_github_ethereum_go_ethereum/core/tx_cacher.go:55 +0x9f

goroutine 88 [chan receive, 2 minutes]:
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0xc0005873f0)
        external/com_github_ethereum_go_ethereum/core/tx_cacher.go:63 +0x97
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher
        external/com_github_ethereum_go_ethereum/core/tx_cacher.go:55 +0x9f

goroutine 89 [chan receive, 2 minutes]:
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0xc0005873f0)
        external/com_github_ethereum_go_ethereum/core/tx_cacher.go:63 +0x97
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher
        external/com_github_ethereum_go_ethereum/core/tx_cacher.go:55 +0x9f

goroutine 90 [chan receive, 2 minutes]:
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0xc0005873f0)
        external/com_github_ethereum_go_ethereum/core/tx_cacher.go:63 +0x97
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher
        external/com_github_ethereum_go_ethereum/core/tx_cacher.go:55 +0x9f

goroutine 91 [chan receive, 2 minutes]:
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0xc0005873f0)
        external/com_github_ethereum_go_ethereum/core/tx_cacher.go:63 +0x97
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher
        external/com_github_ethereum_go_ethereum/core/tx_cacher.go:55 +0x9f

goroutine 115 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000182000)
        external/io_opencensus_go/stats/view/worker.go:276 +0x100
created by go.opencensus.io/stats/view.init.0
        external/io_opencensus_go/stats/view/worker.go:34 +0x68

goroutine 116 [select, 2 minutes]:
github.com/ipfs/go-log/writer.(*MirrorWriter).logRoutine(0xc0007a0600)
        external/com_github_ipfs_go_log/writer/writer.go:71 +0x128
created by github.com/ipfs/go-log/writer.NewMirrorWriter
        external/com_github_ipfs_go_log/writer/writer.go:36 +0xb9

goroutine 158 [chan receive, 2 minutes]:
google.golang.org/grpc.(*addrConn).resetTransport(0xc0001e2000)
        external/org_golang_google_grpc/clientconn.go:1156 +0x6ea
created by google.golang.org/grpc.(*addrConn).connect
        external/org_golang_google_grpc/clientconn.go:799 +0x128

goroutine 157 [select, 2 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0007643c0)
        external/org_golang_google_grpc/balancer_conn_wrappers.go:69 +0xc2
created by google.golang.org/grpc.newCCBalancerWrapper
        external/org_golang_google_grpc/balancer_conn_wrappers.go:60 +0x16d

goroutine 129 [select]:
github.com/prysmaticlabs/prysm/shared/runutil.RunEvery.func1(0xc0000d63c0, 0x2794cd8, 0x59, 0xc000610450, 0x1c87a40, 0xc00007df00)
        shared/runutil/every.go:20 +0x187
created by github.com/prysmaticlabs/prysm/shared/runutil.RunEvery
        shared/runutil/every.go:18 +0x136

goroutine 185 [select, 2 minutes]:
github.com/dgraph-io/ristretto.(*defaultPolicy).processItems(0xc0005f10b0)
        external/com_github_dgraph_io_ristretto/policy.go:96 +0xbe
created by github.com/dgraph-io/ristretto.newDefaultPolicy
        external/com_github_dgraph_io_ristretto/policy.go:80 +0x129

goroutine 186 [select]:
github.com/dgraph-io/ristretto.(*Cache).processItems(0xc000175bc0)
        external/com_github_dgraph_io_ristretto/cache.go:314 +0xed
created by github.com/dgraph-io/ristretto.NewCache
        external/com_github_dgraph_io_ristretto/cache.go:162 +0x22d

goroutine 92 [select, 2 minutes]:
github.com/dgraph-io/ristretto.(*defaultPolicy).processItems(0xc00097a000)
        external/com_github_dgraph_io_ristretto/policy.go:96 +0xbe
created by github.com/dgraph-io/ristretto.newDefaultPolicy
        external/com_github_dgraph_io_ristretto/policy.go:80 +0x129

goroutine 93 [select]:
github.com/dgraph-io/ristretto.(*Cache).processItems(0xc000e300c0)
        external/com_github_dgraph_io_ristretto/cache.go:314 +0xed
created by github.com/dgraph-io/ristretto.NewCache
        external/com_github_dgraph_io_ristretto/cache.go:162 +0x22d

goroutine 95 [select, 2 minutes]:
github.com/dgraph-io/ristretto.(*defaultPolicy).processItems(0xc00059a2d0)
        external/com_github_dgraph_io_ristretto/policy.go:96 +0xbe
created by github.com/dgraph-io/ristretto.newDefaultPolicy
        external/com_github_dgraph_io_ristretto/policy.go:80 +0x129

goroutine 96 [select]:
github.com/dgraph-io/ristretto.(*Cache).processItems(0xc0004c0300)
        external/com_github_dgraph_io_ristretto/cache.go:314 +0xed
created by github.com/dgraph-io/ristretto.NewCache
        external/com_github_dgraph_io_ristretto/cache.go:162 +0x22d

goroutine 97 [select]:
github.com/kevinms/leakybucket-go.(*Collector).PeriodicPrune.func1(0x3b9aca00, 0xc0000e15c0)
        external/com_github_kevinms_leakybucket_go/collector.go:190 +0xd2
created by github.com/kevinms/leakybucket-go.(*Collector).PeriodicPrune
        external/com_github_kevinms_leakybucket_go/collector.go:187 +0x49

goroutine 194 [select, 2 minutes]:
github.com/libp2p/go-libp2p-peerstore/pstoremem.(*memoryAddrBook).background(0xc000186900)
        external/com_github_libp2p_go_libp2p_peerstore/pstoremem/addr_book.go:93 +0x126
created by github.com/libp2p/go-libp2p-peerstore/pstoremem.NewAddrBook
        external/com_github_libp2p_go_libp2p_peerstore/pstoremem/addr_book.go:83 +0x18e

goroutine 195 [select, 2 minutes]:
github.com/jbenet/goprocess/context.CloseAfterContext.func1(0x1c87a40, 0xc000809e40, 0x1c9caa0, 0xc0004c08a0)
        external/com_github_jbenet_goprocess/context/context.go:65 +0xd8
created by github.com/jbenet/goprocess/context.CloseAfterContext
        external/com_github_jbenet_goprocess/context/context.go:64 +0xa7

goroutine 197 [select, 2 minutes]:
github.com/libp2p/go-libp2p-swarm.(*DialBackoff).background(0xc0001a7610, 0x1c87a40, 0xc0000e16c0)
        external/com_github_libp2p_go_libp2p_swarm/swarm_dial.go:126 +0x110
created by github.com/libp2p/go-libp2p-swarm.(*DialBackoff).init
        external/com_github_libp2p_go_libp2p_swarm/swarm_dial.go:119 +0x5d

goroutine 198 [select, 2 minutes]:
github.com/libp2p/go-libp2p/p2p/protocol/identify.(*ObservedAddrManager).worker(0xc0004c0cc0, 0x1c87a40, 0xc00007d400)
        external/com_github_libp2p_go_libp2p/p2p/protocol/identify/obsaddr.go:236 +0x230
created by github.com/libp2p/go-libp2p/p2p/protocol/identify.NewObservedAddrManager
        external/com_github_libp2p_go_libp2p/p2p/protocol/identify/obsaddr.go:127 +0x175

goroutine 199 [select, 2 minutes]:
github.com/libp2p/go-libp2p/p2p/protocol/identify.(*IDService).loop(0xc0008060d0)
        external/com_github_libp2p_go_libp2p/p2p/protocol/identify/id.go:204 +0x4f7
created by github.com/libp2p/go-libp2p/p2p/protocol/identify.NewIDService
        external/com_github_libp2p_go_libp2p/p2p/protocol/identify/id.go:151 +0x2c3

goroutine 200 [select, 2 minutes]:
github.com/libp2p/go-libp2p-circuit.(*RelayListener).Accept(0xc0004c0d20, 0x0, 0x0, 0x0, 0x0)
        external/com_github_libp2p_go_libp2p_circuit/listen.go:27 +0x1ac
github.com/libp2p/go-libp2p-transport-upgrader.(*listener).handleIncoming(0xc0004c0de0)
        external/com_github_libp2p_go_libp2p_transport_upgrader/listener.go:77 +0xf7
created by github.com/libp2p/go-libp2p-transport-upgrader.(*Upgrader).UpgradeListener
        external/com_github_libp2p_go_libp2p_transport_upgrader/upgrader.go:49 +0x185

goroutine 207 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x7f22d57e1490, 0x72, 0x0)
        GOROOT/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc0005ef398, 0x72, 0x0, 0x0, 0x188540d)
        GOROOT/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        GOROOT/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc0005ef380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        GOROOT/src/internal/poll/fd_unix.go:384 +0x1d4
net.(*netFD).accept(0xc0005ef380, 0x0, 0x0, 0x0)
        GOROOT/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc0005fe160, 0x0, 0x0, 0x0)
        GOROOT/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc0005fe160, 0x0, 0x0, 0x42e1a6, 0x0)
        GOROOT/src/net/tcpsock.go:261 +0x64
github.com/multiformats/go-multiaddr-net.(*maListener).Accept(0xc0005fe1e0, 0xc000502000, 0x7f22fd4907d0, 0x0, 0x0)
        external/com_github_multiformats_go_multiaddr_net/net.go:243 +0x37
github.com/libp2p/go-tcp-transport.(*lingerListener).Accept(0xc0005fe220, 0x0, 0x0, 0x0, 0x0)
        external/com_github_libp2p_go_tcp_transport/tcp.go:42 +0x37
github.com/libp2p/go-libp2p-transport-upgrader.(*listener).handleIncoming(0xc0004c11a0)
        external/com_github_libp2p_go_libp2p_transport_upgrader/listener.go:77 +0xf7
created by github.com/libp2p/go-libp2p-transport-upgrader.(*Upgrader).UpgradeListener
        external/com_github_libp2p_go_libp2p_transport_upgrader/upgrader.go:49 +0x185

goroutine 206 [chan receive, 2 minutes]:
github.com/libp2p/go-libp2p-transport-upgrader.(*listener).Accept(0xc0004c0de0, 0x29c6660, 0x29c6670, 0xc0005c2f20, 0x46a07e)
        external/com_github_libp2p_go_libp2p_transport_upgrader/listener.go:155 +0x51
github.com/libp2p/go-libp2p-swarm.(*Swarm).AddListenAddr.func2(0x1c882c0, 0xc0004c0de0, 0xc0001a7500, 0x1ca0f20, 0xc00079ba00)
        external/com_github_libp2p_go_libp2p_swarm/swarm_listen.go:84 +0x18e
created by github.com/libp2p/go-libp2p-swarm.(*Swarm).AddListenAddr
        external/com_github_libp2p_go_libp2p_swarm/swarm_listen.go:69 +0x257

goroutine 188 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x7f22d57e1570, 0x72, 0x0)
        GOROOT/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000928098, 0x72, 0x0, 0x0, 0x188540d)
        GOROOT/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        GOROOT/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000928080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        GOROOT/src/internal/poll/fd_unix.go:384 +0x1d4
net.(*netFD).accept(0xc000928080, 0x37e11d600, 0x0, 0x0)
        GOROOT/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc0000b6360, 0xc0004c9e60, 0xc0004c9e68, 0x18)
        GOROOT/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc0000b6360, 0x1a54e88, 0xc0007881a0, 0x1c9a4e0, 0xc000828000)
        GOROOT/src/net/tcpsock.go:261 +0x64
google.golang.org/grpc.(*Server).Serve(0xc0007881a0, 0x1c80780, 0xc0000b6360, 0x0, 0x0)
        external/org_golang_google_grpc/server.go:621 +0x210
github.com/prysmaticlabs/prysm/beacon-chain/rpc.(*Service).Start.func1(0xc0001f5b80)
        beacon-chain/rpc/service.go:303 +0x5d
created by github.com/prysmaticlabs/prysm/beacon-chain/rpc.(*Service).Start
        beacon-chain/rpc/service.go:301 +0x1122

goroutine 190 [chan receive, 2 minutes]:
github.com/prysmaticlabs/prysm/beacon-chain/blockchain.(*Service).processAttestation(0xc0007c5000, 0xc00009e180)
        beacon-chain/blockchain/receive_attestation.go:83 +0xeb
created by github.com/prysmaticlabs/prysm/beacon-chain/blockchain.(*Service).Start
        beacon-chain/blockchain/service.go:230 +0xd26

goroutine 211 [chan receive, 2 minutes]:
github.com/libp2p/go-libp2p-transport-upgrader.(*listener).Accept(0xc0004c11a0, 0x29c3320, 0x29c3330, 0xc0005c1f20, 0x46a07e)
        external/com_github_libp2p_go_libp2p_transport_upgrader/listener.go:155 +0x51
github.com/libp2p/go-libp2p-swarm.(*Swarm).AddListenAddr.func2(0x1c882c0, 0xc0004c11a0, 0xc0001a7500, 0x1ca0f20, 0xc0005fe1c0)
        external/com_github_libp2p_go_libp2p_swarm/swarm_listen.go:84 +0x18e
created by github.com/libp2p/go-libp2p-swarm.(*Swarm).AddListenAddr
        external/com_github_libp2p_go_libp2p_swarm/swarm_listen.go:69 +0x257

goroutine 214 [select]:
github.com/libp2p/go-libp2p-autonat.(*AmbientAutoNAT).background(0xc0008065b0)
        external/com_github_libp2p_go_libp2p_autonat/autonat.go:175 +0x2f1
created by github.com/libp2p/go-libp2p-autonat.New
        external/com_github_libp2p_go_libp2p_autonat/autonat.go:125 +0x6ac

goroutine 215 [select]:
github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).background(0xc0000ee9a0)
        external/com_github_libp2p_go_libp2p/p2p/host/basic/basic_host.go:520 +0x257
created by github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).Start
        external/com_github_libp2p_go_libp2p/p2p/host/basic/basic_host.go:368 +0x5c

goroutine 216 [select]:
github.com/libp2p/go-libp2p-pubsub.(*GossipSubRouter).heartbeatTimer(0xc000176000)
        external/com_github_libp2p_go_libp2p_pubsub/gossipsub.go:1207 +0x22f
created by github.com/libp2p/go-libp2p-pubsub.(*GossipSubRouter).Attach
        external/com_github_libp2p_go_libp2p_pubsub/gossipsub.go:421 +0x10c

goroutine 217 [select, 2 minutes]:
github.com/libp2p/go-libp2p-pubsub.(*GossipSubRouter).connector(0xc000176000)
        external/com_github_libp2p_go_libp2p_pubsub/gossipsub.go:833 +0x108
created by github.com/libp2p/go-libp2p-pubsub.(*GossipSubRouter).Attach
        external/com_github_libp2p_go_libp2p_pubsub/gossipsub.go:425 +0x137

goroutine 218 [select, 2 minutes]:
github.com/libp2p/go-libp2p-pubsub.(*GossipSubRouter).connector(0xc000176000)
        external/com_github_libp2p_go_libp2p_pubsub/gossipsub.go:833 +0x108
created by github.com/libp2p/go-libp2p-pubsub.(*GossipSubRouter).Attach
        external/com_github_libp2p_go_libp2p_pubsub/gossipsub.go:425 +0x137

goroutine 219 [select, 2 minutes]:
github.com/libp2p/go-libp2p-pubsub.(*GossipSubRouter).connector(0xc000176000)
        external/com_github_libp2p_go_libp2p_pubsub/gossipsub.go:833 +0x108
created by github.com/libp2p/go-libp2p-pubsub.(*GossipSubRouter).Attach
        external/com_github_libp2p_go_libp2p_pubsub/gossipsub.go:425 +0x137

goroutine 220 [select, 2 minutes]:
github.com/libp2p/go-libp2p-pubsub.(*GossipSubRouter).connector(0xc000176000)
        external/com_github_libp2p_go_libp2p_pubsub/gossipsub.go:833 +0x108
created by github.com/libp2p/go-libp2p-pubsub.(*GossipSubRouter).Attach
        external/com_github_libp2p_go_libp2p_pubsub/gossipsub.go:425 +0x137

goroutine 221 [select, 2 minutes]:
github.com/libp2p/go-libp2p-pubsub.(*GossipSubRouter).connector(0xc000176000)
        external/com_github_libp2p_go_libp2p_pubsub/gossipsub.go:833 +0x108
created by github.com/libp2p/go-libp2p-pubsub.(*GossipSubRouter).Attach
        external/com_github_libp2p_go_libp2p_pubsub/gossipsub.go:425 +0x137

goroutine 222 [select, 2 minutes]:
github.com/libp2p/go-libp2p-pubsub.(*GossipSubRouter).connector(0xc000176000)
        external/com_github_libp2p_go_libp2p_pubsub/gossipsub.go:833 +0x108
created by github.com/libp2p/go-libp2p-pubsub.(*GossipSubRouter).Attach
        external/com_github_libp2p_go_libp2p_pubsub/gossipsub.go:425 +0x137

goroutine 223 [select, 2 minutes]:
github.com/libp2p/go-libp2p-pubsub.(*GossipSubRouter).connector(0xc000176000)
        external/com_github_libp2p_go_libp2p_pubsub/gossipsub.go:833 +0x108
created by github.com/libp2p/go-libp2p-pubsub.(*GossipSubRouter).Attach
        external/com_github_libp2p_go_libp2p_pubsub/gossipsub.go:425 +0x137

goroutine 224 [select, 2 minutes]:
github.com/libp2p/go-libp2p-pubsub.(*GossipSubRouter).connector(0xc000176000)
        external/com_github_libp2p_go_libp2p_pubsub/gossipsub.go:833 +0x108
created by github.com/libp2p/go-libp2p-pubsub.(*GossipSubRouter).Attach
        external/com_github_libp2p_go_libp2p_pubsub/gossipsub.go:425 +0x137

goroutine 225 [select, 2 minutes]:
github.com/libp2p/go-libp2p-pubsub.(*validation).validateWorker(0xc0006d01e0)
        external/com_github_libp2p_go_libp2p_pubsub/validation.go:231 +0x129
created by github.com/libp2p/go-libp2p-pubsub.(*validation).Start
        external/com_github_libp2p_go_libp2p_pubsub/validation.go:120 +0x70

goroutine 226 [select, 2 minutes]:
github.com/libp2p/go-libp2p-pubsub.(*validation).validateWorker(0xc0006d01e0)
        external/com_github_libp2p_go_libp2p_pubsub/validation.go:231 +0x129
created by github.com/libp2p/go-libp2p-pubsub.(*validation).Start
        external/com_github_libp2p_go_libp2p_pubsub/validation.go:120 +0x70

goroutine 227 [select, 2 minutes]:
github.com/libp2p/go-libp2p-pubsub.(*validation).validateWorker(0xc0006d01e0)
        external/com_github_libp2p_go_libp2p_pubsub/validation.go:231 +0x129
created by github.com/libp2p/go-libp2p-pubsub.(*validation).Start
        external/com_github_libp2p_go_libp2p_pubsub/validation.go:120 +0x70

goroutine 228 [select, 2 minutes]:
github.com/libp2p/go-libp2p-pubsub.(*validation).validateWorker(0xc0006d01e0)
        external/com_github_libp2p_go_libp2p_pubsub/validation.go:231 +0x129
created by github.com/libp2p/go-libp2p-pubsub.(*validation).Start
        external/com_github_libp2p_go_libp2p_pubsub/validation.go:120 +0x70

goroutine 229 [select, 2 minutes]:
github.com/libp2p/go-libp2p-pubsub.(*validation).validateWorker(0xc0006d01e0)
        external/com_github_libp2p_go_libp2p_pubsub/validation.go:231 +0x129
created by github.com/libp2p/go-libp2p-pubsub.(*validation).Start
        external/com_github_libp2p_go_libp2p_pubsub/validation.go:120 +0x70

goroutine 230 [select, 2 minutes]:
github.com/libp2p/go-libp2p-pubsub.(*validation).validateWorker(0xc0006d01e0)
        external/com_github_libp2p_go_libp2p_pubsub/validation.go:231 +0x129
created by github.com/libp2p/go-libp2p-pubsub.(*validation).Start
        external/com_github_libp2p_go_libp2p_pubsub/validation.go:120 +0x70

goroutine 231 [select]:
github.com/libp2p/go-libp2p-pubsub.(*PubSub).processLoop(0xc0004cd680, 0x1c87a40, 0xc000809e40)
        external/com_github_libp2p_go_libp2p_pubsub/pubsub.go:464 +0x601
created by github.com/libp2p/go-libp2p-pubsub.NewPubSub
        external/com_github_libp2p_go_libp2p_pubsub/pubsub.go:278 +0xb4b

goroutine 232 [select]:
github.com/prysmaticlabs/prysm/beacon-chain/p2p/peers.(*PeerScorerManager).loop(0xc0006d0480, 0x1c87a40, 0xc000809e40)
        beacon-chain/p2p/peers/scorer_manager.go:76 +0x19f
created by github.com/prysmaticlabs/prysm/beacon-chain/p2p/peers.newPeerScorerManager
        beacon-chain/p2p/peers/scorer_manager.go:39 +0x21c

goroutine 233 [select, 2 minutes]:
github.com/prysmaticlabs/prysm/beacon-chain/sync.(*Service).registerHandlers(0xc0007c5200)
        beacon-chain/sync/service.go:235 +0x214
created by github.com/prysmaticlabs/prysm/beacon-chain/sync.NewRegularSync
        beacon-chain/sync/service.go:138 +0x294

goroutine 236 [chan receive, 2 minutes]:
github.com/prysmaticlabs/prysm/beacon-chain/p2p.(*Service).awaitStateInitialized(0xc000258000)
        beacon-chain/p2p/service.go:370 +0xde
github.com/prysmaticlabs/prysm/beacon-chain/p2p.(*Service).Start(0xc000258000)
        beacon-chain/p2p/service.go:183 +0x4c
created by github.com/prysmaticlabs/prysm/shared.(*ServiceRegistry).StartAll
        shared/service_registry.go:46 +0x23e

goroutine 240 [select, 2 minutes]:
github.com/prysmaticlabs/prysm/beacon-chain/sync/initial-sync.(*Service).Start(0xc0001c2540)
        beacon-chain/sync/initial-sync/service.go:94 +0x2a7
created by github.com/prysmaticlabs/prysm/shared.(*ServiceRegistry).StartAll
        shared/service_registry.go:46 +0x23e

goroutine 261 [select]:
github.com/prysmaticlabs/prysm/beacon-chain/operations/attestations.(*Service).prepareForkChoiceAtts(0xc00082a820)
        beacon-chain/operations/attestations/prepare_forkchoice.go:28 +0xfe
created by github.com/prysmaticlabs/prysm/beacon-chain/operations/attestations.(*Service).Start
        beacon-chain/operations/attestations/service.go:59 +0x3f

goroutine 189 [select, 2 minutes]:
github.com/prysmaticlabs/prysm/beacon-chain/blockchain.(*Service).Start.func1(0xc0007c5000, 0xc00009e180)
        beacon-chain/blockchain/service.go:207 +0x20c
created by github.com/prysmaticlabs/prysm/beacon-chain/blockchain.(*Service).Start
        beacon-chain/blockchain/service.go:201 +0x1286

goroutine 161 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x7f22d57e1110, 0x72, 0xffffffffffffffff)
        GOROOT/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000770018, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
        GOROOT/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        GOROOT/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000770000, 0xc000772000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        GOROOT/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000770000, 0xc000772000, 0x8000, 0x8000, 0x10401, 0x0, 0x1040100203000)
        GOROOT/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000e2e060, 0xc000772000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        GOROOT/src/net/net.go:184 +0x8e
bufio.(*Reader).Read(0xc00007ff80, 0xc00076c118, 0x9, 0x9, 0xc0001df500, 0x7f22fd4907d0, 0x0)
        GOROOT/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x1c519c0, 0xc00007ff80, 0xc00076c118, 0x9, 0x9, 0x9, 0x9c62a5, 0xc0007da2d0, 0xc0007d0004)
        GOROOT/src/io/io.go:310 +0x87
io.ReadFull(...)
        GOROOT/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc00076c118, 0x9, 0x9, 0x1c519c0, 0xc00007ff80, 0x0, 0x0, 0xc0007da2d0, 0x0)
        external/org_golang_x_net/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00076c0e0, 0xc0007da2d0, 0xc0007da200, 0x0, 0x0)
        external/org_golang_x_net/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00077a000)
        external/org_golang_google_grpc/internal/transport/http2_client.go:1273 +0x16f
created by google.golang.org/grpc/internal/transport.newHTTP2Client
        external/org_golang_google_grpc/internal/transport/http2_client.go:300 +0xd23

goroutine 245 [chan receive, 2 minutes]:
github.com/prysmaticlabs/prysm/beacon-chain/node.(*BeaconNode).Start.func1(0xc0001f0840)
        beacon-chain/node/node.go:245 +0xfc
created by github.com/prysmaticlabs/prysm/beacon-chain/node.(*BeaconNode).Start
        beacon-chain/node/node.go:241 +0x1c5

goroutine 258 [select, 2 minutes]:
github.com/prysmaticlabs/prysm/shared/runutil.RunEvery.func1(0xc0000d6410, 0x2796fe0, 0x56, 0xc00060e1c0, 0x1c87a40, 0xc00007df00)
        shared/runutil/every.go:20 +0x187
created by github.com/prysmaticlabs/prysm/shared/runutil.RunEvery
        shared/runutil/every.go:18 +0x136

goroutine 259 [select]:
github.com/prysmaticlabs/prysm/shared/runutil.RunEvery.func1(0xc0000d6460, 0x27970f0, 0x50, 0xc000170060, 0x1c87a40, 0xc00007df00)
        shared/runutil/every.go:20 +0x187
created by github.com/prysmaticlabs/prysm/shared/runutil.RunEvery
        shared/runutil/every.go:18 +0x136

goroutine 260 [select]:
github.com/prysmaticlabs/prysm/shared/runutil.RunEvery.func1(0xc0000d6500, 0x27992a8, 0x4c, 0xc000170070, 0x1c87a40, 0xc00007df00)
        shared/runutil/every.go:20 +0x187
created by github.com/prysmaticlabs/prysm/shared/runutil.RunEvery
        shared/runutil/every.go:18 +0x136

goroutine 262 [select]:
github.com/prysmaticlabs/prysm/beacon-chain/operations/attestations.(*Service).pruneAttsPool(0xc00082a820)
        beacon-chain/operations/attestations/prune_expired.go:14 +0xf5
created by github.com/prysmaticlabs/prysm/beacon-chain/operations/attestations.(*Service).Start
        beacon-chain/operations/attestations/service.go:60 +0x61

goroutine 159 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x7f22d57e1810, 0x72, 0x0)
        GOROOT/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000753e98, 0x72, 0x0, 0x0, 0x188540d)
        GOROOT/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        GOROOT/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000753e80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        GOROOT/src/internal/poll/fd_unix.go:384 +0x1d4
net.(*netFD).accept(0xc000753e80, 0xc000079c40, 0xc000090000, 0x7f22fd490e98)
        GOROOT/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc000766840, 0xc000079c80, 0x42ea18, 0x30)
        GOROOT/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc000766840, 0x17706a0, 0xc00075ef30, 0x1630de0, 0x285d4a0)
        GOROOT/src/net/tcpsock.go:261 +0x64
net/http.(*Server).Serve(0xc00076c000, 0x1c80780, 0xc000766840, 0x0, 0x0)
        GOROOT/src/net/http/server.go:2901 +0x25d
net/http.(*Server).ListenAndServe(0xc00076c000, 0x1, 0x0)
        GOROOT/src/net/http/server.go:2830 +0xb7
github.com/prysmaticlabs/prysm/beacon-chain/gateway.(*Gateway).Start.func1(0xc0005eaea0)
        beacon-chain/gateway/gateway.go:84 +0x47
created by github.com/prysmaticlabs/prysm/beacon-chain/gateway.(*Gateway).Start
        beacon-chain/gateway/gateway.go:83 +0x8b3

goroutine 274 [select, 2 minutes]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000727ef0, 0x1, 0x0, 0x0, 0x0, 0x0)
        external/org_golang_google_grpc/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00077c060, 0x0, 0x0)
        external/org_golang_google_grpc/internal/transport/controlbuf.go:513 +0x1cd
google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc00077a000)
        external/org_golang_google_grpc/internal/transport/http2_client.go:346 +0x7b
created by google.golang.org/grpc/internal/transport.newHTTP2Client
        external/org_golang_google_grpc/internal/transport/http2_client.go:344 +0xedb

goroutine 264 [select, 2 minutes]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0000d67d0, 0x1, 0x0, 0x0, 0x0, 0x0)
        external/org_golang_google_grpc/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000e30b40, 0x0, 0x0)
        external/org_golang_google_grpc/internal/transport/controlbuf.go:513 +0x1cd
google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc000604900)
        external/org_golang_google_grpc/internal/transport/http2_server.go:291 +0xcb
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        external/org_golang_google_grpc/internal/transport/http2_server.go:288 +0x10c1

goroutine 265 [select, 2 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000604900)
        external/org_golang_google_grpc/internal/transport/http2_server.go:966 +0x285
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        external/org_golang_google_grpc/internal/transport/http2_server.go:297 +0x10e6

goroutine 266 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x7f22d57e13b0, 0x72, 0xffffffffffffffff)
        GOROOT/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc00080a318, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
        GOROOT/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        GOROOT/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00080a300, 0xc001180000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        GOROOT/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc00080a300, 0xc001180000, 0x8000, 0x8000, 0x10401, 0x0, 0x1040100203000)
        GOROOT/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000828000, 0xc001180000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        GOROOT/src/net/net.go:184 +0x8e
bufio.(*Reader).Read(0xc000e30ae0, 0xc00080c2d8, 0x9, 0x9, 0xc0001df500, 0x7f22fd4907d0, 0x0)
        GOROOT/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x1c519c0, 0xc000e30ae0, 0xc00080c2d8, 0x9, 0x9, 0x9, 0x9c62a5, 0xc0007da300, 0xc0007d0004)
        GOROOT/src/io/io.go:310 +0x87
io.ReadFull(...)
        GOROOT/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc00080c2d8, 0x9, 0x9, 0x1c519c0, 0xc000e30ae0, 0x0, 0x0, 0xc0007da300, 0x0)
        external/org_golang_x_net/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00080c2a0, 0xc0007da300, 0x29a0000, 0x0, 0x0)
        external/org_golang_x_net/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000604900, 0xc0006106f0, 0x1a54ec0)
        external/org_golang_google_grpc/internal/transport/http2_server.go:453 +0x98
google.golang.org/grpc.(*Server).serveStreams(0xc0007881a0, 0x1c9b320, 0xc000604900)
        external/org_golang_google_grpc/server.go:742 +0xdb
google.golang.org/grpc.(*Server).handleRawConn.func1(0xc0007881a0, 0x1c9b320, 0xc000604900)
        external/org_golang_google_grpc/server.go:703 +0x3f
created by google.golang.org/grpc.(*Server).handleRawConn
        external/org_golang_google_grpc/server.go:702 +0x4fd

goroutine 139 [IO wait]:
internal/poll.runtime_pollWait(0x7f22d57e1650, 0x72, 0xffffffffffffffff)
        GOROOT/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc0005efe18, 0x72, 0xdf00, 0xdf3a, 0xffffffffffffffff)
        GOROOT/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        GOROOT/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0005efe00, 0xc001642000, 0xdf3a, 0xdf3a, 0x0, 0x0, 0x0)
        GOROOT/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc0005efe00, 0xc001642000, 0xdf3a, 0xdf3a, 0x203000, 0xc, 0xc00164601b)
        GOROOT/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000e2e080, 0xc001642000, 0xdf3a, 0xdf3a, 0x0, 0x0, 0x0)
        GOROOT/src/net/net.go:184 +0x8e
crypto/tls.(*atLeastReader).Read(0xc00174d600, 0xc001642000, 0xdf3a, 0xdf3a, 0x9f1f, 0xc001646016, 0xc0009539c8)
        GOROOT/src/crypto/tls/conn.go:760 +0x60
bytes.(*Buffer).ReadFrom(0xc000502cd8, 0x1c51b80, 0xc00174d600, 0x42c0f5, 0x1676e60, 0x181f260)
        GOROOT/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc000502a80, 0x1c542a0, 0xc000e2e080, 0x5, 0xc000e2e080, 0xc00164601b)
        GOROOT/src/crypto/tls/conn.go:782 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc000502a80, 0x0, 0x0, 0xc000953d38)
        GOROOT/src/crypto/tls/conn.go:589 +0x115
crypto/tls.(*Conn).readRecord(...)
        GOROOT/src/crypto/tls/conn.go:557
crypto/tls.(*Conn).Read(0xc000502a80, 0xc000811000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        GOROOT/src/crypto/tls/conn.go:1233 +0x15b
bufio.(*Reader).Read(0xc000501560, 0xc000ff49d8, 0x9, 0x9, 0xc000953d38, 0x1a56000, 0x7582e5)
        GOROOT/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x1c519c0, 0xc000501560, 0xc000ff49d8, 0x9, 0x9, 0x9, 0xc0000a8050, 0x0, 0x1c51d80)
        GOROOT/src/io/io.go:310 +0x87
io.ReadFull(...)
        GOROOT/src/io/io.go:329
net/http.http2readFrameHeader(0xc000ff49d8, 0x9, 0x9, 0x1c519c0, 0xc000501560, 0x0, 0x0, 0xc00237b020, 0x0)
        GOROOT/src/net/http/h2_bundle.go:1479 +0x87
net/http.(*http2Framer).ReadFrame(0xc000ff49a0, 0xc00237b020, 0x0, 0x0, 0x0)
        GOROOT/src/net/http/h2_bundle.go:1737 +0xa1
net/http.(*http2clientConnReadLoop).run(0xc000953fa8, 0xb, 0xc000900e80)
        GOROOT/src/net/http/h2_bundle.go:8246 +0x8d
net/http.(*http2ClientConn).readLoop(0xc0001a6900)
        GOROOT/src/net/http/h2_bundle.go:8174 +0x6f
created by net/http.(*http2Transport).newClientConn
        GOROOT/src/net/http/h2_bundle.go:7174 +0x64a
2naive commented 4 years ago 
[2020-08-04 05:08:34] ERROR main: Runtime panic: invalid freelist page: 0, page type is meta
goroutine 1 [running]:
runtime/debug.Stack(0xc0008e5050, 0x15c6500, 0xc0009566e0)
        GOROOT/src/runtime/debug/stack.go:24 +0x9d
main.main.func2(0xc000124310)
        beacon-chain/main.go:156 +0x57
panic(0x15c6500, 0xc0009566e0)
        GOROOT/src/runtime/panic.go:969 +0x166
go.etcd.io/bbolt.(*freelist).read(0xc0000ea500, 0x7f946f005000)
        external/io_etcd_go_bbolt/freelist.go:283 +0x2bb
go.etcd.io/bbolt.(*freelist).reload(0xc0000ea500, 0x7f946f005000)
        external/io_etcd_go_bbolt/freelist.go:351 +0x50
go.etcd.io/bbolt.(*Tx).rollback(0xc00013a0e0)
        external/io_etcd_go_bbolt/tx.go:286 +0xec
go.etcd.io/bbolt.(*DB).Update.func1(0xc00013a0e0)
        external/io_etcd_go_bbolt/db.go:686 +0x3c
panic(0x15c6500, 0xc0009566c0)
        GOROOT/src/runtime/panic.go:969 +0x166
go.etcd.io/bbolt.(*Cursor).search(0xc0008e57a8, 0x2898250, 0x14, 0x14, 0x4)
        external/io_etcd_go_bbolt/cursor.go:250 +0x353
go.etcd.io/bbolt.(*Cursor).seek(0xc0008e57a8, 0x2898250, 0x14, 0x14, 0x29ef560, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        external/io_etcd_go_bbolt/cursor.go:159 +0x7d
go.etcd.io/bbolt.(*Bucket).Bucket(0xc00013a0f8, 0x2898250, 0x14, 0x14, 0xc0005ae400)
        external/io_etcd_go_bbolt/bucket.go:105 +0xd5
go.etcd.io/bbolt.(*Tx).Bucket(...)
        external/io_etcd_go_bbolt/tx.go:102
github.com/prysmaticlabs/prysm/beacon-chain/db/kv.(*Store).verifySlotsPerArchivePoint.func1(0xc00013a0e0, 0xc000956601, 0xc00013a0e0)
        beacon-chain/db/kv/check_historical_state.go:54 +0x6a
go.etcd.io/bbolt.(*DB).Update(0xc0005ae400, 0x1a545f0, 0x0, 0x0)
        external/io_etcd_go_bbolt/db.go:694 +0x8e
github.com/prysmaticlabs/prysm/beacon-chain/db/kv.(*Store).verifySlotsPerArchivePoint(0xc000110980, 0xc000110840, 0x18ad09b)
        beacon-chain/db/kv/check_historical_state.go:53 +0x3a
github.com/prysmaticlabs/prysm/beacon-chain/db/kv.(*Store).HistoricalStatesDeleted(0xc000110980, 0x1c87b00, 0xc0003a43c0, 0x0, 0x0)
        beacon-chain/db/kv/check_historical_state.go:23 +0xd1
github.com/prysmaticlabs/prysm/beacon-chain/node.(*BeaconNode).startDB(0xc00014e840, 0xc0001107c0, 0x0, 0x29ed688)
        beacon-chain/node/node.go:315 +0x5e3
github.com/prysmaticlabs/prysm/beacon-chain/node.NewBeaconNode(0xc0001107c0, 0x4, 0x4, 0x0)
        beacon-chain/node/node.go:160 +0x85a
main.startNode(0xc0001107c0, 0xc0007c9e00, 0x0)
        beacon-chain/main.go:183 +0x96
github.com/urfave/cli/v2.(*App).RunContext(0xc0008ec300, 0x1c87a80, 0xc000040088, 0xc00003c1d0, 0x1, 0x1, 0x0, 0x0)
        external/com_github_urfave_cli_v2/app.go:315 +0x70b
github.com/urfave/cli/v2.(*App).Run(...)
        external/com_github_urfave_cli_v2/app.go:215
main.main()
        beacon-chain/main.go:161 +0x1be

panic: invalid page type: 0: 4
        panic: invalid freelist page: 0, page type is meta [recovered]
        panic: invalid freelist page: 0, page type is meta

goroutine 1 [running]:
main.main.func2(0xc000124310)
        beacon-chain/main.go:157 +0x12f
panic(0x15c6500, 0xc0009566e0)
        GOROOT/src/runtime/panic.go:969 +0x166
go.etcd.io/bbolt.(*freelist).read(0xc0000ea500, 0x7f946f005000)
        external/io_etcd_go_bbolt/freelist.go:283 +0x2bb
go.etcd.io/bbolt.(*freelist).reload(0xc0000ea500, 0x7f946f005000)
        external/io_etcd_go_bbolt/freelist.go:351 +0x50
go.etcd.io/bbolt.(*Tx).rollback(0xc00013a0e0)
        external/io_etcd_go_bbolt/tx.go:286 +0xec
go.etcd.io/bbolt.(*DB).Update.func1(0xc00013a0e0)
        external/io_etcd_go_bbolt/db.go:686 +0x3c
panic(0x15c6500, 0xc0009566c0)
        GOROOT/src/runtime/panic.go:969 +0x166
go.etcd.io/bbolt.(*Cursor).search(0xc0008e57a8, 0x2898250, 0x14, 0x14, 0x4)
        external/io_etcd_go_bbolt/cursor.go:250 +0x353
go.etcd.io/bbolt.(*Cursor).seek(0xc0008e57a8, 0x2898250, 0x14, 0x14, 0x29ef560, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        external/io_etcd_go_bbolt/cursor.go:159 +0x7d
go.etcd.io/bbolt.(*Bucket).Bucket(0xc00013a0f8, 0x2898250, 0x14, 0x14, 0xc0005ae400)
        external/io_etcd_go_bbolt/bucket.go:105 +0xd5
go.etcd.io/bbolt.(*Tx).Bucket(...)
        external/io_etcd_go_bbolt/tx.go:102
github.com/prysmaticlabs/prysm/beacon-chain/db/kv.(*Store).verifySlotsPerArchivePoint.func1(0xc00013a0e0, 0xc000956601, 0xc00013a0e0)
        beacon-chain/db/kv/check_historical_state.go:54 +0x6a
go.etcd.io/bbolt.(*DB).Update(0xc0005ae400, 0x1a545f0, 0x0, 0x0)
        external/io_etcd_go_bbolt/db.go:694 +0x8e
github.com/prysmaticlabs/prysm/beacon-chain/db/kv.(*Store).verifySlotsPerArchivePoint(0xc000110980, 0xc000110840, 0x18ad09b)
        beacon-chain/db/kv/check_historical_state.go:53 +0x3a
github.com/prysmaticlabs/prysm/beacon-chain/db/kv.(*Store).HistoricalStatesDeleted(0xc000110980, 0x1c87b00, 0xc0003a43c0, 0x0, 0x0)
        beacon-chain/db/kv/check_historical_state.go:23 +0xd1
github.com/prysmaticlabs/prysm/beacon-chain/node.(*BeaconNode).startDB(0xc00014e840, 0xc0001107c0, 0x0, 0x29ed688)
        beacon-chain/node/node.go:315 +0x5e3
github.com/prysmaticlabs/prysm/beacon-chain/node.NewBeaconNode(0xc0001107c0, 0x4, 0x4, 0x0)
        beacon-chain/node/node.go:160 +0x85a
main.startNode(0xc0001107c0, 0xc0007c9e00, 0x0)
        beacon-chain/main.go:183 +0x96
github.com/urfave/cli/v2.(*App).RunContext(0xc0008ec300, 0x1c87a80, 0xc000040088, 0xc00003c1d0, 0x1, 0x1, 0x0, 0x0)
        external/com_github_urfave_cli_v2/app.go:315 +0x70b
github.com/urfave/cli/v2.(*App).Run(...)
        external/com_github_urfave_cli_v2/app.go:215
main.main()
        beacon-chain/main.go:161 +0x1be
2naive commented 4 years ago

Ubuntu 20.04 LTS (GNU/Linux 4.4.0-17134-Microsoft x86_64)

2naive commented 4 years ago 
[2020-08-04 05:21:47]  INFO powchain: Processing deposits from Ethereum 1 chain deposits=1536 genesisValidators=1526
panic: invalid page type: 0: 4

goroutine 246 [running]:
go.etcd.io/bbolt.(*Cursor).search(0xc002999490, 0x2897c38, 0xd, 0xd, 0xccd)
        external/io_etcd_go_bbolt/cursor.go:250 +0x353
go.etcd.io/bbolt.(*Cursor).seek(0xc002999490, 0x2897c38, 0xd, 0xd, 0x4fb3f3, 0xc000a62c00, 0xc003568000, 0x4fb3f3, 0x4fb3f3, 0x0, ...)
        external/io_etcd_go_bbolt/cursor.go:159 +0x7d
go.etcd.io/bbolt.(*Bucket).Put(0xc001ab5fc0, 0x2897c38, 0xd, 0xd, 0xc003568000, 0x4fb3f3, 0x4fb3f3, 0xc000518400, 0xc0015f20e0)
        external/io_etcd_go_bbolt/bucket.go:295 +0x103
github.com/prysmaticlabs/prysm/beacon-chain/db/kv.(*Store).SavePowchainData.func1(0xc0015f20e0, 0x1, 0xc0015f20e0)
        beacon-chain/db/kv/powchain.go:23 +0xf9
go.etcd.io/bbolt.(*DB).Update(0xc000518400, 0xc0029995e8, 0x0, 0x0)
        external/io_etcd_go_bbolt/db.go:694 +0x8e
github.com/prysmaticlabs/prysm/beacon-chain/db/kv.(*Store).SavePowchainData(0xc0003d7240, 0x1c87a80, 0xc000040088, 0xc000a62c00, 0x0, 0x0)
        beacon-chain/db/kv/powchain.go:17 +0xd4
github.com/prysmaticlabs/prysm/beacon-chain/powchain.(*Service).ProcessLog(0xc00047f7a0, 0x1c87a80, 0xc000040088, 0xba384ade4f9fb307, 0xc587ac6d542b21ce, 0xdc3ffe8d, 0xc0018c1a80, 0x1, 0x4, 0xc0018e6d80, ...)
        beacon-chain/powchain/log_processing.go:92 +0x53c
github.com/prysmaticlabs/prysm/beacon-chain/powchain.(*Service).processPastLogs(0xc00047f7a0, 0x1c87a80, 0xc000040088, 0x44db3a7f5715fa9a, 0x49dbdb88b2d520a9)
        beacon-chain/powchain/log_processing.go:329 +0x307
github.com/prysmaticlabs/prysm/beacon-chain/powchain.(*Service).initPOWService(0xc00047f7a0)
        beacon-chain/powchain/service.go:638 +0x349
github.com/prysmaticlabs/prysm/beacon-chain/powchain.(*Service).run(0xc00047f7a0, 0xc0018b7da0)
        beacon-chain/powchain/service.go:653 +0x76
github.com/prysmaticlabs/prysm/beacon-chain/powchain.(*Service).Start.func1(0xc00047f7a0)
        beacon-chain/powchain/service.go:222 +0x50
created by github.com/prysmaticlabs/prysm/beacon-chain/powchain.(*Service).Start
        beacon-chain/powchain/service.go:220 +0x3f
nisdas commented 4 years ago

@2naive Are you running on WSL ? this is a common error when running old versions of WSL. You will have to update this.

2naive commented 4 years ago

@nisdas I'll try updating it, thanks.

2naive commented 4 years ago

@nisdas updated to WSL2, cannot reproduce anymore.

But have in validator: [2020-08-08 04:03:09] ERROR validator: Could not submit slot signature to beacon node: rpc error: code = DeadlineExceeded desc = context deadline exceeded slot=25214

-- repeating a few times per hour

dimiandre commented 3 years ago

I'm getting the same error with latest prysm version using docker

Already third time that happen to me during sync

[2020-11-01 01:46:29]  INFO blockchain: Finished applying state transition attestations=128 attesterSlashings=0 deposits=0 proposerSlashings=0 voluntaryExits=0
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x6b3061a816 pc=0x7f18bddfb34b]

runtime stack:
runtime.throw(0x647866, 0x2a)
    GOROOT/src/runtime/panic.go:1116 +0x72
runtime.sigpanic()
    GOROOT/src/runtime/signal_unix.go:679 +0x46a

dimi@vmi469439:~$ free -m
              total        used        free      shared  buff/cache   available
Mem:          16013         234       15571           0         207       15527
Swap:          2047           0        2047
dimi@vmi469439:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7.8G     0  7.8G   0% /dev
tmpfs           1.6G  656K  1.6G   1% /run
/dev/sda2       392G   13G  360G   4% /
tmpfs           7.9G     0  7.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda1       922M   73M  786M   9% /boot
tmpfs           1.6G     0  1.6G   0% /run/user/1000
dimi@vmi469439:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:    20.04
Codename:   focal

fulltrace.log