search

iotaledger / goshimmer

Prototype implementation of IOTA 2.0
https://wiki.iota.org/goshimmer/welcome
Apache License 2.0
389 stars 113 forks source link

Node can't sync and then crash #1182

Closed sdellava closed 3 years ago

sdellava commented 3 years ago

I'm not able to provide more suggestion. db folder was empty. Docker image updated. After running up the container the node was not synced and after a while crashed without error and restart. Then the db is corrupted so it start looping. This is the log:

2021-04-07T17:51:30+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced 2021-04-07T17:51:37+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 116.203.53.228:14626 / VPkMyoe8ZDC 2021-04-07T17:51:39+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 208.87.129.29:14626 / ahCDDMBXrdY 2021-04-07T17:51:40+02:00 INFO Autopeering autopeering/plugin.go:77 Peering dropped: Yyhjq2zPkf8 2021-04-07T17:51:40+02:00 INFO Autopeering autopeering/plugin.go:73 Peering accepted: 95.216.169.178:14626 / Tj8vPSByGDj 2021-04-07T17:51:40+02:00 INFO Gossip gossip/neighbor.go:82 Connection closed {"id": "Yyhjq2zPkf8", "network": "tcp", "addr": "193.50.97.168:53152"} 2021-04-07T17:51:40+02:00 INFO Gossip gossip/neighbor.go:73 Connection established {"id": "Tj8vPSByGDj", "network": "tcp", "addr": "95.216.169.178:45136"} 2021-04-07T17:51:40+02:00 INFO Gossip gossip/plugin.go:118 Neighbor added: 95.216.169.178:14666 / Tj8vPSByGDj 2021-04-07T17:51:40+02:00 INFO Gossip gossip/plugin.go:121 Neighbor removed: 193.50.97.168:14666 / Yyhjq2zPkf8 2021-04-07T17:51:40+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced 2021-04-07T17:51:42+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 78.48.135.162:14626 / h3t31vuuzPi 2021-04-07T17:51:43+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 5.189.163.140:14627 / JhuafrP6KrZ 2021-04-07T17:51:47+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 213.136.79.161:24626 / d9rd9JpyS46 2021-04-07T17:51:49+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 85.235.64.234:14626 / 2RkArSEPwPJ 2021-04-07T17:51:50+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced 2021-04-07T17:51:53+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 51.103.76.211:14626 / UP6nxUaH9ZK 2021-04-07T17:51:59+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 178.254.32.205:14626 / U3SuAo2oPc5 2021-04-07T17:51:59+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 83.135.196.63:14626 / TW65rYd9cKs 2021-04-07T17:52:00+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced 2021-04-07T17:52:09+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 135.181.25.122:33502 / QP63dgE1U9N 2021-04-07T17:52:10+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced 2021-04-07T17:52:19+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 94.130.150.66:14626 / 4xiFJLsJCT1 2021-04-07T17:52:20+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced 2021-04-07T17:52:26+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 135.181.25.122:33501 / VzsXqXuxW9 2021-04-07T17:52:28+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 103.125.217.230:14626 / BsKYZdCKG89 2021-04-07T17:52:29+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 209.126.80.200:14626 / AufcZmaGqU7 2021-04-07T17:52:29+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 135.181.204.175:14626 / SowrtFkcssb 2021-04-07T17:52:30+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced 2021-04-07T17:52:31+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 95.217.236.29:14626 / D1H3HyYQZ9H 2021-04-07T17:52:39+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 85.214.62.43:14626 / W3ZC26eCykm 2021-04-07T17:52:40+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced 2021-04-07T17:52:43+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 20.188.62.21:14626 / bAy1nBXswJU 2021-04-07T17:52:46+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 188.34.179.239:14626 / F3vKR6sC2b8 2021-04-07T17:52:49+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 94.16.108.46:14626 / GvJ7j3YJwnF 2021-04-07T17:52:50+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced 2021-04-07T17:52:52+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 157.90.26.203:14626 / CsUHXSzN6Sm 2021-04-07T17:52:59+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 49.12.35.135:14626 / ManaJQevgSu 2021-04-07T17:53:00+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced 2021-04-07T17:53:00+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 45.83.107.51:14626 / X3jBaNkiXpC 2021-04-07T17:53:03+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 82.165.69.143:14626 / YTki6ebJBmN 2021-04-07T17:53:09+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 95.111.232.166:14626 / i3EXhsaMB4w 2021-04-07T17:53:10+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced 2021-04-07T17:53:19+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 78.47.163.157:14626 / BTYRFKPmC7o 2021-04-07T17:53:19+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 75.119.149.231:14626 / XbCTfJyfk5o 2021-04-07T17:53:20+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced 2021-04-07T17:53:26+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 159.69.84.196:33504 / 6FGgogQCBuJ 2021-04-07T17:53:29+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 207.180.252.59:14626 / 2Vfe2Po9ERJ 2021-04-07T17:53:30+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced 2021-04-07T17:53:35+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 159.69.84.196:33503 / EMGKbywwQ8T 2021-04-07T17:53:37+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 201.140.175.11:14626 / fw3kAezJwCY 2021-04-07T17:53:39+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 149.233.143.193:14626 / fCemFWjkXGn 2021-04-07T17:53:40+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced 2021-04-07T17:53:42+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 173.249.15.200:14626 / BdyZtDCZRtb 2021-04-07T17:53:47+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 75.119.139.115:14626 / cyqRqyCiKWb 2021-04-07T17:53:49+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 116.203.120.60:14626 / LVuUcv2EkFh 2021-04-07T17:53:50+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced 2021-04-07T17:53:53+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 78.47.253.83:14626 / RPDTENBCiA4 2021-04-07T17:53:54+02:00 INFO Autopeering autopeering/plugin.go:77 Peering dropped: Azvi1iRvf3n 2021-04-07T17:53:54+02:00 INFO Autopeering autopeering/plugin.go:73 Peering accepted: 207.180.252.59:14626 / 2Vfe2Po9ERJ 2021-04-07T17:53:54+02:00 INFO Gossip gossip/neighbor.go:82 Connection closed {"id": "Azvi1iRvf3n", "network": "tcp", "addr": "135.181.82.55:47402"} 2021-04-07T17:53:55+02:00 INFO Gossip gossip/neighbor.go:73 Connection established {"id": "2Vfe2Po9ERJ", "network": "tcp", "addr": "207.180.252.59:57696"} 2021-04-07T17:53:55+02:00 INFO Gossip gossip/plugin.go:118 Neighbor added: 207.180.252.59:14666 / 2Vfe2Po9ERJ 2021-04-07T17:53:55+02:00 INFO Gossip gossip/plugin.go:121 Neighbor removed: 135.181.82.55:14666 / Azvi1iRvf3n 2021-04-07T17:53:55+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 49.12.99.5:14626 / NnYRZB1Wkme 2021-04-07T17:53:59+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 52.164.187.150:14626 / aipGrHrQBXr 2021-04-07T17:54:00+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced 2021-04-07T17:54:08+02:00 WARN WebAPI webapi/plugin.go:65 Request failed: code=404, message=Not Found 2021-04-07T17:54:09+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 157.90.230.113:14626 / ees446BuK2z 2021-04-07T17:54:09+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 89.40.13.245:14626 / 7TrraBExNt4 2021-04-07T17:54:10+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced 2021-04-07T17:54:19+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 116.202.49.178:15626 / HGztmPGDTwk 2021-04-07T17:54:20+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced 2021-04-07T17:54:30+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced 2021-04-07T17:54:40+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced 2021-04-07T17:54:41+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 87.78.131.196:14626 / HiSoQut2uiQ 2021-04-07T17:54:45+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 135.181.95.103:14626 / Pe3dd5r4nyv 2021-04-07T17:54:50+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced 2021-04-07T17:54:51+02:00 INFO Autopeering autopeering/plugin.go:56 Discovered: 178.193.129.175:14627 / bzDgncvJMXt 2021-04-07T17:54:52+02:00 INFO Autopeering autopeering/plugin.go:68 Peering chosen: 178.193.129.175:14627 / bzDgncvJMXt 2021-04-07T17:54:52+02:00 INFO Gossip gossip/plugin.go:115 Connection to neighbor 178.193.129.175:14667 / bzDgncvJMXt failed: handshake 178.193.129.175:14667 / bzDgncvJMXt failed: EOF 2021-04-07T17:54:52+02:00 INFO Autopeering autopeering/plugin.go:77 Peering dropped: bzDgncvJMXt 2021-04-07T17:55:00+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced 2021-04-07T17:55:17+02:00 WARN MessageLayer node/plugin.go:109 error issuing statement: can't issue payload: tangle not synced

/ __|/ \ / | | | | | \/ | \/ | | \ | | | | | | (_ | || | | | | \ / | \ / | | | |_) | | | | | | | |_ | | | | | |\/| | |\/| | | | / | || | || |____) | | | || || | | | | | | || | \ \ _|__/|__/|| ||____|| ||| |_|__|| \\ v0.5.6

2021-04-07T17:55:23+02:00 INFO Node banner/plugin.go:50 GoShimmer version v0.5.6 ... 2021-04-07T17:55:23+02:00 INFO Node banner/plugin.go:51 Loading plugins ... 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: Banner ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: Config ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: Logger ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: CLI ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: Graceful Shutdown ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: PortCheck ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: Profiling ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: Database ... done 2021-04-07T17:55:23+02:00 INFO Local local/local.go:87 Initialized local: Peer{identity:B4EhbKUYPU5, publicKey:FNLgJ7dksvt8X2zVdmttNDoXoLfSnnJPLGuwD6QE3gXR, host:0.0.0.0:14626} 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: Autopeering ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: PoW ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: Clock ... done Loading snapshot... TransactionID: 11111111111111111111111111111111 Address: ED25519Address { Digest: 0x7ec9638605ad394675d4730ec33581d44d4fb03af02f61b463cab5d94759d13e Base58: "19XvUnuAHqoWv7VumNqWLBHfT2M2wWWTi6BKGjNG4arFw" } Address Base58: 19XvUnuAHqoWv7VumNqWLBHfT2M2wWWTi6BKGjNG4arFw Balance: ColoredBalances { IOTA: 1000000000000000 } 2021-04-07T17:55:23+02:00 INFO MessageLayer node/plugin.go:99 read snapshot from /snapshot.bin 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: MessageLayer ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: Gossip ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:86 Skipping Plugin: SyncBeacon 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: SyncBeaconFollower ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: DRNG ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:86 Skipping Plugin: Faucet 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: Consensus ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: Metrics ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: Spammer ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: Mana ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:86 Skipping Plugin: ManaEventLogger 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: RemoteLog ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:86 Skipping Plugin: Analysis-Server 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: Analysis-Client ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:86 Skipping Plugin: Analysis-Dashboard 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: Prometheus ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: NetworkDelay ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: Dashboard ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: WebAPI ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: WebAPI data Endpoint ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: WebAPI DRNG Endpoint ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: WebAPI faucet Endpoint ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: WebAPI healthz Endpoint ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: WebAPI message Endpoint ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: WebAPI autopeering Endpoint ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: WebAPI info Endpoint ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: WebAPI Value Endpoint ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: WebAPI tools Endpoint ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: WebAPI Mana Endpoint ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:95 Loading Plugin: WebAPI ledgerstate Endpoint ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:113 Executing plugins ... 2021-04-07T17:55:23+02:00 INFO Node node/node.go:117 Starting Plugin: Banner ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:117 Starting Plugin: Config ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:117 Starting Plugin: Logger ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:117 Starting Plugin: CLI ... done 2021-04-07T17:55:23+02:00 INFO Node node/node.go:117 Starting Plugin: Graceful Shutdown ... done 2021-04-07T17:55:23+02:00 INFO PortCheck portcheck/plugin.go:42 Testing autopeering service ... 2021-04-07T17:55:23+02:00 INFO PortCheck portcheck/plugin.go:74 Pong received from 116.202.49.178 2021-04-07T17:55:26+02:00 INFO PortCheck portcheck/plugin.go:44 Testing autopeering service ... done 2021-04-07T17:55:26+02:00 INFO Node node/node.go:117 Starting Plugin: PortCheck ... done 2021-04-07T17:55:26+02:00 INFO Profiling profiling/plugin.go:53 Profiling started, bind-address=0.0.0.0:6061 2021-04-07T17:55:26+02:00 INFO Node node/node.go:117 Starting Plugin: Profiling ... done 2021-04-07T17:55:26+02:00 FATAL Database database/plugin.go:94 The database is marked as not properly shutdown/corrupted, please delete the database folder and restart. github.com/iotaledger/goshimmer/plugins/database.configure /goshimmer/plugins/database/plugin.go:94 github.com/iotaledger/hive.go/node.pluginCaller /go/pkg/mod/github.com/iotaledger/hive.go@v0.0.0-20210327095456-8b9171badf36/node/events.go:14 github.com/iotaledger/hive.go/events.(Event).Trigger /go/pkg/mod/github.com/iotaledger/hive.go@v0.0.0-20210327095456-8b9171badf36/events/event.go:32 github.com/iotaledger/hive.go/node.(Node).Run /go/pkg/mod/github.com/iotaledger/hive.go@v0.0.0-20210327095456-8b9171badf36/node/node.go:116 github.com/iotaledger/hive.go/node.Run /go/pkg/mod/github.com/iotaledger/hive.go@v0.0.0-20210327095456-8b9171badf36/node/node.go:52 main.main /goshimmer/main.go:12 runtime.main /usr/local/go/src/runtime/proc.go:225

/ __|/ \ / | | | | | \/ | \/ | | \ | | | | | | (_ | || | | | | \ / | \ / | | | |_) | | | | | | | |_ | | | | | |\/| | |\/| | | | / | || | || |____) | | | || || | | | | | | || | \ \ _|__/|__/|| ||____|| ||| |_|__|| \\ v0.5.6

2021-04-07T17:55:27+02:00 INFO Node banner/plugin.go:50 GoShimmer version v0.5.6 ... 2021-04-07T17:55:27+02:00 INFO Node banner/plugin.go:51 Loading plugins ... 2021-04-07T17:55:27+02:00 INFO Node node/node.go:95 Loading Plugin: Banner ... done 2021-04-07T17:55:27+02:00 INFO Node node/node.go:95 Loading Plugin: Config ... done

JannemanDev commented 3 years ago

Is this the full log? Anything before 17:51:30?

The only odd thing in log seems to me failed: handshake 178.193.129.175:14667 / bzDgncvJMXt failed: EOF which is one of the last lines before the crash.

sdellava commented 3 years ago

Solved!. system stop the process due too out of memory. Added a swap file solved the issues.

I just suggest to add in the log that the program exit in response to a system request.