search

dashpay / platform

L2 solution for seriously fast decentralized applications for the Dash network
https://dashplatform.readme.io/docs/introduction-what-is-dash-platform
MIT License
72 stars 39 forks source link

dashmate autostarts and fails on system restart #265

Open dashameter opened 2 years ago

dashameter commented 2 years ago

restarting the system without first manually stopping dashmate results in the dashd containers falling over and tenderdash melting the cpu:

CONTAINER ID   IMAGE                           COMMAND                  CREATED       STATUS                          PORTS                                                                                                   NAMES
cf0f942d2a8a   envoyproxy/envoy:v1.16-latest   "/docker-entrypoint.…"   3 hours ago   Up 25 minutes                   0.0.0.0:3000->10000/tcp, :::3000->10000/tcp, 0.0.0.0:3010->50051/tcp, :::3010->50051/tcp                dash_masternode_local_1_dapi_envoy_1
2c4bce954167   dashpay/dapi:0.21               "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   2500-2501/tcp, 2510/tcp                                                                                 dash_masternode_local_1_dapi_api_1
2dea21632e0b   envoyproxy/envoy:v1.16-latest   "/docker-entrypoint.…"   3 hours ago   Up 25 minutes                   0.0.0.0:3200->10000/tcp, :::3200->10000/tcp, 0.0.0.0:3210->50051/tcp, :::3210->50051/tcp                dash_masternode_local_3_dapi_envoy_1
52a58c2f3a6b   envoyproxy/envoy:v1.16-latest   "/docker-entrypoint.…"   3 hours ago   Up 25 minutes                   0.0.0.0:3100->10000/tcp, :::3100->10000/tcp, 0.0.0.0:3110->50051/tcp, :::3110->50051/tcp                dash_masternode_local_2_dapi_envoy_1
9562bf05a20c   dashpay/dapi:0.21               "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   2500-2501/tcp, 2510/tcp                                                                                 dash_masternode_local_3_dapi_api_1
75ce1c9afc4b   dashpay/tenderdash:0.6.0        "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   0.0.0.0:26656->26656/tcp, :::26656->26656/tcp, 127.0.0.1:26657->26657/tcp, 26660/tcp                    dash_masternode_local_1_drive_tenderdash_1
02d90b813b18   dashpay/dapi:0.21               "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   2500-2501/tcp, 2510/tcp                                                                                 dash_masternode_local_2_dapi_api_1
44a255a284f9   dashpay/tenderdash:0.6.0        "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   26656-26657/tcp, 26660/tcp, 0.0.0.0:26856->26856/tcp, :::26856->26856/tcp, 127.0.0.1:26857->26857/tcp   dash_masternode_local_3_drive_tenderdash_1
7fd001ac1646   dashpay/tenderdash:0.6.0        "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   26656-26657/tcp, 26660/tcp, 0.0.0.0:26756->26756/tcp, :::26756->26756/tcp, 127.0.0.1:26757->26757/tcp   dash_masternode_local_2_drive_tenderdash_1
e392afd8fb7c   dashpay/dapi:0.21               "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   2500-2501/tcp, 2510/tcp                                                                                 dash_masternode_local_3_dapi_tx_filter_stream_1
b4f70b7a5862   dashpay/sentinel:1.6.0          "/bin/sh -c /run.sh"     3 hours ago   Up 25 minutes                                                                                                                           dash_masternode_local_3_sentinel_1
7463c27b39e3   dashpay/dapi:0.21               "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   2500-2501/tcp, 2510/tcp                                                                                 dash_masternode_local_2_dapi_tx_filter_stream_1
8c359ce7cd8b   dashpay/sentinel:1.6.0          "/bin/sh -c /run.sh"     3 hours ago   Up 25 minutes                                                                                                                           dash_masternode_local_2_sentinel_1
a7dd03c63fa7   dashpay/dapi:0.21               "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   2500-2501/tcp, 2510/tcp                                                                                 dash_masternode_local_1_dapi_tx_filter_stream_1
fb32f243fee5   dashpay/sentinel:1.6.0          "/bin/sh -c /run.sh"     3 hours ago   Up 25 minutes                                                                                                                           dash_masternode_local_1_sentinel_1
6ac2e68a3652   dashpay/dashd:0.18.0.0-beta1    "/docker-entrypoint.…"   3 hours ago   Restarting (1) 42 seconds ago                                                                                                           dash_masternode_local_2_core_1
47f396a261ee   mongo:4.2                       "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   27017/tcp                                                                                               dash_masternode_local_2_drive_mongodb_1
43bd24eedc16   mongo:4.2                       "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   27017/tcp                                                                                               dash_masternode_local_3_drive_mongodb_1
36175b229d2a   dashpay/dashd:0.18.0.0-beta1    "/docker-entrypoint.…"   3 hours ago   Restarting (1) 43 seconds ago                                                                                                           dash_masternode_local_3_core_1
9877df2e9929   mongo:4.2                       "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   27017/tcp                                                                                               dash_masternode_local_1_drive_mongodb_1
d881dfd6fe95   dashpay/dashd:0.18.0.0-beta1    "/docker-entrypoint.…"   3 hours ago   Restarting (1) 46 seconds ago

The dashd container log is:

2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) init message: Verifying blocks...
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) : The block database contains a block which appears to be from the future. This may be due to your computer's date and time being set incorrectly. Only rebuild the block database if you are sure that your computer's date and time are correct.
Please restart with -reindex or -reindex-chainstate to recover.
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Aborted block database rebuild. Exiting.
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Interrupting HTTP server                                                                      
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Interrupting HTTP RPC server
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Interrupting RPC
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) PrepareShutdown: In progress...
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Stopping HTTP RPC server
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Unregistering HTTP handler for / (exactmatch 1)
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Unregistering HTTP handler for /wallet/ (exactmatch 0)
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Stopping RPC
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) RPC stopped.
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Stopping HTTP server
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Waiting for HTTP worker threads to exit
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Exited http event loop
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Waiting for HTTP event thread to exit
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Stopped HTTP server
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) cl-schdlr thread exit
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) scheduler thread exit
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write block and undo data to disk started
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) leveldb: Generated table #127@0: 1708 keys, 110723 bytes
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) leveldb: Compacted 4@0 + 1@1 files => 110723 bytes
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write block and undo data to disk completed (0.00ms)
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write block index to disk started
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) leveldb: compacted to: files[ 0 1 0 0 0 0 0 ]                   
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) leveldb: Delete type=2 #114
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) leveldb: Delete type=2 #122
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) leveldb: Delete type=2 #119
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) leveldb: Delete type=2 #125
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) leveldb: Delete type=2 #116
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) WriteBatch memory usage: db=index, before=0.0MiB, after=0.0MiB
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write block index to disk completed (0.00ms)
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write coins cache to disk (0 coins, 0kB) started
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Writing final batch of 0.00 MiB
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) WriteBatch memory usage: db=chainstate, before=0.0MiB, after=0.0MiB
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Committed 0 changed transaction outputs (out of 0) to coin database...
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) WriteBatch memory usage: db=evodb, before=0.0MiB, after=0.0MiB
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write coins cache to disk (0 coins, 0kB) completed (0.00s)
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write block and undo data to disk started
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write block and undo data to disk completed (0.00ms)
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write block index to disk started
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) WriteBatch memory usage: db=index, before=0.0MiB, after=0.0MiB
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write block index to disk completed (0.00ms)
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write coins cache to disk (0 coins, 0kB) started
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Writing final batch of 0.00 MiB
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) WriteBatch memory usage: db=chainstate, before=0.0MiB, after=0.0MiB
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Committed 0 changed transaction outputs (out of 0) to coin database...
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) WriteBatch memory usage: db=evodb, before=0.0MiB, after=0.0MiB
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write coins cache to disk (0 coins, 0kB) completed (0.00s)
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) zmq: Shutdown notification interface
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) zmq: Shutdown notifier pubhashblock at tcp://0.0.0.0:29998
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) zmq: Shutdown notifier pubrawchainlocksig at tcp://0.0.0.0:29998
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) zmq: Shutdown notifier pubrawtx at tcp://0.0.0.0:29998
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) zmq: Shutdown notifier pubrawtxlock at tcp://0.0.0.0:29998
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) zmq: Shutdown notifier pubrawtxlocksig at tcp://0.0.0.0:29998
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) zmq: Close socket at address tcp://0.0.0.0:29998
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Shutdown: done
dash-maverick commented 2 years ago

please recheck this with v22

dashameter commented 2 years ago

retest failed, dashd is still falling over after a system reboot

markin-io commented 2 years ago

@dashameter as a workaround for now - after the restart, kill all dashmate related docker containers. You can just go with docker kill $(docker ps -q), but have in mind that it will kill ALl containers.

After that try with dashmate start.

SamKirby22 commented 2 years ago

Workaround blocked due to new Dashmate issue [to be raised]

dashameter commented 2 years ago

image

retest failed, tenderdash sits at 125% CPU after system reboot

shumkov commented 1 month ago

System restart problem still actual for local network due to unreliable mock time logic