search

cheqd / cheqd-node

Ledger/node software for cheqd's decentralised identity network. Built using the Cosmos SDK blockchain framework.
https://docs.cheqd.io/node
Apache License 2.0
64 stars 43 forks source link

[Bug]: Block 2 - panic: +2/3 committed an invalid block: wrong Block.Header.AppHash. #233

Closed jiraky closed 2 years ago

jiraky commented 2 years ago

Contact Details

mattia.zago@monokee.com

What happened?

New node installed following the up to date guide. Genesis transactions and seeds match the official ones here on Github. The node refuses to include 2nd block in the chain while catching up, crashes and prevents any further operation.

Reproduction steps

1. Clean install with debian packages on Ubuntu 20.04 LTS
2. Standard testnet configuration (https://github.com/cheqd/cheqd-node/blob/main/docs/setup-and-configure/debian/deb-package-install.md)
3. Terminate the `cheqd-noded` service via systemctl
4. Launch the node manually `cheqd-noded start`

Full Debug trace:

9:53AM INF starting ABCI with Tendermint
9:53AM INF Starting multiAppConn service impl=multiAppConn module=proxy
9:53AM INF Starting localClient service connection=query impl=localClient module=abci-client
9:53AM INF Starting localClient service connection=snapshot impl=localClient module=abci-client
9:53AM INF Starting localClient service connection=mempool impl=localClient module=abci-client
9:53AM INF Starting localClient service connection=consensus impl=localClient module=abci-client
9:53AM INF Starting EventBus service impl=EventBus module=events
9:53AM INF Starting PubSub service impl=PubSub module=pubsub
9:53AM INF Starting IndexerService service impl=IndexerService module=txindex
9:53AM INF ABCI Handshake App Info hash= height=0 module=consensus protocol-version=1 software-version=0.2.7
9:53AM INF ABCI Replay Blocks appHeight=0 module=consensus stateHeight=0 storeHeight=0
9:53AM INF asserting crisis invariants inv=0/11 module=x/crisis name=staking/module-accounts
9:53AM INF asserting crisis invariants inv=1/11 module=x/crisis name=staking/nonnegative-power
9:53AM INF asserting crisis invariants inv=2/11 module=x/crisis name=staking/positive-delegation
9:53AM INF asserting crisis invariants inv=3/11 module=x/crisis name=staking/delegator-shares
9:53AM INF asserting crisis invariants inv=4/11 module=x/crisis name=bank/nonnegative-outstanding
9:53AM INF asserting crisis invariants inv=5/11 module=x/crisis name=bank/total-supply
9:53AM INF asserting crisis invariants inv=6/11 module=x/crisis name=gov/module-account
9:53AM INF asserting crisis invariants inv=7/11 module=x/crisis name=distribution/nonnegative-outstanding
9:53AM INF asserting crisis invariants inv=8/11 module=x/crisis name=distribution/can-withdraw
9:53AM INF asserting crisis invariants inv=9/11 module=x/crisis name=distribution/reference-count
9:53AM INF asserting crisis invariants inv=10/11 module=x/crisis name=distribution/module-account
9:53AM INF asserted all invariants duration=1.040631 height=0 module=x/crisis
9:53AM INF created new capability module=ibc name=ports/transfer
9:53AM INF port binded module=x/ibc/port port=transfer
9:53AM INF claimed capability capability=1 module=transfer name=ports/transfer
9:53AM INF Completed ABCI Handshake - Tendermint and App are synced appHash= appHeight=0 module=consensus
9:53AM INF Version info block=11 p2p=8 tendermint_version=0.34.14
9:53AM INF This node is not a validator addr=1720885EAE477222162EAD20B71CBDC2E9C7C598 module=consensus pubKey=8HCqW5QmL22KZZyuwXPUxTOeJij2S5tC/mXRCNZ9khs=
9:53AM INF P2P Node ID ID=9cc89a291315a087576e3635cb204237328f10f1 file=/home/cheqd/.cheqdnode/config/node_key.json module=p2p
9:53AM INF Adding persistent peers addrs=[] module=p2p
9:53AM INF Adding unconditional peer ids ids=[] module=p2p
9:53AM INF Add our address to book addr={"id":"9cc89a291315a087576e3635cb204237328f10f1","ip":"137.117.191.58","port":26656} book=/home/cheqd/.cheqdnode/config/addrbook.json module=p2p
9:53AM INF Add our address to book addr={"id":"9cc89a291315a087576e3635cb204237328f10f1","ip":"0.0.0.0","port":26656} book=/home/cheqd/.cheqdnode/config/addrbook.json module=p2p
9:53AM DBG initialization: tmNode created
9:53AM INF Starting Node service impl=Node
9:53AM INF Starting pprof server laddr=localhost:6060
9:53AM INF Starting P2P Switch service impl="P2P Switch" module=p2p
9:53AM INF Starting Consensus service impl=ConsensusReactor module=consensus
9:53AM INF Starting RPC HTTP server on [::]:26657 module=rpc-server
9:53AM INF Reactor  module=consensus waitSync=false
9:53AM INF Starting State service impl=ConsensusState module=consensus
9:53AM INF Starting baseWAL service impl=baseWAL module=consensus wal=/home/cheqd/.cheqdnode/data/cs.wal/wal
9:53AM INF Starting Group service impl=Group module=consensus wal=/home/cheqd/.cheqdnode/data/cs.wal/wal
9:53AM INF Searching for height height=1 max=0 min=0 module=consensus wal=/home/cheqd/.cheqdnode/data/cs.wal/wal
9:53AM INF Searching for height height=0 max=0 min=0 module=consensus wal=/home/cheqd/.cheqdnode/data/cs.wal/wal
9:53AM INF Found height=0 index=0 module=consensus wal=/home/cheqd/.cheqdnode/data/cs.wal/wal
9:53AM INF Catchup by replaying consensus messages height=1 module=consensus
9:53AM INF Replay: Done module=consensus
9:53AM INF Starting TimeoutTicker service impl=TimeoutTicker module=consensus
9:53AM INF Starting Evidence service impl=Evidence module=evidence
9:53AM INF Starting StateSync service impl=StateSync module=statesync
9:53AM INF Starting PEX service impl=PEX module=pex
9:53AM DBG Starting timeout routine module=consensus
9:53AM INF Starting AddrBook service book=/home/cheqd/.cheqdnode/config/addrbook.json impl=AddrBook module=p2p
9:53AM DBG Received tick module=consensus new_ti={"duration":4927470540,"height":1,"round":0,"step":1} old_ti={"duration":0,"height":0,"round":0,"step":0}
9:53AM DBG Timer already stopped module=consensus
9:53AM DBG Scheduled timeout dur=4927.47054 height=1 module=consensus round=0 step=1
9:53AM INF Starting Mempool service impl=Mempool module=mempool
9:53AM INF Saving AddrBook to file book=/home/cheqd/.cheqdnode/config/addrbook.json module=p2p size=43
9:53AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
9:53AM DBG initialization: tmNode started
9:53AM INF Will dial address addr={"id":"b29ff7dcb5e7f94b719d7f7852382eabf8890d61","ip":"185.4.154.131","port":26656} module=pex
9:53AM INF Will dial address addr={"id":"8c45237d435247270b95edae33fd1a69341eb1bb","ip":"54.73.156.125","port":26656} module=pex
9:53AM INF Will dial address addr={"id":"accd45c888e52ee1bc568e4cc4fe3b665916c12a","ip":"128.199.125.114","port":26656} module=pex
9:53AM INF Will dial address addr={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656} module=pex
9:53AM INF Will dial address addr={"id":"bd995719a8eb75d0f90867a03a285891640d2480","ip":"178.128.141.196","port":26656} module=pex
9:53AM INF Will dial address addr={"id":"e9c8d27532b423b9b811218af21536adbd1ae10b","ip":"167.99.216.115","port":26656} module=pex
9:53AM INF Will dial address addr={"id":"895e08e5af49cb567cef14f62c114805295345ec","ip":"18.116.43.193","port":26656} module=pex
9:53AM INF Will dial address addr={"id":"626d30ef9f0d7734f6db2f54e5325c5ee3b6ac9c","ip":"54.154.56.228","port":26656} module=pex
9:53AM INF Will dial address addr={"id":"1132c4d7b096125945e7a72d0d6dac4a107f979c","ip":"123.58.213.123","port":26656} module=pex
9:53AM INF Will dial address addr={"id":"f6a68bf4f94774f75e2e4b55326670219729ccc6","ip":"34.200.247.255","port":26656} module=pex
9:53AM INF Dialing peer address={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656} module=p2p
9:53AM INF Dialing peer address={"id":"accd45c888e52ee1bc568e4cc4fe3b665916c12a","ip":"128.199.125.114","port":26656} module=p2p
9:53AM INF Dialing peer address={"id":"bd995719a8eb75d0f90867a03a285891640d2480","ip":"178.128.141.196","port":26656} module=p2p
9:53AM INF Dialing peer address={"id":"1132c4d7b096125945e7a72d0d6dac4a107f979c","ip":"123.58.213.123","port":26656} module=p2p
9:53AM INF Dialing peer address={"id":"b29ff7dcb5e7f94b719d7f7852382eabf8890d61","ip":"185.4.154.131","port":26656} module=p2p
9:53AM INF Dialing peer address={"id":"626d30ef9f0d7734f6db2f54e5325c5ee3b6ac9c","ip":"54.154.56.228","port":26656} module=p2p
9:53AM INF Dialing peer address={"id":"8c45237d435247270b95edae33fd1a69341eb1bb","ip":"54.73.156.125","port":26656} module=p2p
9:53AM INF Dialing peer address={"id":"e9c8d27532b423b9b811218af21536adbd1ae10b","ip":"167.99.216.115","port":26656} module=p2p
9:53AM INF Dialing peer address={"id":"895e08e5af49cb567cef14f62c114805295345ec","ip":"18.116.43.193","port":26656} module=p2p
9:53AM INF Dialing peer address={"id":"f6a68bf4f94774f75e2e4b55326670219729ccc6","ip":"34.200.247.255","port":26656} module=p2p
9:53AM ERR dialing failed (attempts: 1): dial tcp 167.99.216.115:26656: connect: connection refused addr={"id":"e9c8d27532b423b9b811218af21536adbd1ae10b","ip":"167.99.216.115","port":26656} module=pex
9:53AM INF Starting Peer service impl="Peer{MConn{23.88.39.28:26656} 7f03b15ce4eb1fbb817c91bfad98226a87a63a76 out}" module=p2p peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM INF Starting MConnection service impl=MConn{23.88.39.28:26656} module=p2p peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Send channel=32 conn={"Logger":{}} module=p2p msgBytes=0A1A0801180120FCFFFFFFFFFFFFFFFF0128FFFFFFFFFFFFFFFFFF01 peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Request addrs from={"Data":{},"Logger":{}} module=pex
9:53AM DBG Send channel=0 conn={"Logger":{}} module=p2p msgBytes=0A00 peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG TrySend channel=64 conn={"Logger":{}} module=p2p msgBytes=2A00 peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG No votes to send, sleeping localPC=_ localPV=_ module=consensus peer={"Data":{},"Logger":{}} peerPC=null peerPV=null prs.Height=0 rs.Height=1
9:53AM INF Added peer module=p2p peer={"Data":{},"Logger":{}}
9:53AM ERR dialing failed (attempts: 1): dial tcp 18.116.43.193:26656: connect: connection refused addr={"id":"895e08e5af49cb567cef14f62c114805295345ec","ip":"18.116.43.193","port":26656} module=pex
9:53AM DBG Flush conn={"Logger":{}} module=p2p peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Read PacketMsg conn={"Logger":{}} module=p2p packet={"data":"EvMFCj0KKDhjZDc2MDRmZmIyYjNkZjRkM2I5YjI3MGE3ZmExZDc4MzVmMTEyZWYSDTEzLjI0NS4zMy4yMDkYoNABCj0KKGI5ZjM4NDlmZjMzMjkyZDYxNWE0M2UxMzAyNWNjMjA1YzliYTBlNzUSDTg4Ljk5LjE4NS4yMjcYoNABCj8KKGFjY2Q0NWM4ODhlNTJlZTFiYzU2OGU0Y2M0ZmUzYjY2NTkxNmMxMmESDzEyOC4xOTkuMTI1LjExNBig0AEKOgooODdhMjhjZGUyNzlhNjYxMDIxZWM1MTA3MjAyYzk2NTBiMzFkZmRiNhIKMy4xOS4yNTEuNhig0AEKPQooZGM1OGM4ZDgwNmExMjQyOTRhODEyZDFkOWU3YjZhMGE3NTZjZmZmYxINMTQzLjExMC4yMTUuORig0AEKOwooOWM1ZTAyOWY4MTJlNzU3Mzg2Y2E2MDM3YTk0YjQwOGZjNzRmYWNlNxILMy4xNDQuMzYuNDYYoNABCj4KKDljYzg5YTI5MTMxNWEwODc1NzZlMzYzNWNiMjA0MjM3MzI4ZjEwZjESDjEzNy4xMTcuMTkxLjU4GKDQAQo8CihjZDI3OWQ1NmFjZTBmZmU5ODE3ZDRjMDY4ZThiYWMzZDVkMWY5Y2VmEgw5NC4xMzAuOTYuODIYoNABCj0KKGFhOTk2ZGRhN2UxYjEwNmZiMmFlYjgwMGU4ZTQ2YzE3YzkyNjgwOWISDTEzNy4xODQuNTkuMjkYoNABCj4KKGQwODM5OTM3ODRiOTM5NzUwYTRiNzhjYTY4YWFhMjY5M2FhODQyMTQSDjEzNC4xMjIuODIuMTA3GKDQAQo+CigxMTMyYzRkN2IwOTYxMjU5NDVlN2E3MmQwZDZkYWM0YTEwN2Y5NzljEg4xMjMuNTguMjEzLjEyMxig0AEKPQooODk1ZTA4ZTVhZjQ5Y2I1NjdjZWYxNGY2MmMxMTQ4MDUyOTUzNDVlYxINMTguMTE2LjQzLjE5Mxig0AE=","eof":true} peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Received bytes chID=0 module=p2p msgBytes="\x12�\x05\n=\n(8cd7604ffb2b3df4d3b9b270a7fa1d7835f112ef\x12\r13.245.33.209\x18��\x01\n=\n(b9f3849ff33292d615a43e13025cc205c9ba0e75\x12\r88.99.185.227\x18��\x01\n?\n(accd45c888e52ee1bc568e4cc4fe3b665916c12a\x12\x0f128.199.125.114\x18��\x01\n:\n(87a28cde279a661021ec5107202c9650b31dfdb6\x12\n3.19.251.6\x18��\x01\n=\n(dc58c8d806a124294a812d1d9e7b6a0a756cfffc\x12\r143.110.215.9\x18��\x01\n;\n(9c5e029f812e757386ca6037a94b408fc74face7\x12\v3.144.36.46\x18��\x01\n>\n(9cc89a291315a087576e3635cb204237328f10f1\x12\x0e137.117.191.58\x18��\x01\n<\n(cd279d56ace0ffe9817d4c068e8bac3d5d1f9cef\x12\f94.130.96.82\x18��\x01\n=\n(aa996dda7e1b106fb2aeb800e8e46c17c926809b\x12\r137.184.59.29\x18��\x01\n>\n(d083993784b939750a4b78ca68aaa2693aa84214\x12\x0e134.122.82.107\x18��\x01\n>\n(1132c4d7b096125945e7a72d0d6dac4a107f979c\x12\x0e123.58.213.123\x18��\x01\n=\n(895e08e5af49cb567cef14f62c114805295345ec\x12\r18.116.43.193\x18��\x01" peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Received message chId=0 module=pex msg={"addrs":[{"id":"8cd7604ffb2b3df4d3b9b270a7fa1d7835f112ef","ip":"13.245.33.209","port":26656},{"id":"b9f3849ff33292d615a43e13025cc205c9ba0e75","ip":"88.99.185.227","port":26656},{"id":"accd45c888e52ee1bc568e4cc4fe3b665916c12a","ip":"128.199.125.114","port":26656},{"id":"87a28cde279a661021ec5107202c9650b31dfdb6","ip":"3.19.251.6","port":26656},{"id":"dc58c8d806a124294a812d1d9e7b6a0a756cfffc","ip":"143.110.215.9","port":26656},{"id":"9c5e029f812e757386ca6037a94b408fc74face7","ip":"3.144.36.46","port":26656},{"id":"9cc89a291315a087576e3635cb204237328f10f1","ip":"137.117.191.58","port":26656},{"id":"cd279d56ace0ffe9817d4c068e8bac3d5d1f9cef","ip":"94.130.96.82","port":26656},{"id":"aa996dda7e1b106fb2aeb800e8e46c17c926809b","ip":"137.184.59.29","port":26656},{"id":"d083993784b939750a4b78ca68aaa2693aa84214","ip":"134.122.82.107","port":26656},{"id":"1132c4d7b096125945e7a72d0d6dac4a107f979c","ip":"123.58.213.123","port":26656},{"id":"895e08e5af49cb567cef14f62c114805295345ec","ip":"18.116.43.193","port":26656}]} src={"Data":{},"Logger":{}}
9:53AM DBG Failed to add new address err="Cannot add ourselves with address 9cc89a291315a087576e3635cb204237328f10f1@137.117.191.58:26656" module=pex
9:53AM DBG Read PacketMsg conn={"Logger":{}} module=p2p packet={"channel_id":64,"data":"KgYI8essEAE=","eof":true} peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Received bytes chID=64 module=p2p msgBytes="*\x06\b��,\x10\x01" peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Receive chID=64 module=blockchain msg={"base":1,"height":734705} src=7f03b15ce4eb1fbb817c91bfad98226a87a63a76
9:53AM DBG Read PacketMsg conn={"Logger":{}} module=p2p packet={"channel_id":32,"data":"CggI8ussGAEoAg==","eof":true} peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Received bytes chID=32 module=p2p msgBytes="\n\b\b��,\x18\x01(\x02" peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Receive chId=32 module=consensus msg={"Height":734706,"LastCommitRound":2,"Round":0,"SecondsSinceStartTime":0,"Step":1} src={"Data":{},"Logger":{}}
9:53AM DBG Read PacketMsg conn={"Logger":{}} module=p2p packet={"channel_id":33,"data":"KowDCAEahwMS3gIK0QIKBAgLEAESD2NoZXFkLXRlc3RuZXQtMhgBIgwI6cm3igYQuJTP3QIqAhIAMiDjsMRCmPwcFJr79MiZb7kkJ65B5GSbk0yklZkbeFK4VTog47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFVCINo71wa/4d6v42cs27tbzHD3V9Tm09SHoS4lh5JafjaeSiDaO9cGv+Her+NnLNu7W8xw91fU5tPUh6EuJYeSWn42nlIgDykIiDoQXHk7dElet9bfLupHntf8k0kgamXLD5mHoLhaIOOwxEKY/BwUmvv0yJlvuSQnrkHkZJuTTKSVmRt4UrhVYiDjsMRCmPwcFJr79MiZb7kkJ65B5GSbk0yklZkbeFK4VWog47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFVyFAuR484fZyAZKvxCYbDui8MYbvToEgAaACIEGgISABokCAEaIE+iQG4e80gH5FL9VitvkCReeY6Y3baAHHhjoSqO73aM","eof":true} peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Received bytes chID=33 module=p2p msgBytes="*�\x03\b\x01\x1a�\x03\x12�\x02\n�\x02\n\x04\b\v\x10\x01\x12\x0fcheqd-testnet-2\x18\x01\"\f\b�ɷ�\x06\x10����\x02*\x02\x12\x002 ���B��\x1c\x14���șo�$'�A�d��L���\x1bxR�U: ���B��\x1c\x14���șo�$'�A�d��L���\x1bxR�UB �;�\x06��ޯ�g,ۻ[�p�W���ԇ�.%��Z~6�J �;�\x06��ޯ�g,ۻ[�p�W���ԇ�.%��Z~6�R \x0f)\b�:\x10\\y;tI^���.�G����I je�\x0f����Z ���B��\x1c\x14���șo�$'�A�d��L���\x1bxR�Ub ���B��\x1c\x14���șo�$'�A�d��L���\x1bxR�Uj ���B��\x1c\x14���șo�$'�A�d��L���\x1bxR�Ur\x14\v���\x1fg \x19*�Ba����\x18n��\x12\x00\x1a\x00\"\x04\x1a\x02\x12\x00\x1a$\b\x01\x1a O�@n\x1e�H\a�R�V+o�$^y��ݶ�\x1cxc�*��v�" peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Receive chId=33 module=consensus msg={"Height":1,"Part":{"bytes":"0AD1020A04080B1001120F63686571642D746573746E65742D321801220C08E9C9B78A0610B894CFDD022A0212003220E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B8553A20E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B8554220DA3BD706BFE1DEAFE3672CDBBB5BCC70F757D4E6D3D487A12E2587925A7E369E4A20DA3BD706BFE1DEAFE3672CDBBB5BCC70F757D4E6D3D487A12E2587925A7E369E52200F2908883A105C793B74495EB7D6DF2EEA479ED7FC9349206A65CB0F9987A0B85A20E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B8556220E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B8556A20E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B85572140B91E3CE1F6720192AFC4261B0EE8BC3186EF4E812001A0022041A021200","index":0,"proof":{"aunts":null,"index":0,"leaf_hash":"T6JAbh7zSAfkUv1WK2+QJF55jpjdtoAceGOhKo7vdow=","total":1}},"Round":0} src={"Data":{},"Logger":{}}
9:53AM DBG Read PacketMsg conn={"Logger":{}} module=p2p packet={"channel_id":34,"data":"MrcBCrQBCAIQASJICiCpS9EXNuaIEatxIRO4IDY8mRbPFck3UN2TLyrQS+VTVBIkCAESIE+iQG4e80gH5FL9VitvkCReeY6Y3baAHHhjoSqO73aMKgwIsvHcigYQmoGpkQEyFAuR484fZyAZKvxCYbDui8MYbvToQkC1wyTrzYrn72XNnooJyHxVDvPpOh5+SVweME4WIAwMHgNUfhkVYlFxwAVuP+CDFAb9rUE3ItffUnmk0tCFp9QO","eof":true} peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Received bytes chID=34 module=p2p msgBytes="2�\x01\n�\x01\b\x02\x10\x01\"H\n �K�\x176��\x11�q!\x13� 6<�\x16�\x15�7Pݓ/*�K�ST\x12$\b\x01\x12 O�@n\x1e�H\a�R�V+o�$^y��ݶ�\x1cxc�*��v�*\f\b��܊\x06\x10����\x012\x14\v���\x1fg \x19*�Ba����\x18n��B@��$�͊��e͞�\t�|U\x0e��:\x1e~I\\\x1e0N\x16 \f\f\x1e\x03T~\x19\x15bQq�\x05n?��\x14\x06��A7\"��Ry��Ѕ��\x0e" peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG received a block part when we are not expecting any height=1 index=0 module=consensus peer=7f03b15ce4eb1fbb817c91bfad98226a87a63a76 round=0
9:53AM DBG Receive chId=34 module=consensus msg={"Vote":{"block_id":{"hash":"A94BD11736E68811AB712113B820363C9916CF15C93750DD932F2AD04BE55354","parts":{"hash":"4FA2406E1EF34807E452FD562B6F90245E798E98DDB6801C7863A12A8EEF768C","total":1}},"height":1,"round":0,"signature":"tcMk682K5+9lzZ6KCch8VQ7z6ToefklcHjBOFiAMDB4DVH4ZFWJRccAFbj/ggxQG/a1BNyLX31J5pNLQhafUDg==","timestamp":"2021-10-01T16:34:58.304758938Z","type":2,"validator_address":"0B91E3CE1F6720192AFC4261B0EE8BC3186EF4E8","validator_index":0}} src={"Data":{},"Logger":{}}
9:53AM DBG setHasVote H/R=1/0 index=0 module=consensus peerH/R=734706/0 type=2
9:53AM DBG adding vote cs_height=1 module=consensus val_index=0 vote_height=1 vote_type=2
9:53AM DBG Broadcast channel=32 module=p2p msgBytes=3A0408011802
9:53AM DBG added vote to precommit data=Votes:40000000000/40000000000(1.000) height=1 module=consensus round=0 validator=0B91E3CE1F6720192AFC4261B0EE8BC3186EF4E8 vote_timestamp=2021-10-01T16:34:58Z
9:53AM DBG need to set a buffer and log message here for sanity height=1 module=consensus now=2021-11-26T09:53:27Z round=0 start_time=2021-11-26T09:53:32Z
9:53AM DBG Send channel=32 conn={"Logger":{}} module=p2p msgBytes=3A0408011802 peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG entering new round current=1/0/RoundStepNewHeight height=1 module=consensus round=0
9:53AM DBG entering propose step current=1/0/RoundStepNewRound height=1 module=consensus round=0
9:53AM DBG node is a validator height=1 module=consensus round=0
9:53AM DBG Received tick module=consensus new_ti={"duration":3000000000,"height":1,"round":0,"step":3} old_ti={"duration":4927470540,"height":1,"round":0,"step":1}
9:53AM DBG node is not a validator addr=1720885EAE477222162EAD20B71CBDC2E9C7C598 height=1 module=consensus round=0 vals={"proposer":{"address":"0B91E3CE1F6720192AFC4261B0EE8BC3186EF4E8","proposer_priority":0,"pub_key":"29j2cmO9QnsW49AHwGpfYc/f12mTrG95iHCSe4WkUf8=","voting_power":40000000000},"validators":[{"address":"0B91E3CE1F6720192AFC4261B0EE8BC3186EF4E8","proposer_priority":0,"pub_key":"29j2cmO9QnsW49AHwGpfYc/f12mTrG95iHCSe4WkUf8=","voting_power":40000000000}]}
9:53AM DBG Scheduled timeout dur=3000 height=1 module=consensus round=0 step=3
9:53AM DBG Broadcast channel=32 module=p2p msgBytes=0A1A0801180320FCFFFFFFFFFFFFFFFF0128FFFFFFFFFFFFFFFFFF01
9:53AM DBG entering precommit step current=1/0/RoundStepPropose height=1 module=consensus round=0
9:53AM DBG precommit step; no +2/3 prevotes during enterPrecommit; precommitting nil height=1 module=consensus round=0
9:53AM DBG Send channel=32 conn={"Logger":{}} module=p2p msgBytes=0A1A0801180320FCFFFFFFFFFFFFFFFF0128FFFFFFFFFFFFFFFFFF01 peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Broadcast channel=32 module=p2p msgBytes=0A1A0801180620FCFFFFFFFFFFFFFFFF0128FFFFFFFFFFFFFFFFFF01
9:53AM DBG entering commit step commit_round=0 current=1/0/RoundStepPrecommit height=1 module=consensus
9:53AM INF commit is for a block we do not know about; set ProposalBlock=nil commit=A94BD11736E68811AB712113B820363C9916CF15C93750DD932F2AD04BE55354 commit_round=0 height=1 module=consensus proposal=
9:53AM DBG Send channel=32 conn={"Logger":{}} module=p2p msgBytes=0A1A0801180620FCFFFFFFFFFFFFFFFF0128FFFFFFFFFFFFFFFFFF01 peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Broadcast channel=32 module=p2p msgBytes=122F08011A24080112204FA2406E1EF34807E452FD562B6F90245E798E98DDB6801C7863A12A8EEF768C22050801120100
9:53AM DBG Broadcast channel=32 module=p2p msgBytes=0A1A0801180820FCFFFFFFFFFFFFFFFF0128FFFFFFFFFFFFFFFFFF01
9:53AM DBG Send channel=32 conn={"Logger":{}} module=p2p msgBytes=122F08011A24080112204FA2406E1EF34807E452FD562B6F90245E798E98DDB6801C7863A12A8EEF768C22050801120100 peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG failed attempt to finalize commit; we do not have the commit block commit_block=A94BD11736E68811AB712113B820363C9916CF15C93750DD932F2AD04BE55354 height=1 module=consensus proposal_block=
9:53AM DBG Send channel=32 conn={"Logger":{}} module=p2p msgBytes=0A1A0801180820FCFFFFFFFFFFFFFFFF0128FFFFFFFFFFFFFFFFFF01 peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Flush conn={"Logger":{}} module=p2p peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Read PacketMsg conn={"Logger":{}} module=p2p packet={"channel_id":33,"data":"KowDCAEahwMS3gIK0QIKBAgLEAESD2NoZXFkLXRlc3RuZXQtMhgBIgwI6cm3igYQuJTP3QIqAhIAMiDjsMRCmPwcFJr79MiZb7kkJ65B5GSbk0yklZkbeFK4VTog47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFVCINo71wa/4d6v42cs27tbzHD3V9Tm09SHoS4lh5JafjaeSiDaO9cGv+Her+NnLNu7W8xw91fU5tPUh6EuJYeSWn42nlIgDykIiDoQXHk7dElet9bfLupHntf8k0kgamXLD5mHoLhaIOOwxEKY/BwUmvv0yJlvuSQnrkHkZJuTTKSVmRt4UrhVYiDjsMRCmPwcFJr79MiZb7kkJ65B5GSbk0yklZkbeFK4VWog47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFVyFAuR484fZyAZKvxCYbDui8MYbvToEgAaACIEGgISABokCAEaIE+iQG4e80gH5FL9VitvkCReeY6Y3baAHHhjoSqO73aM","eof":true} peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Received bytes chID=33 module=p2p msgBytes="*�\x03\b\x01\x1a�\x03\x12�\x02\n�\x02\n\x04\b\v\x10\x01\x12\x0fcheqd-testnet-2\x18\x01\"\f\b�ɷ�\x06\x10����\x02*\x02\x12\x002 ���B��\x1c\x14���șo�$'�A�d��L���\x1bxR�U: ���B��\x1c\x14���șo�$'�A�d��L���\x1bxR�UB �;�\x06��ޯ�g,ۻ[�p�W���ԇ�.%��Z~6�J �;�\x06��ޯ�g,ۻ[�p�W���ԇ�.%��Z~6�R \x0f)\b�:\x10\\y;tI^���.�G����I je�\x0f����Z ���B��\x1c\x14���șo�$'�A�d��L���\x1bxR�Ub ���B��\x1c\x14���șo�$'�A�d��L���\x1bxR�Uj ���B��\x1c\x14���șo�$'�A�d��L���\x1bxR�Ur\x14\v���\x1fg \x19*�Ba����\x18n��\x12\x00\x1a\x00\"\x04\x1a\x02\x12\x00\x1a$\b\x01\x1a O�@n\x1e�H\a�R�V+o�$^y��ݶ�\x1cxc�*��v�" peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Receive chId=33 module=consensus msg={"Height":1,"Part":{"bytes":"0AD1020A04080B1001120F63686571642D746573746E65742D321801220C08E9C9B78A0610B894CFDD022A0212003220E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B8553A20E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B8554220DA3BD706BFE1DEAFE3672CDBBB5BCC70F757D4E6D3D487A12E2587925A7E369E4A20DA3BD706BFE1DEAFE3672CDBBB5BCC70F757D4E6D3D487A12E2587925A7E369E52200F2908883A105C793B74495EB7D6DF2EEA479ED7FC9349206A65CB0F9987A0B85A20E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B8556220E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B8556A20E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B85572140B91E3CE1F6720192AFC4261B0EE8BC3186EF4E812001A0022041A021200","index":0,"proof":{"aunts":null,"index":0,"leaf_hash":"T6JAbh7zSAfkUv1WK2+QJF55jpjdtoAceGOhKo7vdow=","total":1}},"Round":0} src={"Data":{},"Logger":{}}
9:53AM INF received complete proposal block hash=A94BD11736E68811AB712113B820363C9916CF15C93750DD932F2AD04BE55354 height=1 module=consensus
9:53AM INF finalizing commit of block hash=A94BD11736E68811AB712113B820363C9916CF15C93750DD932F2AD04BE55354 height=1 module=consensus num_txs=0 root=E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
9:53AM DBG Block{
  Header{
    Version:        {11 1}
    ChainID:        cheqd-testnet-2
    Height:         1
    Time:           2021-09-24 14:47:05.733203 +0000 UTC
    LastBlockID:    :0:000000000000
    LastCommit:     E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
    Data:           E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
    Validators:     DA3BD706BFE1DEAFE3672CDBBB5BCC70F757D4E6D3D487A12E2587925A7E369E
    NextValidators: DA3BD706BFE1DEAFE3672CDBBB5BCC70F757D4E6D3D487A12E2587925A7E369E
    App:            E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
    Consensus:      0F2908883A105C793B74495EB7D6DF2EEA479ED7FC9349206A65CB0F9987A0B8
    Results:        E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
    Evidence:       E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
    Proposer:       0B91E3CE1F6720192AFC4261B0EE8BC3186EF4E8
  }#A94BD11736E68811AB712113B820363C9916CF15C93750DD932F2AD04BE55354
  Data{

  }#E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
  EvidenceData{

  }#E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
  Commit{
    Height:     0
    Round:      0
    BlockID:    :0:000000000000
    Signatures:

  }#E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
}#A94BD11736E68811AB712113B820363C9916CF15C93750DD932F2AD04BE55354 height=1 module=consensus
9:53AM INF minted coins from module account amount=1584404859ncheq from=mint module=x/bank
9:53AM INF executed block height=1 module=state num_invalid_txs=0 num_valid_txs=0
9:53AM INF commit synced commit=436F6D6D697449447B5B32333820313837203131342031343220313138203438203231322030203130352031333320323134203133392031393420313230203138322031383820373720363020333220373120313030203230302031323020313632203636203134352031373720313633203330203133372032203230345D3A317D
9:53AM INF committed state app_hash=EEBB728E7630D4006985D68BC278B6BC4D3C204764C878A24291B1A31E8902CC height=1 module=state num_txs=0
9:53AM DBG Updating evidence pool last_block_height=1 last_block_time=2021-09-24T14:47:05Z module=evidence
9:53AM DBG Broadcast channel=32 module=p2p msgBytes=0A0F0802180120FCFFFFFFFFFFFFFFFF01
9:53AM DBG Send channel=32 conn={"Logger":{}} module=p2p msgBytes=0A0F0802180120FCFFFFFFFFFFFFFFFF01 peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Received tick module=consensus new_ti={"duration":4717751009,"height":2,"round":0,"step":1} old_ti={"duration":3000000000,"height":1,"round":0,"step":3}
9:53AM DBG Scheduled timeout dur=4717.751009 height=2 module=consensus round=0 step=1
9:53AM INF indexed block height=1 module=txindex
9:53AM DBG indexed block txs height=1 module=txindex num_txs=0
9:53AM INF Starting Peer service impl="Peer{MConn{123.58.213.123:26656} 1132c4d7b096125945e7a72d0d6dac4a107f979c out}" module=p2p peer={"id":"1132c4d7b096125945e7a72d0d6dac4a107f979c","ip":"123.58.213.123","port":26656}
9:53AM INF Starting MConnection service impl=MConn{123.58.213.123:26656} module=p2p peer={"id":"1132c4d7b096125945e7a72d0d6dac4a107f979c","ip":"123.58.213.123","port":26656}
9:53AM DBG TrySend channel=64 conn={"Logger":{}} module=p2p msgBytes=2A0408011001 peer={"id":"1132c4d7b096125945e7a72d0d6dac4a107f979c","ip":"123.58.213.123","port":26656}
9:53AM DBG Send channel=32 conn={"Logger":{}} module=p2p msgBytes=0A0F0802180120FCFFFFFFFFFFFFFFFF01 peer={"id":"1132c4d7b096125945e7a72d0d6dac4a107f979c","ip":"123.58.213.123","port":26656}
9:53AM DBG Request addrs from={"Data":{},"Logger":{}} module=pex
9:53AM DBG Send channel=0 conn={"Logger":{}} module=p2p msgBytes=0A00 peer={"id":"1132c4d7b096125945e7a72d0d6dac4a107f979c","ip":"123.58.213.123","port":26656}
9:53AM INF Added peer module=p2p peer={"Data":{},"Logger":{}}
9:53AM DBG No votes to send, sleeping localPC=_ localPV=_ module=consensus peer={"Data":{},"Logger":{}} peerPC=null peerPV=null prs.Height=0 rs.Height=2
9:53AM INF Starting Peer service impl="Peer{MConn{128.199.125.114:26656} accd45c888e52ee1bc568e4cc4fe3b665916c12a out}" module=p2p peer={"id":"accd45c888e52ee1bc568e4cc4fe3b665916c12a","ip":"128.199.125.114","port":26656}
9:53AM INF Starting MConnection service impl=MConn{128.199.125.114:26656} module=p2p peer={"id":"accd45c888e52ee1bc568e4cc4fe3b665916c12a","ip":"128.199.125.114","port":26656}
9:53AM DBG TrySend channel=64 conn={"Logger":{}} module=p2p msgBytes=2A0408011001 peer={"id":"accd45c888e52ee1bc568e4cc4fe3b665916c12a","ip":"128.199.125.114","port":26656}
9:53AM DBG Send channel=32 conn={"Logger":{}} module=p2p msgBytes=0A0F0802180120FCFFFFFFFFFFFFFFFF01 peer={"id":"accd45c888e52ee1bc568e4cc4fe3b665916c12a","ip":"128.199.125.114","port":26656}
9:53AM DBG Request addrs from={"Data":{},"Logger":{}} module=pex
9:53AM DBG Send channel=0 conn={"Logger":{}} module=p2p msgBytes=0A00 peer={"id":"accd45c888e52ee1bc568e4cc4fe3b665916c12a","ip":"128.199.125.114","port":26656}
9:53AM INF Added peer module=p2p peer={"Data":{},"Logger":{}}
9:53AM DBG No votes to send, sleeping localPC=_ localPV=_ module=consensus peer={"Data":{},"Logger":{}} peerPC=null peerPV=null prs.Height=0 rs.Height=2
9:53AM DBG Flush conn={"Logger":{}} module=p2p peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Flush conn={"Logger":{}} module=p2p peer={"id":"1132c4d7b096125945e7a72d0d6dac4a107f979c","ip":"123.58.213.123","port":26656}
9:53AM DBG Flush conn={"Logger":{}} module=p2p peer={"id":"accd45c888e52ee1bc568e4cc4fe3b665916c12a","ip":"128.199.125.114","port":26656}
9:53AM DBG Read PacketMsg conn={"Logger":{}} module=p2p packet={"channel_id":34,"data":"MrcBCrQBCAIQAiJICiCbrRvQ9kgFNN8RIDKuazRrDLFFBZFIvnYyenSdG8GWzxIkCAESIDrv+YMSk8cyJMdXtAIs0t2pltuCNAa7FVBnuYLLmlyWKgwIt/HcigYQgM/enQEyFAuR484fZyAZKvxCYbDui8MYbvToQkCoPIR9XohId5qq1vvlvlPt9v62bA4d7Keow2zMdSOymVDtxNG7qTWYdDM71YFzCpD5Nv+8axHSjev9E62PbHYF","eof":true} peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Received bytes chID=34 module=p2p msgBytes="2�\x01\n�\x01\b\x02\x10\x02\"H\n ��\x1b��H\x054�\x11 2�k4k\f�E\x05�H�v2zt�\x1b���\x12$\b\x01\x12 :���\x12��2$�W�\x02,�ݩ�ۂ4\x06�\x15Pg��˚\\�*\f\b��܊\x06\x10��ޝ\x012\x14\v���\x1fg \x19*�Ba����\x18n��B@�<�}^�Hw������S����l\x0e\x1d짨�l�u#��P��ѻ�5�t3;Ձs\n��6��k\x11ҍ��\x13��lv\x05" peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Receive chId=34 module=consensus msg={"Vote":{"block_id":{"hash":"9BAD1BD0F6480534DF112032AE6B346B0CB145059148BE76327A749D1BC196CF","parts":{"hash":"3AEFF9831293C73224C757B4022CD2DDA996DB823406BB155067B982CB9A5C96","total":1}},"height":2,"round":0,"signature":"qDyEfV6ISHeaqtb75b5T7fb+tmwOHeynqMNszHUjsplQ7cTRu6k1mHQzO9WBcwqQ+Tb/vGsR0o3r/ROtj2x2BQ==","timestamp":"2021-10-01T16:35:03.330803072Z","type":2,"validator_address":"0B91E3CE1F6720192AFC4261B0EE8BC3186EF4E8","validator_index":0}} src={"Data":{},"Logger":{}}
9:53AM DBG setHasVote H/R=2/0 index=0 module=consensus peerH/R=734706/0 type=2
9:53AM DBG Read PacketMsg conn={"Logger":{}} module=p2p packet={"channel_id":33,"data":"KoMFCAIa/gQS1QQKlQMKAggLEg9jaGVxZC10ZXN0bmV0LTIYAiIMCLLx3IoGEJqBqZEBKkgKIKlL0Rc25ogRq3EhE7ggNjyZFs8VyTdQ3ZMvKtBL5VNUEiQIARIgT6JAbh7zSAfkUv1WK2+QJF55jpjdtoAceGOhKo7vdowyIB31xQSCDYw1W+VJZfNfCxVG7YigkfLBynEb7rS4OBSnOiDjsMRCmPwcFJr79MiZb7kkJ65B5GSbk0yklZkbeFK4VUIg2jvXBr/h3q/jZyzbu1vMcPdX1ObT1IehLiWHklp+Np5KINo71wa/4d6v42cs27tbzHD3V9Tm09SHoS4lh5JafjaeUiAPKQiIOhBceTt0SV631t8u6kee1/yTSSBqZcsPmYeguFogyK+9oZZWA9ef6+0E8BjXiRDRv75LplZvfocOD7GYlb9iIOOwxEKY/BwUmvv0yJlvuSQnrkHkZJuTTKSVmRt4UrhVaiDjsMRCmPwcFJr79MiZb7kkJ65B5GSbk0yklZkbeFK4VXIUC5Hjzh9nIBkq/EJhsO6Lwxhu9OgSABoAIrYBCAEaSAogqUvRFzbmiBGrcSETuCA2PJkWzxXJN1Ddky8q0EvlU1QSJAgBEiBPokBuHvNIB+RS/VYrb5AkXnmOmN22gBx4Y6Eqju92jCJoCAISFAuR484fZyAZKvxCYbDui8MYbvToGgwIsvHcigYQmoGpkQEiQLXDJOvNiufvZc2eignIfFUO8+k6Hn5JXB4wThYgDAweA1R+GRViUXHABW4/4IMUBv2tQTci199SeaTS0IWn1A4aJAgBGiA67/mDEpPHMiTHV7QCLNLdqZbbgjQGuxVQZ7mCy5pclg==","eof":true} peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG adding vote cs_height=2 module=consensus val_index=0 vote_height=2 vote_type=2
9:53AM DBG Broadcast channel=32 module=p2p msgBytes=3A0408021802
9:53AM DBG added vote to precommit data=Votes:40000000000/40000000000(1.000) height=2 module=consensus round=0 validator=0B91E3CE1F6720192AFC4261B0EE8BC3186EF4E8 vote_timestamp=2021-10-01T16:35:03Z
9:53AM DBG need to set a buffer and log message here for sanity height=2 module=consensus now=2021-11-26T09:53:28Z round=0 start_time=2021-11-26T09:53:32Z
9:53AM DBG entering new round current=2/0/RoundStepNewHeight height=2 module=consensus round=0
9:53AM DBG Received bytes chID=33 module=p2p msgBytes="*�\x05\b\x02\x1a�\x04\x12�\x04\n�\x03\n\x02\b\v\x12\x0fcheqd-testnet-2\x18\x02\"\f\b��܊\x06\x10����\x01*H\n �K�\x176��\x11�q!\x13� 6<�\x16�\x15�7Pݓ/*�K�ST\x12$\b\x01\x12 O�@n\x1e�H\a�R�V+o�$^y��ݶ�\x1cxc�*��v�2 \x1d��\x04�\r�5[�Ie�_\v\x15F툠����q\x1b\ued388\x14�: ���B��\x1c\x14���șo�$'�A�d��L���\x1bxR�UB �;�\x06��ޯ�g,ۻ[�p�W���ԇ�.%��Z~6�J �;�\x06��ޯ�g,ۻ[�p�W���ԇ�.%��Z~6�R \x0f)\b�:\x10\\y;tI^���.�G����I je�\x0f����Z ȯ���V\x03ן��\x04�\x18\u05c9\x10ѿ�K�Vo~�\x0e\x0f����b ���B��\x1c\x14���șo�$'�A�d��L���\x1bxR�Uj ���B��\x1c\x14���șo�$'�A�d��L���\x1bxR�Ur\x14\v���\x1fg \x19*�Ba����\x18n��\x12\x00\x1a\x00\"�\x01\b\x01\x1aH\n �K�\x176��\x11�q!\x13� 6<�\x16�\x15�7Pݓ/*�K�ST\x12$\b\x01\x12 O�@n\x1e�H\a�R�V+o�$^y��ݶ�\x1cxc�*��v�\"h\b\x02\x12\x14\v���\x1fg \x19*�Ba����\x18n��\x1a\f\b��܊\x06\x10����\x01\"@��$�͊��e͞�\t�|U\x0e��:\x1e~I\\\x1e0N\x16 \f\f\x1e\x03T~\x19\x15bQq�\x05n?��\x14\x06��A7\"��Ry��Ѕ��\x0e\x1a$\b\x01\x1a :���\x12��2$�W�\x02,�ݩ�ۂ4\x06�\x15Pg��˚\\�" peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Send channel=32 conn={"Logger":{}} module=p2p msgBytes=3A0408021802 peer={"id":"7f03b15ce4eb1fbb817c91bfad98226a87a63a76","ip":"23.88.39.28","port":26656}
9:53AM DBG Send channel=32 conn={"Logger":{}} module=p2p msgBytes=3A0408021802 peer={"id":"1132c4d7b096125945e7a72d0d6dac4a107f979c","ip":"123.58.213.123","port":26656}
9:53AM DBG entering propose step current=2/0/RoundStepNewRound height=2 module=consensus round=0
9:53AM DBG Send channel=32 conn={"Logger":{}} module=p2p msgBytes=3A0408021802 peer={"id":"accd45c888e52ee1bc568e4cc4fe3b665916c12a","ip":"128.199.125.114","port":26656}
9:53AM DBG node is a validator height=2 module=consensus round=0
...

Bug prevalence

All the time I start the node

Environment

testnet

What browsers are you seeing the problem on?

No response

Relevant log output

$ cheqd-noded start
10:32AM INF starting ABCI with Tendermint
10:32AM INF Starting multiAppConn service impl=multiAppConn module=proxy
10:32AM INF Starting localClient service connection=query impl=localClient module=abci-client
10:32AM INF Starting localClient service connection=snapshot impl=localClient module=abci-client
10:32AM INF Starting localClient service connection=mempool impl=localClient module=abci-client
10:32AM INF Starting localClient service connection=consensus impl=localClient module=abci-client
10:32AM INF Starting EventBus service impl=EventBus module=events
10:32AM INF Starting PubSub service impl=PubSub module=pubsub
10:32AM INF Starting IndexerService service impl=IndexerService module=txindex
10:32AM INF ABCI Handshake App Info hash="��r�v0�\x00i�\u058b�x��M< Gd�x�B���\x1e�\x02�" height=1 module=consensus protocol-version=1 software-version=0.2.7
10:32AM INF ABCI Replay Blocks appHeight=1 module=consensus stateHeight=1 storeHeight=1
10:32AM INF Completed ABCI Handshake - Tendermint and App are synced appHash="��r�v0�\x00i�\u058b�x��M< Gd�x�B���\x1e�\x02�" appHeight=1 module=consensus
10:32AM INF Version info block=11 p2p=8 tendermint_version=0.34.14
10:32AM INF This node is not a validator addr=1720885EAE477222162EAD20B71CBDC2E9C7C598 module=consensus pubKey=8HCqW5QmL22KZZyuwXPUxTOeJij2S5tC/mXRCNZ9khs=
10:32AM INF P2P Node ID ID=9cc89a291315a087576e3635cb204237328f10f1 file=/home/cheqd/.cheqdnode/config/node_key.json module=p2p
10:32AM INF Adding persistent peers addrs=[] module=p2p
10:32AM INF Adding unconditional peer ids ids=[] module=p2p
10:32AM INF Add our address to book addr={"id":"9cc89a291315a087576e3635cb204237328f10f1","ip":"137.117.191.58","port":26656} book=/home/cheqd/.cheqdnode/config/addrbook.json module=p2p
10:32AM INF Add our address to book addr={"id":"9cc89a291315a087576e3635cb204237328f10f1","ip":"0.0.0.0","port":26656} book=/home/cheqd/.cheqdnode/config/addrbook.json module=p2p
10:32AM INF Starting Node service impl=Node
10:32AM INF Starting pprof server laddr=localhost:6060
10:32AM INF Starting P2P Switch service impl="P2P Switch" module=p2p
10:32AM INF Starting Mempool service impl=Mempool module=mempool
10:32AM INF Starting Consensus service impl=ConsensusReactor module=consensus
10:32AM INF Reactor  module=consensus waitSync=false
10:32AM INF Starting State service impl=ConsensusState module=consensus
10:32AM INF Starting RPC HTTP server on [::]:26657 module=rpc-server
10:32AM INF Starting baseWAL service impl=baseWAL module=consensus wal=/home/cheqd/.cheqdnode/data/cs.wal/wal
10:32AM INF Starting Group service impl=Group module=consensus wal=/home/cheqd/.cheqdnode/data/cs.wal/wal
10:32AM INF Searching for height height=2 max=0 min=0 module=consensus wal=/home/cheqd/.cheqdnode/data/cs.wal/wal
10:32AM INF Searching for height height=1 max=0 min=0 module=consensus wal=/home/cheqd/.cheqdnode/data/cs.wal/wal
10:32AM INF Found height=1 index=0 module=consensus wal=/home/cheqd/.cheqdnode/data/cs.wal/wal
10:32AM INF Catchup by replaying consensus messages height=2 module=consensus
10:32AM INF Replay: New Step height=2 module=consensus round=0 step=RoundStepNewHeight
10:32AM INF Replay: Vote blockID={"hash":"A94BD11736E68811AB712113B820363C9916CF15C93750DD932F2AD04BE55354","parts":{"hash":"4FA2406E1EF34807E452FD562B6F90245E798E98DDB6801C7863A12A8EEF768C","total":1}} height=1 module=consensus peer=dc58c8d806a124294a812d1d9e7b6a0a756cfffc round=0 type=2
10:32AM INF Replay: BlockPart height=1 module=consensus peer=dc58c8d806a124294a812d1d9e7b6a0a756cfffc round=0
10:32AM INF Replay: Vote blockID={"hash":"9BAD1BD0F6480534DF112032AE6B346B0CB145059148BE76327A749D1BC196CF","parts":{"hash":"3AEFF9831293C73224C757B4022CD2DDA996DB823406BB155067B982CB9A5C96","total":1}} height=2 module=consensus peer=87a28cde279a661021ec5107202c9650b31dfdb6 round=0 type=2
10:32AM INF commit is for a block we do not know about; set ProposalBlock=nil commit=9BAD1BD0F6480534DF112032AE6B346B0CB145059148BE76327A749D1BC196CF commit_round=0 height=2 module=consensus proposal=
10:32AM INF Replay: New Step height=2 module=consensus round=0 step=RoundStepPropose
10:32AM INF Replay: New Step height=2 module=consensus round=0 step=RoundStepPrecommit
10:32AM INF Replay: New Step height=2 module=consensus round=0 step=RoundStepCommit
10:32AM INF Replay: BlockPart height=2 module=consensus peer=87a28cde279a661021ec5107202c9650b31dfdb6 round=0
10:32AM INF received complete proposal block hash=9BAD1BD0F6480534DF112032AE6B346B0CB145059148BE76327A749D1BC196CF height=2 module=consensus
panic: +2/3 committed an invalid block: wrong Block.Header.AppHash.  Expected EEBB728E7630D4006985D68BC278B6BC4D3C204764C878A24291B1A31E8902CC, got C8AFBDA1965603D79FEBED04F018D78910D1BFBE4BA6566F7E870E0FB19895BF

goroutine 1 [running]:
github.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0xc0004cb880, 0x2)
        /home/runner/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/state.go:1575 +0x1385
github.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0xc0004cb880, 0x2)
        /home/runner/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/state.go:1546 +0x428
github.com/tendermint/tendermint/consensus.(*State).addProposalBlockPart(0xc0004cb880, 0xc0000c2ed0, 0xc000e01dd0, 0x28, 0xc00017b000, 0x15, 0x1f4)
        /home/runner/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/state.go:1919 +0x7c5
github.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc0004cb880, 0x22541c0, 0xc0000c2ed0, 0xc000e01dd0, 0x28)
        /home/runner/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/state.go:820 +0x665
github.com/tendermint/tendermint/consensus.(*State).readReplayMessage(0xc0004cb880, 0xc000d49590, 0x0, 0x0, 0x0, 0x0)
        /home/runner/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/replay.go:81 +0x3d8
github.com/tendermint/tendermint/consensus.(*State).catchupReplay(0xc0004cb880, 0x2, 0x0, 0x0)
        /home/runner/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/replay.go:160 +0x4b9
github.com/tendermint/tendermint/consensus.(*State).OnStart(0xc0004cb880, 0xc000e36630, 0x16)
        /home/runner/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/state.go:314 +0xa8
github.com/tendermint/tendermint/libs/service.(*BaseService).Start(0xc0004cb880, 0x2065500, 0xc00049ad00)
        /home/runner/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/libs/service/service.go:140 +0x322
github.com/tendermint/tendermint/consensus.(*Reactor).OnStart(0xc00049ad00, 0xc000042b80, 0x1a)
        /home/runner/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/reactor.go:83 +0x145
github.com/tendermint/tendermint/libs/service.(*BaseService).Start(0xc00049ad00, 0x0, 0x0)
        /home/runner/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/libs/service/service.go:140 +0x322
github.com/tendermint/tendermint/p2p.(*Switch).OnStart(0xc0004ced80, 0xc000042b20, 0x1b)
        /home/runner/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/p2p/switch.go:226 +0xb8
github.com/tendermint/tendermint/libs/service.(*BaseService).Start(0xc0004ced80, 0xc00049d700, 0x28)
        /home/runner/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/libs/service/service.go:140 +0x322
github.com/tendermint/tendermint/node.(*Node).OnStart(0xc0000e2580, 0xc000e36300, 0x15)
        /home/runner/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/node/node.go:941 +0x2ba
github.com/tendermint/tendermint/libs/service.(*BaseService).Start(0xc0000e2580, 0x1e1fae5, 0x1e)
        /home/runner/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/libs/service/service.go:140 +0x322
github.com/cosmos/cosmos-sdk/server.startInProcess(0xc001015300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x229c310, 0xc000e2e7f0, ...)
        /home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.3/server/start.go:278 +0x70e
github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc00101c500, 0x2fb3a80, 0x0, 0x0, 0x0, 0x0)
        /home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.3/server/start.go:129 +0x209
github.com/spf13/cobra.(*Command).execute(0xc00101c500, 0x2fb3a80, 0x0, 0x0, 0xc00101c500, 0x2fb3a80)
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:856 +0x472
github.com/spf13/cobra.(*Command).ExecuteC(0xc000d8b180, 0x0, 0x0, 0xc000b2ede0)
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:974 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:902
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:895
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0xc000d8b180, 0xc000b2ede0, 0x16, 0x22ae218, 0xc000e2e7f0)
        /home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.3/server/cmd/execute.go:36 +0x285
main.main()
        /home/runner/work/cheqd-node/cheqd-node/cmd/cheqd-noded/main.go:13 +0x45

Code of Conduct

ankurdotb commented 2 years ago

Hey @jiraky, are you running v0.2.7 or v0.3.1 when trying to connect to testnet? Testnet is currently on v0.2.7 and undergoing a software upgrade proposal to allow the upgrade to the new version.

jiraky commented 2 years ago

Hey @jiraky, are you running v0.2.7 or v0.3.1 when trying to connect to testnet? Testnet is currently on v0.2.7 and undergoing a software upgrade proposal to allow the upgrade to the new version.

I confirm version v0.2.7 for the testnet.

lampkin-diet commented 2 years ago

Hey, @jiraky . Can you try the same steps with 0.2.3 version? NOT NEEDED ANYMORE

lampkin-diet commented 2 years ago

Hey, @jiraky again. If it’s possible, can you run the next steps for 0.2.7 version, it should work:

jiraky commented 2 years ago

Hello, as posted on Slack, the result is the same. Cheers

ankurdotb commented 2 years ago

@jiraky Hey so sorry for the late reply here. Is there an IP address/node RPC I can look up? And I'm not sure whether there's been any progress since the last update.

jiraky commented 2 years ago

Hello, I have updated the testnode to v0.3.1. I have resetted the configurations and reconfigured the node entirely. I still get the same block error. The rpc is available at: http://node0.testnet.cheqd.monokee.com:26657/