search

darwinia-network / dj

Darwinia bridge relayer tool implemented in nodejs
GNU General Public License v3.0
4 stars 5 forks source link

Batch relayer #43

Closed clearloop closed 4 years ago

clearloop commented 4 years ago

Solution

Note: Using Darwinia Relay API to batch relay directly has low continuance(fails easily, restart takes 3s

  1. Batch blocks from shadow services
  2. Keep relaying without checking finalized

Desc

Now the relay command has a new optional arg --batch [number] with a default value 1, for example, if we exec dj relay -b 6, things will happen as below:

BUG

If we relay too fast, for example, we had relayed 12 blocks already, but somehow it failed at No.6, all the left 6 blocks will be failed, and the service will restart from No.6.

Conclusion

Batch relay is faster than the old version, but not too many times, the relative safe signal is inBlock event(without Finalized, relay ordered blocks at a same fails easily), this makes relay process faster but safe less.

dj will keep both relay and batch relay, faster or higher success rate, this should be decided by the relayers.

Testing package

# version: 0.1.29-alpha.0@canary
yarn global add @darwinia/dj@canary

Steps

hackfisher commented 4 years ago

I thinks it make sense to remove the finalized prerequisite for relay extrinsics, since in most case, the finalized is quite stable and multi relayers are running at the same time, there are redundancy for extrinsic failure or finalization failure.

dj only need to honestly report what are happening in the relay process and improve if possible.

hackfisher commented 4 years ago

Will run relay on Crab using this after resetting the latest confirm block(genesis), This PR looks good to me, put a please review tag if ready for merging.

clearloop commented 4 years ago

Will run relay on Crab using this after resetting the latest confirm block(genesis), This PR looks good to me, put a please review tag if ready for merging.

Have you tried downloading the canary version?

Here is one more update on the relay service, but it will break API, I want to place it in next PR, (after upgrading the API of dargo.

yarn global add @darwinia/dj@canary

try 

# watch the log, it should continue to relay the next block after `inBlock` event
dj relay 

# check if the relay service batch relay as the passing arg
dj relay -b 6

# check if relay a specfied block works
dj relay -n <number>
hackfisher commented 4 years ago 
promote:~ denny$ dj relay -b 3
2020-05-28 00:55:51        API/INIT: RPC methods not decorated: balances_usableBalance, staking_powerOf
Unknown signed extensions CheckEthRelayHeaderHash found, treating them as no-effect
Unknown signed extensions CheckEthRelayHeaderHash found, treating them as no-effect
Unknown types found, no types for ElectionResultT
Unknown signed extensions CheckEthRelayHeaderHash found, treating them as no-effect
Unknown signed extensions CheckEthRelayHeaderHash found, treating them as no-effect
[ info ]: start from the lastest eth header of darwinia...
[ info ]: fetching proofs from 9965030 to 9965033...
[ event ]: relay block 9965031 to darwinia...
[ info ]: Transaction status: Ready
[ info ]: Ready
[ info ]: Transaction status: Broadcast
[ info ]: {"Broadcast":["QmVVq8SXrMcUJhd5voZMz8Cru2feFRBUoAUCbdVdDgVBr3","QmStpRcmjTjBJ5onzefm2nV6HxWHYDSEYzvokCVcJ9CjbF","QmWfdcdFaiCr6gfJvh7jjmQ6hUGshA4BrGSLSDMuFBf823","QmQy26JaaH7dUez67WXpGubWfBvstymjBZvm5UarGrFtM1","QmNY3sejrd4vRLRrxahmLScJ5DFpt8r2k9iPyFEZGvmRBr","Qmb1CWK3eSnvr2fspR4UHdhTwez3gWpwZ9ajMzWg75T76V","QmfFWwDJ3KrjHrtma5HccA6VKkXpLsw4oN77rS3pvsnwTT","QmXsTfB7yjY6VbWzZ3MuYvvp4s1oWMYLgDnQEmdDnZF736","QmXUEEeYVNbVcK1qe2ghQuwdZ9g69QvwExTeWLvXzpzNTf","QmdNGh7Zf7dcA9SkSayNNTwhwDaSwbqN1WG79DKPvf978Q","QmXopHqYzSZWMC66FjpxDznR7WiVYp87v6tTGkxcBAa6sz","QmSH8A6gkUa35nLDyceGCicVLbRsFWPmDbus3CVuuQfCdm","QmSDkrTsBD1ZpNffhm1uyYMxkv4tdyqWc4P6evV1uKJDPe","QmTu1hSK7eX1TnpbiVMB4BpKjbeiDvq8TeTL8cXSCGveC8","QmUeJYFxDCCrC3xcNHUiYKaU1RmU7XEKGSWnSrYhA77kxM","QmPhewyq3sCJyq8RphYSec13GrzBzchrpo8GjKVP8vFvtG","QmbsnrKGk5G9eAYCyasM8sxs5ivnn7RRMmvrQHuREZXM7Y","QmWddQyNgXQKdEuqC4U1WZUtK99Zu38tuAAcyJAiV9jJzq","QmRznqmDf38MhSyGxmdYrDDhGffVtzWkKv3dMf9DZzBuib","QmdmtXDLPy96nzXeBD1K5QAuWpe8YDsR6iPmRBQ3BvDNWt","QmcFb9KmSDBaiKGniBGtKCmggKHY3NLsB6WmeFTr3Z73yP","QmS36CESFrRQQxHKgnk9w88bVzTWFVUqESswrKrfSgmVWn","QmXe9b6hnQCGibxepRMfBbstZRg1vx9XxFYoK6WzkqFeFW","Qme2VxbvZFxNbF3xKxWBQczRzbcudmny9n3eSube73ZmEw","QmdFvFBfCMFMH8a8Umo3W1LuK3qtcRchHnf3ws5Pe6TG9H","QmemgR3HkSGSYpXeMAshycQppTXNNHgUBi1ggSWVKJMBmd","QmRFQ1fLEjcBb7RAi4wuW8Cj9pYy4jdsWx8aM7d6Z7vq8M","QmfJN2UB64RosSxmtpQQw5cwVSscG4ftZqswfjQeNzduiT","QmVBxSyZbtT9gH1ptFCAX6PA9iNAr8AGQvNJvTwwyLUppq"]}
[ info ]: Transaction status: InBlock
[ info ]: {"InBlock":"0x3ce12a887063868df916b492ecdb7b904e9c1bf8d9709c675a5341161fe9824e"}
[ info ]:   {"ApplyExtrinsic":1}: ethRelay.RelayHeader["5GBa6gfumh5rFiSJ7tvPUTYV7JRxYBj4n59g9ffGPqPh5ipo",{"parent_hash":"0xe87d41263ee64676a6e819eecd22c8618f1cf9098c6f58abd14d68233d021853","timestamp":1588131270,"number":9965031,"author":"0x99c85bb64564d9ef9a99621301f22c9993cb89e3","transactions_root":"0x473c2197df3013b03edabad61be02bd93c332f1fa2273b414f75572eb66576ff","uncles_hash":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","extra_data":"0x626565706f6f6c2e6f72675f36","state_root":"0x5cd9a5d970ea3d2c95006c70b4fac7c819928207ccef39506541366165c0d452","receipts_root":"0x9dac3de58157b28ad11faaac67d7143c0b8a805db19e776d4214cc2c36d3209a","log_bloom":"0x84d631261d82c61049e44c9a2b47e210a2e29be21c0804b597836373932c85200d60127546076547b188550726860514ebd1880229072822f90c8882302c1a0069b008701008cc8159a0e1c8c08054827f08274e42ec7065304bdfac8c426258527403142244418c130403940e6c0d78068c46ca94c00400198c2196524c64ae2d0480ce4ddcd92244317809a41984d91202cc9cc36045b3a54a50556e30004c0e4c660644fa32cdcc04239018d1da06c4e7e46cb00212a12b729485202111a2045d20c38baa1f948c928040819934522902e31b44a3511d412da00b926774842010712929c3342020a05410044d9103006f85c2e43088e340e3e30468b8e3d8","gas_used":9635791,"gas_limit":9970195,"difficulty":2185015788426802,"seal":["0xa0868ed31553b9579625020611f730046de79c01f0c73f93a37fecdaed6b890ffc","0x8858e53f88023077e1"],"hash":"0xf6a1096372ef1e169fed4ab056019b5258af6880232f62e419a97c2787846d2b"}]
[ info ]:   {"ApplyExtrinsic":1}: treasury.DepositRing[80000000]
[ info ]:   {"ApplyExtrinsic":1}: balances.Deposit["5GjpfZkbsD8daYtcoPQjT5KoRQ3sV5N6WxyhEBgcR1GT8Gnz",20000000]
[ info ]:   {"ApplyExtrinsic":1}: system.ExtrinsicSuccess[{"weight":200000000,"class":"Normal","paysFee":"Yes"}]
[ ok ]: Extrinsic relay header 9965031 is in block!
[ event ]: relay block 9965032 to darwinia...
[ info ]: Transaction status: Ready
[ info ]: Ready
[ info ]: Transaction status: Broadcast
[ info ]: {"Broadcast":["QmVVq8SXrMcUJhd5voZMz8Cru2feFRBUoAUCbdVdDgVBr3","QmStpRcmjTjBJ5onzefm2nV6HxWHYDSEYzvokCVcJ9CjbF","QmWfdcdFaiCr6gfJvh7jjmQ6hUGshA4BrGSLSDMuFBf823","QmQy26JaaH7dUez67WXpGubWfBvstymjBZvm5UarGrFtM1","QmNY3sejrd4vRLRrxahmLScJ5DFpt8r2k9iPyFEZGvmRBr","Qmb1CWK3eSnvr2fspR4UHdhTwez3gWpwZ9ajMzWg75T76V","QmfFWwDJ3KrjHrtma5HccA6VKkXpLsw4oN77rS3pvsnwTT","QmXsTfB7yjY6VbWzZ3MuYvvp4s1oWMYLgDnQEmdDnZF736","QmXUEEeYVNbVcK1qe2ghQuwdZ9g69QvwExTeWLvXzpzNTf","QmdNGh7Zf7dcA9SkSayNNTwhwDaSwbqN1WG79DKPvf978Q","QmXopHqYzSZWMC66FjpxDznR7WiVYp87v6tTGkxcBAa6sz","QmSH8A6gkUa35nLDyceGCicVLbRsFWPmDbus3CVuuQfCdm","QmSDkrTsBD1ZpNffhm1uyYMxkv4tdyqWc4P6evV1uKJDPe","QmTu1hSK7eX1TnpbiVMB4BpKjbeiDvq8TeTL8cXSCGveC8","QmUeJYFxDCCrC3xcNHUiYKaU1RmU7XEKGSWnSrYhA77kxM","QmPhewyq3sCJyq8RphYSec13GrzBzchrpo8GjKVP8vFvtG","QmbsnrKGk5G9eAYCyasM8sxs5ivnn7RRMmvrQHuREZXM7Y","QmWddQyNgXQKdEuqC4U1WZUtK99Zu38tuAAcyJAiV9jJzq","QmRznqmDf38MhSyGxmdYrDDhGffVtzWkKv3dMf9DZzBuib","QmdmtXDLPy96nzXeBD1K5QAuWpe8YDsR6iPmRBQ3BvDNWt","QmcFb9KmSDBaiKGniBGtKCmggKHY3NLsB6WmeFTr3Z73yP","QmS36CESFrRQQxHKgnk9w88bVzTWFVUqESswrKrfSgmVWn","QmXe9b6hnQCGibxepRMfBbstZRg1vx9XxFYoK6WzkqFeFW","Qme2VxbvZFxNbF3xKxWBQczRzbcudmny9n3eSube73ZmEw","QmdFvFBfCMFMH8a8Umo3W1LuK3qtcRchHnf3ws5Pe6TG9H","QmemgR3HkSGSYpXeMAshycQppTXNNHgUBi1ggSWVKJMBmd","QmRFQ1fLEjcBb7RAi4wuW8Cj9pYy4jdsWx8aM7d6Z7vq8M","QmfJN2UB64RosSxmtpQQw5cwVSscG4ftZqswfjQeNzduiT","QmVBxSyZbtT9gH1ptFCAX6PA9iNAr8AGQvNJvTwwyLUppq"]}
[ info ]: Transaction status: InBlock
[ info ]: {"InBlock":"0x65231f4b562e99ccec9d08ae68d4c1d796815cffa1ad134f82c86a7885fa037d"}
[ info ]:   {"ApplyExtrinsic":2}: ethRelay.RelayHeader["5GBa6gfumh5rFiSJ7tvPUTYV7JRxYBj4n59g9ffGPqPh5ipo",{"parent_hash":"0xf6a1096372ef1e169fed4ab056019b5258af6880232f62e419a97c2787846d2b","timestamp":1588131273,"number":9965032,"author":"0xeea5b82b61424df8020f5fedd81767f2d0d25bfb","transactions_root":"0xc7edfe755fdaba5a59cd0c600d3d4dffb1a5c3160c6456264025b60e25f238c3","uncles_hash":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","extra_data":"0x424a30312f4254432e434f4d2f0004d5eb","state_root":"0x7c3fc1874a828df3af2d53ac2342f660489c42d5dd1807e76a2e25956992aa3e","receipts_root":"0x9f2f70ffdab86245e45ed12e8907e635efd16958f4f6fbdb4e012762ba748dcc","log_bloom":"0x8300b08081522074141018535870e563e5b28431331cc0024904041007924d40011fbc3c4a0040b08e004986e260852661080080c128567030080c20982c804b084718107c0457510206c0ca048268a02718265022c9204eba4c4d10838024009680a006a6050440a8249250d0a13820042047a73cc90ad20c09a0328d306f302e94058a00c16810f000002927c96614345aec555a908087490d246708114ad0020c0092048a00d18c0002e840c188030131a00caa428144402794ad187c395a708805e205f0896808544695c41e945845104200018bf0165822e1c988507948801f61cb088654021a2444804231e08040b53026a49008d28830d6c829886428","gas_used":9629785,"gas_limit":9960460,"difficulty":2186082690667372,"seal":["0xa04ff49f5850283032ab01d18167399f0c52c44939d83369010031d3504e78e63e","0x88535bb7296e3adb4e"],"hash":"0x25f28ddc91c74017c7a8e4639105f00b72c5f97cf429fd1f6ac4218c7f3ee96d"}]
[ info ]:   {"ApplyExtrinsic":2}: treasury.DepositRing[80000000]
[ info ]:   {"ApplyExtrinsic":2}: balances.Deposit["5H92KwasoEsTor8QnmorTtVH4MCiwkzzSTXPUcowPzTAAGfE",20000000]
[ info ]:   {"ApplyExtrinsic":2}: system.ExtrinsicSuccess[{"weight":200000000,"class":"Normal","paysFee":"Yes"}]
[ ok ]: Extrinsic relay header 9965032 is in block!
[ event ]: relay block 9965033 to darwinia...
[ info ]: Transaction status: Ready
[ info ]: Ready
[ info ]: Transaction status: Broadcast
[ info ]: {"Broadcast":["QmVVq8SXrMcUJhd5voZMz8Cru2feFRBUoAUCbdVdDgVBr3","QmStpRcmjTjBJ5onzefm2nV6HxWHYDSEYzvokCVcJ9CjbF","QmWfdcdFaiCr6gfJvh7jjmQ6hUGshA4BrGSLSDMuFBf823","QmQy26JaaH7dUez67WXpGubWfBvstymjBZvm5UarGrFtM1","QmNY3sejrd4vRLRrxahmLScJ5DFpt8r2k9iPyFEZGvmRBr","Qmb1CWK3eSnvr2fspR4UHdhTwez3gWpwZ9ajMzWg75T76V","QmfFWwDJ3KrjHrtma5HccA6VKkXpLsw4oN77rS3pvsnwTT","QmXsTfB7yjY6VbWzZ3MuYvvp4s1oWMYLgDnQEmdDnZF736","QmXUEEeYVNbVcK1qe2ghQuwdZ9g69QvwExTeWLvXzpzNTf","QmdNGh7Zf7dcA9SkSayNNTwhwDaSwbqN1WG79DKPvf978Q","QmXopHqYzSZWMC66FjpxDznR7WiVYp87v6tTGkxcBAa6sz","QmSH8A6gkUa35nLDyceGCicVLbRsFWPmDbus3CVuuQfCdm","QmSDkrTsBD1ZpNffhm1uyYMxkv4tdyqWc4P6evV1uKJDPe","QmTu1hSK7eX1TnpbiVMB4BpKjbeiDvq8TeTL8cXSCGveC8","QmUeJYFxDCCrC3xcNHUiYKaU1RmU7XEKGSWnSrYhA77kxM","QmPhewyq3sCJyq8RphYSec13GrzBzchrpo8GjKVP8vFvtG","QmbsnrKGk5G9eAYCyasM8sxs5ivnn7RRMmvrQHuREZXM7Y","QmWddQyNgXQKdEuqC4U1WZUtK99Zu38tuAAcyJAiV9jJzq","QmRznqmDf38MhSyGxmdYrDDhGffVtzWkKv3dMf9DZzBuib","QmdmtXDLPy96nzXeBD1K5QAuWpe8YDsR6iPmRBQ3BvDNWt","QmcFb9KmSDBaiKGniBGtKCmggKHY3NLsB6WmeFTr3Z73yP","QmS36CESFrRQQxHKgnk9w88bVzTWFVUqESswrKrfSgmVWn","QmXe9b6hnQCGibxepRMfBbstZRg1vx9XxFYoK6WzkqFeFW","Qme2VxbvZFxNbF3xKxWBQczRzbcudmny9n3eSube73ZmEw","QmdFvFBfCMFMH8a8Umo3W1LuK3qtcRchHnf3ws5Pe6TG9H","QmemgR3HkSGSYpXeMAshycQppTXNNHgUBi1ggSWVKJMBmd","QmRFQ1fLEjcBb7RAi4wuW8Cj9pYy4jdsWx8aM7d6Z7vq8M","QmfJN2UB64RosSxmtpQQw5cwVSscG4ftZqswfjQeNzduiT","QmVBxSyZbtT9gH1ptFCAX6PA9iNAr8AGQvNJvTwwyLUppq"]}
[ info ]: Transaction status: Broadcast
[ info ]: {"Broadcast":["Qme2VxbvZFxNbF3xKxWBQczRzbcudmny9n3eSube73ZmEw"]}
[ info ]: Transaction status: InBlock
[ info ]: {"InBlock":"0xa3426e67210926bc8a3b7ed1fb2590635efc97b754768a1b77cef0f7854b6b83"}
[ info ]:   {"ApplyExtrinsic":2}: ethRelay.RelayHeader["5GBa6gfumh5rFiSJ7tvPUTYV7JRxYBj4n59g9ffGPqPh5ipo",{"parent_hash":"0x25f28ddc91c74017c7a8e4639105f00b72c5f97cf429fd1f6ac4218c7f3ee96d","timestamp":1588131280,"number":9965033,"author":"0x84a0d77c693adabe0ebc48f88b3ffff010577051","transactions_root":"0x575fea0ddcd2d20921754101f1e0936e9c38ca274fae6686ec39dda3f823c1e4","uncles_hash":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","extra_data":"0xd88301090b846765746888676f312e31332e38856c696e7578","state_root":"0xf966df69fbb6297a5074f87a1c2a689db0c036d631ad1b4f9a5714e10d4ef911","receipts_root":"0x28059b02ae4d7b8ae460828f68a75dd0f394b9f09f0b7d650913e4876988f8ac","log_bloom":"0x92de41a349f1a83b30103ec02e603e895ce318906d183060664bc762182b4d0311215575e5030d04e3294706568a07439280104709a64e97210ed18212cc808837342c408aa9dc14cbb8cf8f0086f4a73e4a2e28aaee204c0860678e0040f05b94a62d0482d0e01061adf78424018c6b44663668b8ee0e590a2535b2d010420a3c2598c7192cfc31ded502398889ec1e8c51dfb965a10ce4b0856834851e9a40fe4ce6940569b601be046881c422ecf2e33c605fb22e182b4c8632a058821d4160ba280e0c32309205067aa404d2a0642902c2ce560a913d9abb20431a956df2a710e4817988b20ceca41484885d314d4b49cf8a8251e300263952167d84386c","gas_used":9933104,"gas_limit":9950734,"difficulty":2187150113856302,"seal":["0xa067bb4e1fdf1b98acf0e6cad841d1e9c83f0ad2629ea6550a56554b6a25995f04","0x8844d12350020d6963"],"hash":"0x9dc9af76bd47537d72a94e33a9d56bcf6e90d704fc42ba4613bdc24027fa51a6"}]
[ info ]:   {"ApplyExtrinsic":2}: treasury.DepositRing[80000000]
[ info ]:   {"ApplyExtrinsic":2}: balances.Deposit["5GdsFsHXQLPCSSEWDZ5EFkwf1U5cLD15Evuz65My1ghPHKsx",20000000]
[ info ]:   {"ApplyExtrinsic":2}: system.ExtrinsicSuccess[{"weight":200000000,"class":"Normal","paysFee":"Yes"}]
[ ok ]: Extrinsic relay header 9965033 is in block!
[ info ]: fetching proofs from 9965033 to 9965036...
[ event ]: relay block 9965034 to darwinia...
2020-05-28 00:56:59        RPC-CORE: submitAndWatchExtrinsic(extrinsic: Extrinsic): ExtrinsicStatus:: 1010: Invalid Transaction: Stale
2020-05-28 00:56:59             DRR: Error: 1010: Invalid Transaction: Stale
    at RpcCoder._checkError (/Users/denny/.config/yarn/global/node_modules/@polkadot/rpc-provider/coder/index.js:83:13)
    at RpcCoder.decodeResponse (/Users/denny/.config/yarn/global/node_modules/@polkadot/rpc-provider/coder/index.js:46:10)
    at WsProvider.value (/Users/denny/.config/yarn/global/node_modules/@polkadot/rpc-provider/ws/Provider.js:166:90)
    at W3CWebSocket.value [as onmessage] (/Users/denny/.config/yarn/global/node_modules/@polkadot/rpc-provider/ws/Provider.js:146:153)
    at W3CWebSocket._dispatchEvent [as dispatchEvent] (/Users/denny/.config/yarn/global/node_modules/yaeti/lib/EventTarget.js:107:17)
    at W3CWebSocket.onMessage (/Users/denny/.config/yarn/global/node_modules/websocket/lib/W3CWebSocket.js:234:14)
    at WebSocketConnection.<anonymous> (/Users/denny/.config/yarn/global/node_modules/websocket/lib/W3CWebSocket.js:205:19)
    at WebSocketConnection.emit (events.js:315:20)
    at WebSocketConnection.processFrame (/Users/denny/.config/yarn/global/node_modules/websocket/lib/WebSocketConnection.js:554:26)
    at /Users/denny/.config/yarn/global/node_modules/websocket/lib/WebSocketConnection.js:323:40
2020-05-28 00:56:59        RPC-CORE: submitAndWatchExtrinsic(extrinsic: Extrinsic): ExtrinsicStatus:: 1010: Invalid Transaction: Stale
(node:42259) UnhandledPromiseRejectionWarning: Error: 1010: Invalid Transaction: Stale
    at RpcCoder._checkError (/Users/denny/.config/yarn/global/node_modules/@polkadot/rpc-provider/coder/index.js:83:13)
    at RpcCoder.decodeResponse (/Users/denny/.config/yarn/global/node_modules/@polkadot/rpc-provider/coder/index.js:46:10)
    at WsProvider.value (/Users/denny/.config/yarn/global/node_modules/@polkadot/rpc-provider/ws/Provider.js:166:90)
    at W3CWebSocket.value [as onmessage] (/Users/denny/.config/yarn/global/node_modules/@polkadot/rpc-provider/ws/Provider.js:146:153)
    at W3CWebSocket._dispatchEvent [as dispatchEvent] (/Users/denny/.config/yarn/global/node_modules/yaeti/lib/EventTarget.js:107:17)
    at W3CWebSocket.onMessage (/Users/denny/.config/yarn/global/node_modules/websocket/lib/W3CWebSocket.js:234:14)
    at WebSocketConnection.<anonymous> (/Users/denny/.config/yarn/global/node_modules/websocket/lib/W3CWebSocket.js:205:19)
    at WebSocketConnection.emit (events.js:315:20)
    at WebSocketConnection.processFrame (/Users/denny/.config/yarn/global/node_modules/websocket/lib/WebSocketConnection.js:554:26)
    at /Users/denny/.config/yarn/global/node_modules/websocket/lib/WebSocketConnection.js:323:40
(node:42259) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:42259) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
^C
hackfisher commented 4 years ago

The above error might related to nonce management:

if self.0 != account.nonce {
            return Err(
                if self.0 < account.nonce {
                    InvalidTransaction::Stale
                } else {
                    InvalidTransaction::Future
                }.into()
            )
        }
clearloop commented 4 years ago
Stale Error 
> promote:~ denny$ dj relay -b 3
> 2020-05-28 00:55:51        API/INIT: RPC methods not decorated: balances_usableBalance, staking_powerOf
> Unknown signed extensions CheckEthRelayHeaderHash found, treating them as no-effect
> Unknown signed extensions CheckEthRelayHeaderHash found, treating them as no-effect
> Unknown types found, no types for ElectionResultT
> Unknown signed extensions CheckEthRelayHeaderHash found, treating them as no-effect
> Unknown signed extensions CheckEthRelayHeaderHash found, treating them as no-effect
> [ info ]: start from the lastest eth header of darwinia...
> [ info ]: fetching proofs from 9965030 to 9965033...
> [ event ]: relay block 9965031 to darwinia...
> [ info ]: Transaction status: Ready
> [ info ]: Ready
> [ info ]: Transaction status: Broadcast
> [ info ]: {"Broadcast":["QmVVq8SXrMcUJhd5voZMz8Cru2feFRBUoAUCbdVdDgVBr3","QmStpRcmjTjBJ5onzefm2nV6HxWHYDSEYzvokCVcJ9CjbF","QmWfdcdFaiCr6gfJvh7jjmQ6hUGshA4BrGSLSDMuFBf823","QmQy26JaaH7dUez67WXpGubWfBvstymjBZvm5UarGrFtM1","QmNY3sejrd4vRLRrxahmLScJ5DFpt8r2k9iPyFEZGvmRBr","Qmb1CWK3eSnvr2fspR4UHdhTwez3gWpwZ9ajMzWg75T76V","QmfFWwDJ3KrjHrtma5HccA6VKkXpLsw4oN77rS3pvsnwTT","QmXsTfB7yjY6VbWzZ3MuYvvp4s1oWMYLgDnQEmdDnZF736","QmXUEEeYVNbVcK1qe2ghQuwdZ9g69QvwExTeWLvXzpzNTf","QmdNGh7Zf7dcA9SkSayNNTwhwDaSwbqN1WG79DKPvf978Q","QmXopHqYzSZWMC66FjpxDznR7WiVYp87v6tTGkxcBAa6sz","QmSH8A6gkUa35nLDyceGCicVLbRsFWPmDbus3CVuuQfCdm","QmSDkrTsBD1ZpNffhm1uyYMxkv4tdyqWc4P6evV1uKJDPe","QmTu1hSK7eX1TnpbiVMB4BpKjbeiDvq8TeTL8cXSCGveC8","QmUeJYFxDCCrC3xcNHUiYKaU1RmU7XEKGSWnSrYhA77kxM","QmPhewyq3sCJyq8RphYSec13GrzBzchrpo8GjKVP8vFvtG","QmbsnrKGk5G9eAYCyasM8sxs5ivnn7RRMmvrQHuREZXM7Y","QmWddQyNgXQKdEuqC4U1WZUtK99Zu38tuAAcyJAiV9jJzq","QmRznqmDf38MhSyGxmdYrDDhGffVtzWkKv3dMf9DZzBuib","QmdmtXDLPy96nzXeBD1K5QAuWpe8YDsR6iPmRBQ3BvDNWt","QmcFb9KmSDBaiKGniBGtKCmggKHY3NLsB6WmeFTr3Z73yP","QmS36CESFrRQQxHKgnk9w88bVzTWFVUqESswrKrfSgmVWn","QmXe9b6hnQCGibxepRMfBbstZRg1vx9XxFYoK6WzkqFeFW","Qme2VxbvZFxNbF3xKxWBQczRzbcudmny9n3eSube73ZmEw","QmdFvFBfCMFMH8a8Umo3W1LuK3qtcRchHnf3ws5Pe6TG9H","QmemgR3HkSGSYpXeMAshycQppTXNNHgUBi1ggSWVKJMBmd","QmRFQ1fLEjcBb7RAi4wuW8Cj9pYy4jdsWx8aM7d6Z7vq8M","QmfJN2UB64RosSxmtpQQw5cwVSscG4ftZqswfjQeNzduiT","QmVBxSyZbtT9gH1ptFCAX6PA9iNAr8AGQvNJvTwwyLUppq"]}
> [ info ]: Transaction status: InBlock
> [ info ]: {"InBlock":"0x3ce12a887063868df916b492ecdb7b904e9c1bf8d9709c675a5341161fe9824e"}
> [ info ]:     {"ApplyExtrinsic":1}: ethRelay.RelayHeader["5GBa6gfumh5rFiSJ7tvPUTYV7JRxYBj4n59g9ffGPqPh5ipo",{"parent_hash":"0xe87d41263ee64676a6e819eecd22c8618f1cf9098c6f58abd14d68233d021853","timestamp":1588131270,"number":9965031,"author":"0x99c85bb64564d9ef9a99621301f22c9993cb89e3","transactions_root":"0x473c2197df3013b03edabad61be02bd93c332f1fa2273b414f75572eb66576ff","uncles_hash":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","extra_data":"0x626565706f6f6c2e6f72675f36","state_root":"0x5cd9a5d970ea3d2c95006c70b4fac7c819928207ccef39506541366165c0d452","receipts_root":"0x9dac3de58157b28ad11faaac67d7143c0b8a805db19e776d4214cc2c36d3209a","log_bloom":"0x84d631261d82c61049e44c9a2b47e210a2e29be21c0804b597836373932c85200d60127546076547b188550726860514ebd1880229072822f90c8882302c1a0069b008701008cc8159a0e1c8c08054827f08274e42ec7065304bdfac8c426258527403142244418c130403940e6c0d78068c46ca94c00400198c2196524c64ae2d0480ce4ddcd92244317809a41984d91202cc9cc36045b3a54a50556e30004c0e4c660644fa32cdcc04239018d1da06c4e7e46cb00212a12b729485202111a2045d20c38baa1f948c928040819934522902e31b44a3511d412da00b926774842010712929c3342020a05410044d9103006f85c2e43088e340e3e30468b8e3d8","gas_used":9635791,"gas_limit":9970195,"difficulty":2185015788426802,"seal":["0xa0868ed31553b9579625020611f730046de79c01f0c73f93a37fecdaed6b890ffc","0x8858e53f88023077e1"],"hash":"0xf6a1096372ef1e169fed4ab056019b5258af6880232f62e419a97c2787846d2b"}]
> [ info ]:     {"ApplyExtrinsic":1}: treasury.DepositRing[80000000]
> [ info ]:     {"ApplyExtrinsic":1}: balances.Deposit["5GjpfZkbsD8daYtcoPQjT5KoRQ3sV5N6WxyhEBgcR1GT8Gnz",20000000]
> [ info ]:     {"ApplyExtrinsic":1}: system.ExtrinsicSuccess[{"weight":200000000,"class":"Normal","paysFee":"Yes"}]
> [ ok ]: Extrinsic relay header 9965031 is in block!
> [ event ]: relay block 9965032 to darwinia...
> [ info ]: Transaction status: Ready
> [ info ]: Ready
> [ info ]: Transaction status: Broadcast
> [ info ]: {"Broadcast":["QmVVq8SXrMcUJhd5voZMz8Cru2feFRBUoAUCbdVdDgVBr3","QmStpRcmjTjBJ5onzefm2nV6HxWHYDSEYzvokCVcJ9CjbF","QmWfdcdFaiCr6gfJvh7jjmQ6hUGshA4BrGSLSDMuFBf823","QmQy26JaaH7dUez67WXpGubWfBvstymjBZvm5UarGrFtM1","QmNY3sejrd4vRLRrxahmLScJ5DFpt8r2k9iPyFEZGvmRBr","Qmb1CWK3eSnvr2fspR4UHdhTwez3gWpwZ9ajMzWg75T76V","QmfFWwDJ3KrjHrtma5HccA6VKkXpLsw4oN77rS3pvsnwTT","QmXsTfB7yjY6VbWzZ3MuYvvp4s1oWMYLgDnQEmdDnZF736","QmXUEEeYVNbVcK1qe2ghQuwdZ9g69QvwExTeWLvXzpzNTf","QmdNGh7Zf7dcA9SkSayNNTwhwDaSwbqN1WG79DKPvf978Q","QmXopHqYzSZWMC66FjpxDznR7WiVYp87v6tTGkxcBAa6sz","QmSH8A6gkUa35nLDyceGCicVLbRsFWPmDbus3CVuuQfCdm","QmSDkrTsBD1ZpNffhm1uyYMxkv4tdyqWc4P6evV1uKJDPe","QmTu1hSK7eX1TnpbiVMB4BpKjbeiDvq8TeTL8cXSCGveC8","QmUeJYFxDCCrC3xcNHUiYKaU1RmU7XEKGSWnSrYhA77kxM","QmPhewyq3sCJyq8RphYSec13GrzBzchrpo8GjKVP8vFvtG","QmbsnrKGk5G9eAYCyasM8sxs5ivnn7RRMmvrQHuREZXM7Y","QmWddQyNgXQKdEuqC4U1WZUtK99Zu38tuAAcyJAiV9jJzq","QmRznqmDf38MhSyGxmdYrDDhGffVtzWkKv3dMf9DZzBuib","QmdmtXDLPy96nzXeBD1K5QAuWpe8YDsR6iPmRBQ3BvDNWt","QmcFb9KmSDBaiKGniBGtKCmggKHY3NLsB6WmeFTr3Z73yP","QmS36CESFrRQQxHKgnk9w88bVzTWFVUqESswrKrfSgmVWn","QmXe9b6hnQCGibxepRMfBbstZRg1vx9XxFYoK6WzkqFeFW","Qme2VxbvZFxNbF3xKxWBQczRzbcudmny9n3eSube73ZmEw","QmdFvFBfCMFMH8a8Umo3W1LuK3qtcRchHnf3ws5Pe6TG9H","QmemgR3HkSGSYpXeMAshycQppTXNNHgUBi1ggSWVKJMBmd","QmRFQ1fLEjcBb7RAi4wuW8Cj9pYy4jdsWx8aM7d6Z7vq8M","QmfJN2UB64RosSxmtpQQw5cwVSscG4ftZqswfjQeNzduiT","QmVBxSyZbtT9gH1ptFCAX6PA9iNAr8AGQvNJvTwwyLUppq"]}
> [ info ]: Transaction status: InBlock
> [ info ]: {"InBlock":"0x65231f4b562e99ccec9d08ae68d4c1d796815cffa1ad134f82c86a7885fa037d"}
> [ info ]:     {"ApplyExtrinsic":2}: ethRelay.RelayHeader["5GBa6gfumh5rFiSJ7tvPUTYV7JRxYBj4n59g9ffGPqPh5ipo",{"parent_hash":"0xf6a1096372ef1e169fed4ab056019b5258af6880232f62e419a97c2787846d2b","timestamp":1588131273,"number":9965032,"author":"0xeea5b82b61424df8020f5fedd81767f2d0d25bfb","transactions_root":"0xc7edfe755fdaba5a59cd0c600d3d4dffb1a5c3160c6456264025b60e25f238c3","uncles_hash":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","extra_data":"0x424a30312f4254432e434f4d2f0004d5eb","state_root":"0x7c3fc1874a828df3af2d53ac2342f660489c42d5dd1807e76a2e25956992aa3e","receipts_root":"0x9f2f70ffdab86245e45ed12e8907e635efd16958f4f6fbdb4e012762ba748dcc","log_bloom":"0x8300b08081522074141018535870e563e5b28431331cc0024904041007924d40011fbc3c4a0040b08e004986e260852661080080c128567030080c20982c804b084718107c0457510206c0ca048268a02718265022c9204eba4c4d10838024009680a006a6050440a8249250d0a13820042047a73cc90ad20c09a0328d306f302e94058a00c16810f000002927c96614345aec555a908087490d246708114ad0020c0092048a00d18c0002e840c188030131a00caa428144402794ad187c395a708805e205f0896808544695c41e945845104200018bf0165822e1c988507948801f61cb088654021a2444804231e08040b53026a49008d28830d6c829886428","gas_used":9629785,"gas_limit":9960460,"difficulty":2186082690667372,"seal":["0xa04ff49f5850283032ab01d18167399f0c52c44939d83369010031d3504e78e63e","0x88535bb7296e3adb4e"],"hash":"0x25f28ddc91c74017c7a8e4639105f00b72c5f97cf429fd1f6ac4218c7f3ee96d"}]
> [ info ]:     {"ApplyExtrinsic":2}: treasury.DepositRing[80000000]
> [ info ]:     {"ApplyExtrinsic":2}: balances.Deposit["5H92KwasoEsTor8QnmorTtVH4MCiwkzzSTXPUcowPzTAAGfE",20000000]
> [ info ]:     {"ApplyExtrinsic":2}: system.ExtrinsicSuccess[{"weight":200000000,"class":"Normal","paysFee":"Yes"}]
> [ ok ]: Extrinsic relay header 9965032 is in block!
> [ event ]: relay block 9965033 to darwinia...
> [ info ]: Transaction status: Ready
> [ info ]: Ready
> [ info ]: Transaction status: Broadcast
> [ info ]: {"Broadcast":["QmVVq8SXrMcUJhd5voZMz8Cru2feFRBUoAUCbdVdDgVBr3","QmStpRcmjTjBJ5onzefm2nV6HxWHYDSEYzvokCVcJ9CjbF","QmWfdcdFaiCr6gfJvh7jjmQ6hUGshA4BrGSLSDMuFBf823","QmQy26JaaH7dUez67WXpGubWfBvstymjBZvm5UarGrFtM1","QmNY3sejrd4vRLRrxahmLScJ5DFpt8r2k9iPyFEZGvmRBr","Qmb1CWK3eSnvr2fspR4UHdhTwez3gWpwZ9ajMzWg75T76V","QmfFWwDJ3KrjHrtma5HccA6VKkXpLsw4oN77rS3pvsnwTT","QmXsTfB7yjY6VbWzZ3MuYvvp4s1oWMYLgDnQEmdDnZF736","QmXUEEeYVNbVcK1qe2ghQuwdZ9g69QvwExTeWLvXzpzNTf","QmdNGh7Zf7dcA9SkSayNNTwhwDaSwbqN1WG79DKPvf978Q","QmXopHqYzSZWMC66FjpxDznR7WiVYp87v6tTGkxcBAa6sz","QmSH8A6gkUa35nLDyceGCicVLbRsFWPmDbus3CVuuQfCdm","QmSDkrTsBD1ZpNffhm1uyYMxkv4tdyqWc4P6evV1uKJDPe","QmTu1hSK7eX1TnpbiVMB4BpKjbeiDvq8TeTL8cXSCGveC8","QmUeJYFxDCCrC3xcNHUiYKaU1RmU7XEKGSWnSrYhA77kxM","QmPhewyq3sCJyq8RphYSec13GrzBzchrpo8GjKVP8vFvtG","QmbsnrKGk5G9eAYCyasM8sxs5ivnn7RRMmvrQHuREZXM7Y","QmWddQyNgXQKdEuqC4U1WZUtK99Zu38tuAAcyJAiV9jJzq","QmRznqmDf38MhSyGxmdYrDDhGffVtzWkKv3dMf9DZzBuib","QmdmtXDLPy96nzXeBD1K5QAuWpe8YDsR6iPmRBQ3BvDNWt","QmcFb9KmSDBaiKGniBGtKCmggKHY3NLsB6WmeFTr3Z73yP","QmS36CESFrRQQxHKgnk9w88bVzTWFVUqESswrKrfSgmVWn","QmXe9b6hnQCGibxepRMfBbstZRg1vx9XxFYoK6WzkqFeFW","Qme2VxbvZFxNbF3xKxWBQczRzbcudmny9n3eSube73ZmEw","QmdFvFBfCMFMH8a8Umo3W1LuK3qtcRchHnf3ws5Pe6TG9H","QmemgR3HkSGSYpXeMAshycQppTXNNHgUBi1ggSWVKJMBmd","QmRFQ1fLEjcBb7RAi4wuW8Cj9pYy4jdsWx8aM7d6Z7vq8M","QmfJN2UB64RosSxmtpQQw5cwVSscG4ftZqswfjQeNzduiT","QmVBxSyZbtT9gH1ptFCAX6PA9iNAr8AGQvNJvTwwyLUppq"]}
> [ info ]: Transaction status: Broadcast
> [ info ]: {"Broadcast":["Qme2VxbvZFxNbF3xKxWBQczRzbcudmny9n3eSube73ZmEw"]}
> [ info ]: Transaction status: InBlock
> [ info ]: {"InBlock":"0xa3426e67210926bc8a3b7ed1fb2590635efc97b754768a1b77cef0f7854b6b83"}
> [ info ]:     {"ApplyExtrinsic":2}: ethRelay.RelayHeader["5GBa6gfumh5rFiSJ7tvPUTYV7JRxYBj4n59g9ffGPqPh5ipo",{"parent_hash":"0x25f28ddc91c74017c7a8e4639105f00b72c5f97cf429fd1f6ac4218c7f3ee96d","timestamp":1588131280,"number":9965033,"author":"0x84a0d77c693adabe0ebc48f88b3ffff010577051","transactions_root":"0x575fea0ddcd2d20921754101f1e0936e9c38ca274fae6686ec39dda3f823c1e4","uncles_hash":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","extra_data":"0xd88301090b846765746888676f312e31332e38856c696e7578","state_root":"0xf966df69fbb6297a5074f87a1c2a689db0c036d631ad1b4f9a5714e10d4ef911","receipts_root":"0x28059b02ae4d7b8ae460828f68a75dd0f394b9f09f0b7d650913e4876988f8ac","log_bloom":"0x92de41a349f1a83b30103ec02e603e895ce318906d183060664bc762182b4d0311215575e5030d04e3294706568a07439280104709a64e97210ed18212cc808837342c408aa9dc14cbb8cf8f0086f4a73e4a2e28aaee204c0860678e0040f05b94a62d0482d0e01061adf78424018c6b44663668b8ee0e590a2535b2d010420a3c2598c7192cfc31ded502398889ec1e8c51dfb965a10ce4b0856834851e9a40fe4ce6940569b601be046881c422ecf2e33c605fb22e182b4c8632a058821d4160ba280e0c32309205067aa404d2a0642902c2ce560a913d9abb20431a956df2a710e4817988b20ceca41484885d314d4b49cf8a8251e300263952167d84386c","gas_used":9933104,"gas_limit":9950734,"difficulty":2187150113856302,"seal":["0xa067bb4e1fdf1b98acf0e6cad841d1e9c83f0ad2629ea6550a56554b6a25995f04","0x8844d12350020d6963"],"hash":"0x9dc9af76bd47537d72a94e33a9d56bcf6e90d704fc42ba4613bdc24027fa51a6"}]
> [ info ]:     {"ApplyExtrinsic":2}: treasury.DepositRing[80000000]
> [ info ]:     {"ApplyExtrinsic":2}: balances.Deposit["5GdsFsHXQLPCSSEWDZ5EFkwf1U5cLD15Evuz65My1ghPHKsx",20000000]
> [ info ]:     {"ApplyExtrinsic":2}: system.ExtrinsicSuccess[{"weight":200000000,"class":"Normal","paysFee":"Yes"}]
> [ ok ]: Extrinsic relay header 9965033 is in block!
> [ info ]: fetching proofs from 9965033 to 9965036...
> [ event ]: relay block 9965034 to darwinia...
> 2020-05-28 00:56:59        RPC-CORE: submitAndWatchExtrinsic(extrinsic: Extrinsic): ExtrinsicStatus:: 1010: Invalid Transaction: Stale
> 2020-05-28 00:56:59             DRR: Error: 1010: Invalid Transaction: Stale
>     at RpcCoder._checkError (/Users/denny/.config/yarn/global/node_modules/@polkadot/rpc-provider/coder/index.js:83:13)
>     at RpcCoder.decodeResponse (/Users/denny/.config/yarn/global/node_modules/@polkadot/rpc-provider/coder/index.js:46:10)
>     at WsProvider.value (/Users/denny/.config/yarn/global/node_modules/@polkadot/rpc-provider/ws/Provider.js:166:90)
>     at W3CWebSocket.value [as onmessage] (/Users/denny/.config/yarn/global/node_modules/@polkadot/rpc-provider/ws/Provider.js:146:153)
>     at W3CWebSocket._dispatchEvent [as dispatchEvent] (/Users/denny/.config/yarn/global/node_modules/yaeti/lib/EventTarget.js:107:17)
>     at W3CWebSocket.onMessage (/Users/denny/.config/yarn/global/node_modules/websocket/lib/W3CWebSocket.js:234:14)
>     at WebSocketConnection. (/Users/denny/.config/yarn/global/node_modules/websocket/lib/W3CWebSocket.js:205:19)
>     at WebSocketConnection.emit (events.js:315:20)
>     at WebSocketConnection.processFrame (/Users/denny/.config/yarn/global/node_modules/websocket/lib/WebSocketConnection.js:554:26)
>     at /Users/denny/.config/yarn/global/node_modules/websocket/lib/WebSocketConnection.js:323:40
> 2020-05-28 00:56:59        RPC-CORE: submitAndWatchExtrinsic(extrinsic: Extrinsic): ExtrinsicStatus:: 1010: Invalid Transaction: Stale
> (node:42259) UnhandledPromiseRejectionWarning: Error: 1010: Invalid Transaction: Stale
>     at RpcCoder._checkError (/Users/denny/.config/yarn/global/node_modules/@polkadot/rpc-provider/coder/index.js:83:13)
>     at RpcCoder.decodeResponse (/Users/denny/.config/yarn/global/node_modules/@polkadot/rpc-provider/coder/index.js:46:10)
>     at WsProvider.value (/Users/denny/.config/yarn/global/node_modules/@polkadot/rpc-provider/ws/Provider.js:166:90)
>     at W3CWebSocket.value [as onmessage] (/Users/denny/.config/yarn/global/node_modules/@polkadot/rpc-provider/ws/Provider.js:146:153)
>     at W3CWebSocket._dispatchEvent [as dispatchEvent] (/Users/denny/.config/yarn/global/node_modules/yaeti/lib/EventTarget.js:107:17)
>     at W3CWebSocket.onMessage (/Users/denny/.config/yarn/global/node_modules/websocket/lib/W3CWebSocket.js:234:14)
>     at WebSocketConnection. (/Users/denny/.config/yarn/global/node_modules/websocket/lib/W3CWebSocket.js:205:19)
>     at WebSocketConnection.emit (events.js:315:20)
>     at WebSocketConnection.processFrame (/Users/denny/.config/yarn/global/node_modules/websocket/lib/WebSocketConnection.js:554:26)
>     at /Users/denny/.config/yarn/global/node_modules/websocket/lib/WebSocketConnection.js:323:40
> (node:42259) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
> (node:42259) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
>

Well, I remember this, the tx API could not catch this error, a solution is touching a LOCK file, restart the relay service while it stuck at a block for too many times(maybe about 1 min

The solution above is not the perfect one, I'll try to catch this error again anyway tomorrow.

The patch should be committed into this pr, and be the break-API part might join this pr together, tomorrow

hackfisher commented 4 years ago

@clearloop please update the label to A0-pleasereview if it is ready for merge

clearloop commented 4 years ago

@clearloop please update the label to A0-pleasereview if it is ready for merge

It's ready for merging now.