search

bosagora / agora

POC Node implementation for CoinNet
https://bosagora.io
MIT License
37 stars 22 forks source link

`Fullnode` stops fetching blocks from `testnet` #3206

Closed hewison-chris closed 2 years ago

hewison-chris commented 2 years ago

After fetching about 70 blocks it stops fetching reporting many log warnings like:

2022-03-17 03:21:55,991 Warn [agora.network.Manager] - Could not find mapping in registry for key boa1xqp3sqa27jsygxpmt6ekpm393qjrf00whfadl40wauxrutg236yay3q48g0
2022-03-17 03:21:55,993 Warn [agora.network.Manager] - Could not find mapping in registry for key boa1xrjwgpgnpjuxzf452tja7ha3qhm83vz8hyq9kse8y5unewqtgkp2kptgvt9
2022-03-17 03:21:55,998 Warn [agora.network.Manager] - Could not find mapping in registry for key boa1xrmjg8ezuw0qmzu9gd0aarqd9d2ad43v42m7vxwzuaczg54m93v6qnvcxse
2022-03-17 03:22:01,16 Warn [agora.network.Manager] - Could not find mapping in registry for key boa1xrjwgpgnpjuxzf452tja7ha3qhm83vz8hyq9kse8y5unewqtgkp2kptgvt9
2022-03-17 03:22:01,16 Warn [agora.network.Manager] - Could not find mapping in registry for key boa1xqp3sqa27jsygxpmt6ekpm393qjrf00whfadl40wauxrutg236yay3q48g0
2022-03-17 03:22:01,25 Warn [agora.network.Manager] - Could not find mapping in registry for key boa1xrmjg8ezuw0qmzu9gd0aarqd9d2ad43v42m7vxwzuaczg54m93v6qnvcxse
2022-03-17 03:22:06,45 Warn [agora.network.Manager] - Could not find mapping in registry for key boa1xrjwgpgnpjuxzf452tja7ha3qhm83vz8hyq9kse8y5unewqtgkp2kptgvt9
2022-03-17 03:22:06,45 Warn [agora.network.Manager] - Could not find mapping in registry for key boa1xqp3sqa27jsygxpmt6ekpm393qjrf00whfadl40wauxrutg236yay3q48g0
2022-03-17 03:22:06,50 Warn [agora.network.Manager] - Could not find mapping in registry for key boa1xrmjg8ezuw0qmzu9gd0aarqd9d2ad43v42m7vxwzuaczg54m93v6qnvcxse
2022-03-17 03:22:11,54 Warn [agora.network.Manager] - Could not find mapping in registry for key boa1xqp3sqa27jsygxpmt6ekpm393qjrf00whfadl40wauxrutg236yay3q48g0
2022-03-17 03:22:11,58 Warn [agora.network.Manager] - Could not find mapping in registry for key boa1xrjwgpgnpjuxzf452tja7ha3qhm83vz8hyq9kse8y5unewqtgkp2kptgvt9
2022-03-17 03:22:11,61 Warn [agora.network.Manager] - Could not find mapping in registry for key boa1xrmjg8ezuw0qmzu9gd0aarqd9d2ad43v42m7vxwzuaczg54m93v6qnvcxse
2022-03-17 03:22:16,105 Warn [agora.network.Manager] - Could not find mapping in registry for key boa1xrmjg8ezuw0qmzu9gd0aarqd9d2ad43v42m7vxwzuaczg54m93v6qnvcxse
2022-03-17 03:22:16,106 Warn [agora.network.Manager] - Could not find mapping in registry for key boa1xqp3sqa27jsygxpmt6ekpm393qjrf00whfadl40wauxrutg236yay3q48g0
2022-03-17 03:22:16,106 Warn [agora.network.Manager] - Could not find mapping in registry for key boa1xrjwgpgnpjuxzf452tja7ha3qhm83vz8hyq9kse8y5unewqtgkp2kptgvt9
2022-03-17 03:22:21,60 Warn [agora.network.Manager] - Could not find mapping in registry for key boa1xqp3sqa27jsygxpmt6ekpm393qjrf00whfadl40wauxrutg236yay3q48g0
2022-03-17 03:22:21,60 Warn [agora.network.Manager] - Could not find mapping in registry for key boa1xrjwgpgnpjuxzf452tja7ha3qhm83vz8hyq9kse8y5unewqtgkp2kptgvt9
2022-03-17 03:22:21,65 Warn [agora.network.Manager] - Could not find mapping in registry for key boa1xrmjg8ezuw0qmzu9gd0aarqd9d2ad43v42m7vxwzuaczg54m93v6qnvcxse
2022-03-17 03:22:26,628 Warn [agora.network.Manager] - Could not find mapping in registry for key boa1xrmjg8ezuw0qmzu9gd0aarqd9d2ad43v42m7vxwzuaczg54m93v6qnvcxse
2022-03-17 03:22:26,628 Warn [agora.network.Manager] - Could not find mapping in registry for key boa1xqp3sqa27jsygxpmt6ekpm393qjrf00whfadl40wauxrutg236yay3q48g0
2022-03-17 03:22:26,628 Warn [agora.network.Manager] - Could not find mapping in registry for key boa1xrjwgpgnpjuxzf452tja7ha3qhm83vz8hyq9kse8y5unewqtgkp2kptgvt9
2022-03-17 03:22:31,92 Warn [agora.network.Manager] - Could not find mapping in registry for key boa1xqp3sqa27jsygxpmt6ekpm393qjrf00whfadl40wauxrutg236yay3q48g0
Geod24 commented 2 years ago

Sounds a lot like https://github.com/bosagora/agora/issues/3162 ?

hewison-chris commented 2 years ago

local_fullnode_joining_testnet.log

Sounds a lot like #3162 ?

Looks like different errors and is for fullnode not validator.

hewison-chris commented 2 years ago

This is reproducible when running on a Mac in the office network.

hewison-chris commented 2 years ago 
sqlite> select * from registry_validator_addresses ;
boa1xzval2a3cdxv28n6slr62wlczslk3juvk7cu05qt3z55ty2rlfqfc6egsh2|eu-002.bosagora.io.|5|600|0
boa1xzval2a3cdxv28n6slr62wlczslk3juvk7cu05qt3z55ty2rlfqfc6egsh2|agora://eu-002.bosagora.io/|256|600|0
boa1xrval5rzmma29zh4aqgv3mvcarhwa0w8rgthy3l9vaj3fywf9894ycmjkm8|na-001.bosagora.io.|5|600|0
boa1xrval5rzmma29zh4aqgv3mvcarhwa0w8rgthy3l9vaj3fywf9894ycmjkm8|agora://na-001.bosagora.io:3826/|256|600|0
boa1xzval3ah8z7ewhuzx6mywveyr79f24w49rdypwgurhjkr8z2ke2mycftv9n|eu-002.bosagora.io.|5|600|0
boa1xzval3ah8z7ewhuzx6mywveyr79f24w49rdypwgurhjkr8z2ke2mycftv9n|agora://eu-002.bosagora.io:3826/|256|600|0
boa1xrval6hd8szdektyz69fnqjwqfejhu4rvrpwlahh9rhaazzpvs5g6lh34l5|na-002.bosagora.io.|5|600|0
boa1xrval6hd8szdektyz69fnqjwqfejhu4rvrpwlahh9rhaazzpvs5g6lh34l5|agora://na-002.bosagora.io/|256|600|0
boa1xzval4nvru2ej9m0rptq7hatukkavemryvct4f8smyy3ky9ct5u0s8w6gfy|na-001.bosagora.io.|5|600|0
boa1xzval4nvru2ej9m0rptq7hatukkavemryvct4f8smyy3ky9ct5u0s8w6gfy|agora://na-001.bosagora.io/|256|600|0
boa1xrval7gwhjz4k9raqukcnv2n4rl4fxt74m2y9eay6l5mqdf4gntnzhhscrh|na-002.bosagora.io.|5|600|0
boa1xrval7gwhjz4k9raqukcnv2n4rl4fxt74m2y9eay6l5mqdf4gntnzhhscrh|agora://na-002.bosagora.io:3826/|256|600|0
boa1xz9fcvee9sh2xj7n7dcmmngsntg8cmqpuq2pyjcvt8wsq823pa5wqtph3ul|eu-003.bosagora.io.|5|600|0
boa1xz9fcvee9sh2xj7n7dcmmngsntg8cmqpuq2pyjcvt8wsq823pa5wqtph3ul|agora://eu-003.bosagora.io:5826/|256|600|0
boa1xpvtrlw4vlkrm4wlwhhdenk8yrah4hw7gtadd0w4fq8re2sj6vczq20dhw9|eu-005.bosagora.io.|5|600|0
boa1xpvtrlw4vlkrm4wlwhhdenk8yrah4hw7gtadd0w4fq8re2sj6vczq20dhw9|agora://eu-005.bosagora.io/|256|600|0
boa1xpval9gv8wjk5s05w0vplpgd5wrmzlvhj4e6zym302f2t6xeklzw2meepv9|eu-003.bosagora.io.|5|600|0
boa1xpval9gv8wjk5s05w0vplpgd5wrmzlvhj4e6zym302f2t6xeklzw2meepv9|agora://eu-003.bosagora.io:3826/|256|600|0
boa1xzcttukq96qxtlwa4xra6fn0lr580t07k27ama0pkrm8hqsn3n2vsaexzmd|eu-005.bosagora.io.|5|600|0
boa1xzcttukq96qxtlwa4xra6fn0lr580t07k27ama0pkrm8hqsn3n2vsaexzmd|agora://eu-005.bosagora.io:4826/|256|600|0
boa1xrcxupnkwpqmkukk3zge7hl0f3ye2258u2yppx2qxwqhelwrcqvhx8uvy7m|na-004.bosagora.io.|5|600|0
boa1xrcxupnkwpqmkukk3zge7hl0f3ye2258u2yppx2qxwqhelwrcqvhx8uvy7m|agora://na-004.bosagora.io:5826/|256|600|0
boa1xqyewv34jhp0e8w9a6u7knnf028qa9h0k432ny90lv58gfy0wgn5ymerru0|na-001.bosagora.io.|5|600|0
boa1xqyewv34jhp0e8w9a6u7knnf028qa9h0k432ny90lv58gfy0wgn5ymerru0|agora://na-001.bosagora.io:4826/|256|600|0
boa1xr3l5ktrp04ql3ddxh5hwc44yuh99safurcqu7clw37kmmea3gxguvkpn8g|eu-005.bosagora.io.|5|600|0
boa1xr3l5ktrp04ql3ddxh5hwc44yuh99safurcqu7clw37kmmea3gxguvkpn8g|agora://eu-005.bosagora.io:5826/|256|600|0
boa1xpkry8xst7zazssfqgxxaq7wjw20eq746h9thep8jwmtexfwl796z9fzpsd|na-003.bosagora.io.|5|600|0
boa1xpkry8xst7zazssfqgxxaq7wjw20eq746h9thep8jwmtexfwl796z9fzpsd|agora://na-003.bosagora.io:3826/|256|600|0
boa1xzn6vy0fuqnsfsukelt7pmtpp5qy32m57ju0jvgt3x045wqfzx9jkeq37hr|na-003.bosagora.io.|5|600|0
boa1xzn6vy0fuqnsfsukelt7pmtpp5qy32m57ju0jvgt3x045wqfzx9jkeq37hr|agora://na-003.bosagora.io/|256|600|0
boa1xpeyczthhgkvsuett7vueh8txqywtj5ckat8ce7rnxwq5qypmey7sqgdrll|na-004.bosagora.io.|5|600|0
boa1xpeyczthhgkvsuett7vueh8txqywtj5ckat8ce7rnxwq5qypmey7sqgdrll|agora://na-004.bosagora.io:3826/|256|600|0
boa1xrd0458p7mshrc3cvj7k6g7qy3cfllgv4vwkn3g2mp0nvlwr7vnr2u04kas|na-003.bosagora.io.|5|600|0
boa1xrd0458p7mshrc3cvj7k6g7qy3cfllgv4vwkn3g2mp0nvlwr7vnr2u04kas|agora://na-003.bosagora.io:4826/|256|600|0
boa1xr6cgz4ule6ssgl2s7xggfkfj7xpuex8930rekhjjtmvqp5d77kncd5ec20|eu-003.bosagora.io.|5|600|0
boa1xr6cgz4ule6ssgl2s7xggfkfj7xpuex8930rekhjjtmvqp5d77kncd5ec20|agora://eu-003.bosagora.io:4826/|256|600|0
boa1xpy9fl6yenjhha2zr7yqfmpmypggylz9dwdgngjyec6ckz9mdvncgtw3qn8|na-004.bosagora.io.|5|600|0
boa1xpy9fl6yenjhha2zr7yqfmpmypggylz9dwdgngjyec6ckz9mdvncgtw3qn8|agora://na-004.bosagora.io/|256|600|0
boa1xz3qed5979y0v0mjjsndh5lcjmyvp9hrcsu78qs4mjw8gugqnkxx25e3qfd|eu-002.bosagora.io.|5|600|0
boa1xz3qed5979y0v0mjjsndh5lcjmyvp9hrcsu78qs4mjw8gugqnkxx25e3qfd|agora://eu-002.bosagora.io:5826/|256|600|0
boa1xqvalc7v34kr9crh4e882zmguvt3dgmtdhxtqx0wsljej5f9xdxl6xftcay|eu-004.bosagora.io.|5|600|0
boa1xqvalc7v34kr9crh4e882zmguvt3dgmtdhxtqx0wsljej5f9xdxl6xftcay|agora://eu-004.bosagora.io/|256|600|0
boa1xr8rjwcrlksw90nml0z3u25vkhluw3dhamz97v6auszfs2ltme9lz2vulqg|na-004.bosagora.io.|5|600|0
boa1xr8rjwcrlksw90nml0z3u25vkhluw3dhamz97v6auszfs2ltme9lz2vulqg|agora://na-004.bosagora.io:4826/|256|600|0
boa1xrvaldd5au5d5xs6pd6js7zah6m4h5d0r5tpwjasp99gvz3gmj2ex432u5x|eu-004.bosagora.io.|5|600|0
boa1xrvaldd5au5d5xs6pd6js7zah6m4h5d0r5tpwjasp99gvz3gmj2ex432u5x|agora://eu-004.bosagora.io:3826/|256|600|0
boa1xza59a84he9wm8fyk743dhu5hvs6q0ryjeng40fajgvucku8jeazvstug6q|eu-004.bosagora.io.|5|600|0
boa1xza59a84he9wm8fyk743dhu5hvs6q0ryjeng40fajgvucku8jeazvstug6q|agora://eu-004.bosagora.io:4826/|256|600|0
boa1xqquu5pxsm9zmgfc5yu9z363pptpju2phk3vzdfnptzc5f8vmf6jyjxdgkf|eu-005.bosagora.io.|5|600|0
boa1xqquu5pxsm9zmgfc5yu9z363pptpju2phk3vzdfnptzc5f8vmf6jyjxdgkf|agora://eu-005.bosagora.io:3826/|256|600|0
boa1xr52fh2zlfzjfnv9zuxfcy3uktjzm0n8zmulrup6vp2qed84pee5x6rerh4|eu-004.bosagora.io.|5|600|0
boa1xr52fh2zlfzjfnv9zuxfcy3uktjzm0n8zmulrup6vp2qed84pee5x6rerh4|agora://eu-004.bosagora.io:5826/|256|600|0
boa1xpz3k5f7907jgc8q67m62kl0qs5rdktl29mqutk5a7kgmw0sr40pzrrpw9q|na-001.bosagora.io.|5|600|0
boa1xpz3k5f7907jgc8q67m62kl0qs5rdktl29mqutk5a7kgmw0sr40pzrrpw9q|agora://na-001.bosagora.io:5826/|256|600|0
boa1xzvale54hw9zk69t7hpgu422ht2nkv3gkx7k8nhph5vg2tkpwpnzuarah4d|eu-002.bosagora.io.|5|600|0
boa1xzvale54hw9zk69t7hpgu422ht2nkv3gkx7k8nhph5vg2tkpwpnzuarah4d|agora://eu-002.bosagora.io:4826/|256|600|0
boa1xzval8mq887lkjsqwyl58xyrkxxz8mphv5dx9qv2z750fxvcs9gtvpal0dm|eu-003.bosagora.io.|5|600|0
boa1xzval8mq887lkjsqwyl58xyrkxxz8mphv5dx9qv2z750fxvcs9gtvpal0dm|agora://eu-003.bosagora.io/|256|600|0
boa1xqj28ajhdq0cc6hvx0pynltstgvxqnawz75249gk4sahwt5yn9tc63783q0|na-002.bosagora.io.|5|600|0
boa1xqj28ajhdq0cc6hvx0pynltstgvxqnawz75249gk4sahwt5yn9tc63783q0|agora://na-002.bosagora.io:4826/|256|600|0
linked0 commented 2 years ago

Here are full logs on my validator built with the latest commit 29cc3c8, which shows the same logs as @hewison-chris posted. One more issue besides this issue is that the validator candidate with a frozen UTXO does not enroll. I will raise this issue after checking one more time.

linked0 commented 2 years ago

^ This full logs also contains the logs that are shown when issue 3168 happens, which is about a validator never enrolling.

My config is as follows.

node:
  realm: "testnet.bosagora.io"
  testing: true
  registry_address: https://ns1.bosagora.io/

registry:
  public: true
  port: 8053

logging:
  root:
    # You might want to use `Trace` and separate output
    level: Info
    console: true
    propagate: true
    file: log/root.log

validator:
  enabled: true
  seed: SDMBCPHMZFBJC46JSYCN2POPBVFGR4NXQNP5B5YN6AU376S7TILQGROY
  addresses_to_register:
    - "agora://3.39.57.68:2826"
hewison-chris commented 2 years ago

I tested in AWS with an instance of t2-medium and Ubuntu 20.04 (ami-0454bb2fefc7de534) using command docker run -v $(pwd):/agora/ bosagora/agora --testnet Agora v.0.37.1

The FullNode is keeping up to date with the TestNet for over 4 days now.