forbole / big-dipper

A block explorer for Cosmos
https://cosmos.bigdipper.live
Apache License 2.0
179 stars 243 forks source link

New created validator can't be shown properly #204

Closed pengqi-bc closed 4 years ago

pengqi-bc commented 5 years ago

Describe the bug I ran a cosmos blockchain network with two validators, then I created a new validator and the blockchain network works well. However, the explorer report an error and the new validator can't be shown on the explorer page. This is the explorer page: http://39.100.156.135:3000/, and this is the new created validator: http://39.100.156.135:3000/validator/0CA9FC583D32C0E90323E1D51307136D0D1C68C9

error msg:

{ BulkWriteError: E11000 duplicate key error collection: meteor.validators index: address_1 dup key: { : "0CA9FC583D32C0E90323E1D51307136D0D1C68C9" }
    at UnorderedBulkOperation.handleWriteError (/home/ubuntu/.meteor/packages/npm-mongo/.3.1.2.bp6gle.h3dfi++os+web.browser+web.browser.legacy+web.cordova/npm/node_modules/mongodb/lib/bulk/common.js:1056:11)
    at /home/ubuntu/.meteor/packages/npm-mongo/.3.1.2.bp6gle.h3dfi++os+web.browser+web.browser.legacy+web.cordova/npm/node_modules/mongodb/lib/bulk/unordered.js:201:27
    at handleCallback (/home/ubuntu/.meteor/packages/npm-mongo/.3.1.2.bp6gle.h3dfi++os+web.browser+web.browser.legacy+web.cordova/npm/node_modules/mongodb/lib/utils.js:128:55)
    at resultHandler (/home/ubuntu/.meteor/packages/npm-mongo/.3.1.2.bp6gle.h3dfi++os+web.browser+web.browser.legacy+web.cordova/npm/node_modules/mongodb/lib/bulk/unordered.js:161:5)
    at /home/ubuntu/.meteor/packages/npm-mongo/.3.1.2.bp6gle.h3dfi++os+web.browser+web.browser.legacy+web.cordova/npm/node_modules/mongodb-core/lib/connection/pool.js:532:18
    at _combinedTickCallback (internal/process/next_tick.js:132:7)
    at process._tickDomainCallback (internal/process/next_tick.js:219:9)
  index: 3,
  code: 11000,
  errmsg: 'E11000 duplicate key error collection: meteor.validators index: address_1 dup key: { : "0CA9FC583D32C0E90323E1D51307136D0D1C68C9" }',
  op:
   { q:
      { consensus_pubkey: 'barkisvalconspub1zcjduepqj7c7343xk5yny8zmgkyy54hpumnsjc5cx5gmzxm7gsrau0kxkttqwetuhe' },
     u: { '$set': [Object] },
     multi: false,
     upsert: true },
  name: 'BulkWriteError',
  driver: true,
  err:
   { index: 3,
     code: 11000,
     errmsg: 'E11000 duplicate key error collection: meteor.validators index: address_1 dup key: { : "0CA9FC583D32C0E90323E1D51307136D0D1C68C9" }',
     op: { q: [Object], u: [Object], multi: false, upsert: true } },
  result:
   BulkWriteResult {
     result:
      { ok: 1,
        writeErrors: [Array],
        writeConcernErrors: [],
        insertedIds: [],
        nInserted: 0,
        nUpserted: 1,
        nMatched: 4,
        nModified: 4,
        nRemoved: 0,
        upserted: [Array],
        lastOp: [Object] } },
  [Symbol(mongoErrorContextSymbol)]: {} }
Get height time: 0.006seconds.
kwunyeung commented 5 years ago

May I know which commit you are using? Have you ever run it be ore and there are data in the database? The error mentioned that the address is already in the database and can’t insert duplicate records.

pengqi-bc commented 5 years ago

git hash: 7c405810243f76773c9e49ff4c9d6cc3b536f7ce

0CA9FC583D32C0E90323E1D51307136D0D1C68C9 is the hex encoding address of validator consensus key. The validator is new created and the private key is new initilized. It should never be duplicated unless the random conflict in private key generation.

pengqi-bc commented 5 years ago

@kwunyeung Any further comment?

kwunyeung commented 5 years ago

Sorry I don't know why both github and slack doesn't poke me on this earlier.

I have verified that your consensus address and the hex value matches. Can you verify that you don't have this record in the database? Do you only have this validator not adding to the database?

The commit you are using is the one we use on gaia-13006 and kava-testnet-3000 now. At least it doesn't happen on those networks and I cannot replicate the issue.

nnkken commented 5 years ago

Same issue here.

Related operations

  1. Sent a CreateValidator transaction, with very little coins delegated (0.0001 coins, where the total of other validators is ~1,000,000 coins). Transaction page: http://34.66.207.129:3000/transactions/2A6D4DEF01DD554167D73E246B875A39C53E473A7D9A0C2DB826D44ADCB53BED
  2. Sent a CreateValidator transaction again with the same validator address and more coins, which failed since validator already exists. Transaction page: http://34.66.207.129:3000/transactions/A5D4B27C4ACA170322DFF7A5E84A6D79DCBE776DE945874B715D9AD56CADC8AE
  3. Sent a Delegate transaction to delegate more (~29,999.6 coins) onto the validator created at step 1. Transaction page: http://34.66.207.129:3000/transactions/904F6A4DE0BAD2524D3AD994AAD52E53C8473A367729BF6799F7DAD6B8B0D365

Result

While the validator is operating normally and proposing blocks after step 3, the validator list on BigDipper does not show the validator.

From the block list, there is often blocks which the proposer is 13B24E2D7B1CF4F31C34488655777C8B580AB77B, which I think is the related validator.

Clicking on 13B24E2D7B1CF4F31C34488655777C8B580AB77B shows the validator page with name 13B24E2D7B1CF4F31C34488655777C8B580AB77B, with correct uptime info but empty vaildator info and voting power. The power change section shows a 0 -> 29,999,600 (+29,999,600) entry for each single block in the chain. See: http://34.66.207.129:3000/validator/13B24E2D7B1CF4F31C34488655777C8B580AB77B

Clicking the pages related to the validator (e.g. the block page which proposer is the related validator, delegations page of the validator) shows We're sorry - something's gone wrong.. Example: http://34.66.207.129:3000/blocks/918248

Related error messages from BigDipper logs

{ BulkWriteError: E11000 duplicate key error collection: meteor.validators index: address_1 dup key: { : "13B24E2D7B1CF4F31C34488655777C8B580AB77B" }
     at UnorderedBulkOperation.handleWriteError (/home/chung/.meteor/packages/npm-mongo/.3.1.2.bp6gle.h3dfi++os+web.browser+web.browser.legacy+web.cordova/npm/node_modules/mongodb/lib/bulk/common.js:1056:11)
     at /home/chung/.meteor/packages/npm-mongo/.3.1.2.bp6gle.h3dfi++os+web.browser+web.browser.legacy+web.cordova/npm/node_modules/mongodb/lib/bulk/unordered.js:201:27
     at handleCallback (/home/chung/.meteor/packages/npm-mongo/.3.1.2.bp6gle.h3dfi++os+web.browser+web.browser.legacy+web.cordova/npm/node_modules/mongodb/lib/utils.js:128:55)
     at resultHandler (/home/chung/.meteor/packages/npm-mongo/.3.1.2.bp6gle.h3dfi++os+web.browser+web.browser.legacy+web.cordova/npm/node_modules/mongodb/lib/bulk/unordered.js:161:5)
     at /home/chung/.meteor/packages/npm-mongo/.3.1.2.bp6gle.h3dfi++os+web.browser+web.browser.legacy+web.cordova/npm/node_modules/mongodb-core/lib/connection/pool.js:532:18
     at _combinedTickCallback (internal/process/next_tick.js:132:7)
     at process._tickDomainCallback (internal/process/next_tick.js:219:9)
   index: 0,
   code: 11000,
   errmsg: 'E11000 duplicate key error collection: meteor.validators index: address_1 dup key: { : "13B24E2D7B1CF4F31C34488655777C8B580AB77B" }',
   op:
    { q:
       { consensus_pubkey: 'cosmosvalconspub1zcjduepqn4f7ukx9txqn6vdrsp5ah80lazcz4c5gttrqcdnpgf2edxecnqxshfsjvt' },
      u: { '$set': [Object] },
      multi: false,
      upsert: true },
   name: 'BulkWriteError',
   driver: true,
   err:
    { index: 0,
      code: 11000,
      errmsg: 'E11000 duplicate key error collection: meteor.validators index: address_1 dup key: { : "13B24E2D7B1CF4F31C34488655777C8B580AB77B" }',
      op: { q: [Object], u: [Object], multi: false, upsert: true } },
   result:
    BulkWriteResult {
      result:
       { ok: 1,
         writeErrors: [Array],
         writeConcernErrors: [],
         insertedIds: [],
         nInserted: 0,
         nUpserted: 0,
         nMatched: 7,
         nModified: 7,
         nRemoved: 0,
         upserted: [],
         lastOp: [Object] } },
   [Symbol(mongoErrorContextSymbol)]: {} }

The above log is copied from terminal output. I'm running in development mode, and cannot find where the log file is.

Meteor Database

I'm not familiar with Meteor, MongoDB or BigDipper, but still got some basic queries.

The validators collection in the database shows a strange entry:

{ "_id" : ObjectId("5dc1a80de1930689e2839178"), "address" : "13B24E2D7B1CF4F31C34488655777C8B580AB77B", "uptime" : 100, "lastSeen" : ISODate("2019-11-06T10:27:22.227Z") }

which missed many info compared to other entries:

{ "_id" : "wL6oeETqeTJnYT3Kr", "consensus_pubkey" : "cosmosvalconspub1zcjduepq48cw3ku2gpth38k8zqcr73rgrddgy953hwzyk06kj8jm53jg7zcs6d0m0y", "description" : { "moniker" : "likecoin", "identity" : "", "website" : "https://like.co", "details" : "LikeCoin Foundation's validator node, only for testnet" }, "commission" : { "commission_rates" : { "rate" : "0.500000000000000000", "max_rate" : "1.000000000000000000", "max_change_rate" : "0.010000000000000000" }, "update_time" : "2019-09-12T07:00:00Z" }, "min_self_delegation" : "1", "operator_address" : "cosmosvaloper17nuw43an3smnfsakcnz9j608t0pjvwxyps9cvu", "delegator_address" : "cosmos17nuw43an3smnfsakcnz9j608t0pjvwxyyy3dq0", "voting_power" : 30004000, "jailed" : false, "status" : 2, "pub_key" : { "type" : "tendermint/PubKeyEd25519", "value" : "qfDo24pAV3iexxAwP0RoG1qCFpG7hEs/VpHlukZI8LE=" }, "address" : "7A0ED73405451ED21EBE4EE098983D990FF414E0", "accpub" : "cosmospub1zcjduepq48cw3ku2gpth38k8zqcr73rgrddgy953hwzyk06kj8jm53jg7zcs8m2s4k", "operator_pubkey" : "cosmosvaloperpub1zcjduepq48cw3ku2gpth38k8zqcr73rgrddgy953hwzyk06kj8jm53jg7zcsh8gaqv", "lastSeen" : ISODate("2019-11-06T10:27:22.227Z"), "uptime" : 100, "delegator_shares" : "30004000000000.000000000000000000", "proposer_priority" : -195459408, "tokens" : "30004000000000", "unbonding_height" : "0", "unbonding_time" : "1970-01-01T00:00:00Z", "self_delegation" : 0.00003332888948140248 }

The first record of the related validator in the validator_records is at height 906675, which is right after step 3 (delegate more onto the validator).

Commit

The commit I'm using is base on 7c405810243f76773c9e49ff4c9d6cc3b536f7ce, with some custom changes on frontend styles.

nnkken commented 5 years ago

Made a branch for PoC of the bug: https://github.com/nnkken/big_dipper/tree/issue204/issue204

The script run.sh will do the followings:

Then if you are lucky / unlucky enough, you should be able to see a validator with hex string as name producing blocks in Big Dipper block list.

It seems that the bug depends on timing, so sometimes the script is not able to reproduce the bug. If you see validator-2 instead of some hex string, that means the bug is not reproduced.

On my machine it reproduces the bug with ~70% of chance.

Without the second failed CreateValidator transaction, I cannot reproduce the bug, so it could be related, but it's also possible that I'm just (un?)lucky.

kwunyeung commented 5 years ago

@nnkken Thank you! Will follow this up soon.

nnkken commented 4 years ago

It seems the bug is still present. Currently there are 13 validators on LikeCoin chain, but only 11 (10 active + 1 inactive) are shown in the block explorer: https://likecoin.bigdipper.live/validators/inactive

Below are the list of validators not shown: