search

Joystream / substrate-node-joystream

Joystream Full Node
https://www.joystream.org
GNU General Public License v3.0
15 stars 16 forks source link

Bounty $25: Queued transaction on synced nodes #74

Closed bwhm closed 4 years ago

bwhm commented 5 years ago

Description

It seems some nodes are "racking" up queued transactions, despite being synced.

From monitoring the telemetry, I have made the following observations.

  1. Of the 38 nodes reporting to the telemetry, these 10 have unconfirmed transactions in them: image In general, they have lower peer counts than other nodes (although joyful-blue-tiger is clearly an exception). Disregarding the ones with 0 peers:

    • Average peer count of nodes with tx in queue: 4.6
    • Average peer count of nodes w/o tx in queue: 9.3

  2. Taking #68 and #69 in to account, it is reasonable to assume these nodes have higher peer counts due to passing the --in-peers n --out-peers n where n>25. It is also reasonable to assume they are restarting their nodes more frequently, as has been recommended by us.

  3. The queued txes clear at restart.

  4. Even the bootnodes have had sustained queued txes. The network we are involuntary connecting to ChainX has the same issue, whereas the other networks are largely unaffected.

  5. Before this issue was noticed, I started the three nodes joyful-xxx in an attempt to improve the #68.

    • Brazil gaucho has not dropped off. (1p-34q)
    • South Africa springbok has been dropping off and restarted plenty of times. (9p-4q)
    • India blue-tiger has been reliable. (14-24).

  6. I think it's worth noticing how the highest number of queued txes correlate somewhat with the geographical distance to other nodes.

Note: The joyful-gaucho unfortunately shutdown during building of pioneer so I was not able to get the queued txes.

How to claim:

Node on computer with GUI

If you have a node running with queued transactions on a computer with a GUI, the easiest way of getting the queued tx is to:

  1. Open pioneer on a chromium based browser.
  2. Go to settings, and have it pointing to your own node, and fully featured mode.
  3. Explorer -> Node info scroll to bottom. copy/paste and screenshot.

Node on VPS/raspbian/armbian etc.

  1. Login, and install node and yarn. 
    $ git clone https://github.com/Joystream/apps.git
$ cd apps
$ yarn
$ yarn build
$ wget https://github.com/Joystream/substrate-node-joystream/files/3170217/queue.js.gz
$ gunzip queue.js.gz
$ node queue.js

    If output is [] there are no queued txes on your node...

queue.js.gz

perisky commented 5 years ago
authority QmcVs5LDjuT4RDD1P4uRFpPiogQn5tMRTaUyQq7jSMqMQo 392,069 0x504b2699c1d8aac006f908cd9051b0e0a657307f8824c62e0258f6fb0138762f
authority QmeDa8jASqMRpTh4YCkeVEuHo6nbMcFDzD9pkUxTr3WxhM 392,069 0x504b2699c1d8aac006f908cd9051b0e0a657307f8824c62e0258f6fb0138762f
full QmbjzmNMjzQUMHpzqcPHW5DnFeUjM3x4hbiDSMkYv1McD3 392,067 0x31f4bdc514374c30728607e151fffc8155d53cf18e9ef4ab4809659a66dd4f08
full QmSe2PWJUchtxHVsGnbGcQdKZJyb3F2UD2MdjPT14tUoCp 392,065 0xdac86c615d4d2b83f08643dacd9fef9f26808a493ec726c8804a84986509d3fe
full Qmc2jVdVpfseeSPYycmAgUWq65LFadMwi686KzmyPgX6aY 392,064 0xee51d248910b871287157862c034edf6282307ae7b0584494ed2613fe35fdba2
bwhm commented 5 years ago

Hey @perisky, Unfortunately in this case, your node (I assume it's perisky) does not have any queued txes.

wjdfx commented 5 years ago

node: bitcat-storge-provider

["0x250381ff0a2eb9cc01951facbe1549dc48827e78efdc78bc4d742d3b74326821d1be730472fb1d17d202d219bb52e996132c997aa993cb63d48440a12fe1e87c1ee2d01a11bc9306c5d57740c2022dfacb3b6a2003b7f4b3808957e53a61cc7aad15f50308000c0085012234344167645a62544d3136383178465834347632656d4c6a5575444745733531614a6173733538625656546b43683866363231585978505052357a62744854734a3451665445776f335a795857444a4452474e61475764524577435346716122"]

bwhm commented 5 years ago

Thanks for this @wjdfx I was not able to debug the issue, but you still qualify for the bounty. Please provide your xmr address, or joystream address. The latter requires some extra work (contact me on telegram).

wjdfx commented 5 years ago

Thanks for this @wjdfx I was not able to debug the issue, but you still qualify for the bounty. Please provide your xmr address, or joystream address. The latter requires some extra work (contact me on telegram).

44psk8KTt2yA7qBGS9ymTgJxoDbujDKXfadzXXNChZCLHw8fzkYFBy7YK3c9WGFkwQDc7mtNGntTSLbe6dcSG25e2TzBTcv

Thanks Martin.

finnyfound commented 5 years ago

Hi, i am getting the following error while running yarn build node version : v10.15.3 OS : ubuntu 18.04.02 yarn version : 1.16..0

yarn build
yarn run v1.16.0
$ NODE_ENV=production polkadot-dev-build-ts
$ /root/apps/apps/node_modules/.bin/polkadot-dev-clean-build

*** Cleaning build directory ./build

*** Cleaning build directory packages/app-123code/build

*** Cleaning build directory packages/app-accounts/build

*** Cleaning build directory packages/app-address-book/build

*** Cleaning build directory packages/app-contracts/build

*** Cleaning build directory packages/app-democracy/build

*** Cleaning build directory packages/app-explorer/build

*** Cleaning build directory packages/app-extrinsics/build

*** Cleaning build directory packages/app-js/build

*** Cleaning build directory packages/app-settings/build

*** Cleaning build directory packages/app-staking/build

*** Cleaning build directory packages/app-storage/build

*** Cleaning build directory packages/app-toolbox/build

*** Cleaning build directory packages/app-transfer/build

*** Cleaning build directory packages/apps/build

*** Cleaning build directory packages/joy-election/build

*** Cleaning build directory packages/joy-help/build

*** Cleaning build directory packages/joy-media/build

*** Cleaning build directory packages/joy-members/build

*** Cleaning build directory packages/joy-pages/build

*** Cleaning build directory packages/joy-proposals/build

*** Cleaning build directory packages/joy-roles/build

*** Cleaning build directory packages/joy-settings/build

*** Cleaning build directory packages/joy-types/build

*** Cleaning build directory packages/joy-utils/build

*** Cleaning build directory packages/ui-api/build

*** Cleaning build directory packages/ui-app/build

*** Cleaning build directory packages/ui-params/build

*** Cleaning build directory packages/ui-reactive/build

*** Cleaning build directory packages/ui-signer/build
Version 3.4.3
/root/apps/apps/node_modules/.bin/polkadot-dev-build-ts: line 57: 20247 Killed                  tsc --emitDeclarationOnly --outdir ../build
error Command failed with exit code 137.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
bwhm commented 5 years ago

@siman

mnaamani commented 5 years ago

Hi, i am getting the following error while running yarn build node version : v10.15.3 OS : ubuntu 18.04.02 yarn version : 1.16..0

yarn build
yarn run v1.16.0
$ NODE_ENV=production polkadot-dev-build-ts
$ /root/apps/apps/node_modules/.bin/polkadot-dev-clean-build

*** Cleaning build directory ./build

*** Cleaning build directory packages/app-123code/build

*** Cleaning build directory packages/app-accounts/build

*** Cleaning build directory packages/app-address-book/build

*** Cleaning build directory packages/app-contracts/build

*** Cleaning build directory packages/app-democracy/build

*** Cleaning build directory packages/app-explorer/build

*** Cleaning build directory packages/app-extrinsics/build

*** Cleaning build directory packages/app-js/build

*** Cleaning build directory packages/app-settings/build

*** Cleaning build directory packages/app-staking/build

*** Cleaning build directory packages/app-storage/build

*** Cleaning build directory packages/app-toolbox/build

*** Cleaning build directory packages/app-transfer/build

*** Cleaning build directory packages/apps/build

*** Cleaning build directory packages/joy-election/build

*** Cleaning build directory packages/joy-help/build

*** Cleaning build directory packages/joy-media/build

*** Cleaning build directory packages/joy-members/build

*** Cleaning build directory packages/joy-pages/build

*** Cleaning build directory packages/joy-proposals/build

*** Cleaning build directory packages/joy-roles/build

*** Cleaning build directory packages/joy-settings/build

*** Cleaning build directory packages/joy-types/build

*** Cleaning build directory packages/joy-utils/build

*** Cleaning build directory packages/ui-api/build

*** Cleaning build directory packages/ui-app/build

*** Cleaning build directory packages/ui-params/build

*** Cleaning build directory packages/ui-reactive/build

*** Cleaning build directory packages/ui-signer/build
Version 3.4.3
/root/apps/apps/node_modules/.bin/polkadot-dev-build-ts: line 57: 20247 Killed                  tsc --emitDeclarationOnly --outdir ../build
error Command failed with exit code 137.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Exit code 137 suggests the build exited because it ran out of memory. Does you machine have low memory perhaps?

finnyfound commented 5 years ago

I had 50%(~512mb) of ram left while doing it. I will restart the VPS and try again..

bwhm commented 5 years ago

Correct me if I'm wrong, but I think you said your node was killed simultaneously? That would be an indication @mnaamani is correct...

finnyfound commented 5 years ago

Node got killed when i restarted it and made validator again. not before..

mnaamani commented 4 years ago

This is outdated and no longer relevant to our current running testnet. Closing.