feat: MMD-1309 / Convert from Babe to Aura. Single initial authority for consistent block time using Aura

Summary of changes

Uses Substrate commit 4d28ebeb8b027ca0227fe7779c5beb70a7b56467 for dependencies, since that's what the Substrate Node Template is up to.
Only using one (1) initial authority for all chains in the chain_spec.rs, since when the chain is run without the other initial authorities using Aura, and session keys are inserted, and then validator nodes are all reset so they finalise blocks, then if this script is run then there should not be any delay between some block headers (which is an issue we're always experiencing with Babe, but not experiencing with Aura as long as there are only 1 or 2 initial authorities in the chain_spec.rs), the duration between some block headers should all be consistent with the block time https://github.com/DataHighway-DHX/open-polkadot-js-library/tree/main/block-delay

Steps to setup validator nodes on brickable chain to test block time consistency:

rustup toolchain install nightly-2021-07-30 \
rustup target add wasm32-unknown-unknown --toolchain nightly-2021-07-30 \
rustup default nightly-2021-07-30 \
rustup override set nightly-2021-07-30

Remove old temporarily brickable chain data so starts from genesis

rm -rf /tmp/polkadot-chains

Run the five (5) initial authorities in separate terminal tabs on the local machine

./target/release/datahighway --validator \
  --unsafe-ws-external \
  --unsafe-rpc-external \
  --rpc-cors=all \
  --base-path /tmp/polkadot-chains/alice \
  --chain brickable \
  --node-key 88dc3417d5058ec4b4503e0c12ea1a0a89be200fe98922423d4334014fa6b0ee \
  --name "brickable Validator Alice" \
  --port 30333 \
  --ws-port 9944 \
  --rpc-port 9933 \
  --telemetry-url "wss://telemetry.polkadot.io/submit/ 0" \
  --execution=native \
  -lruntime=debug \
  --rpc-methods=Unsafe

./target/release/datahighway --validator \
  --unsafe-ws-external \
  --unsafe-rpc-external \
  --rpc-cors=all \
  --base-path /tmp/polkadot-chains/bob \
  --bootnodes /ip4/127.0.0.1/tcp/30333/p2p/12D3KooWKS7jU8ti7S5PDqCNWEj692eUSK3DLssHNwTQsto9ynVo \
  --chain brickable \
  --name "brickable Validator Bob" \
  --port 30334 \
  --ws-port 9945 \
  --rpc-port 9934 \
  --telemetry-url "wss://telemetry.polkadot.io/submit/ 0" \
  --execution=native \
  -lruntime=debug \
  --rpc-methods=Unsafe

./target/release/datahighway --validator \
  --unsafe-ws-external \
  --unsafe-rpc-external \
  --rpc-cors=all \
  --base-path /tmp/polkadot-chains/charlie \
  --bootnodes /ip4/127.0.0.1/tcp/30333/p2p/12D3KooWKS7jU8ti7S5PDqCNWEj692eUSK3DLssHNwTQsto9ynVo \
  --chain brickable \
  --name "brickable Validator Charlie" \
  --port 30335 \
  --ws-port 9946 \
  --rpc-port 9935 \
  --telemetry-url "wss://telemetry.polkadot.io/submit/ 0" \
  --execution=native \
  -lruntime=debug \
  --rpc-methods=Unsafe

./target/release/datahighway --validator \
  --unsafe-ws-external \
  --unsafe-rpc-external \
  --rpc-cors=all \
  --base-path /tmp/polkadot-chains/dave \
  --bootnodes /ip4/127.0.0.1/tcp/30333/p2p/12D3KooWKS7jU8ti7S5PDqCNWEj692eUSK3DLssHNwTQsto9ynVo \
  --chain brickable \
  --name "brickable Validator Dave" \
  --port 30336 \
  --ws-port 9947 \
  --rpc-port 9936 \
  --telemetry-url "wss://telemetry.polkadot.io/submit/ 0" \
  --execution=native \
  -lruntime=debug \
  --rpc-methods=Unsafe

./target/release/datahighway --validator \
  --unsafe-ws-external \
  --unsafe-rpc-external \
  --rpc-cors=all \
  --base-path /tmp/polkadot-chains/eve \
  --bootnodes /ip4/127.0.0.1/tcp/30333/p2p/12D3KooWKS7jU8ti7S5PDqCNWEj692eUSK3DLssHNwTQsto9ynVo \
  --chain brickable \
  --name "brickable Validator Eve" \
  --port 30337 \
  --ws-port 9948 \
  --rpc-port 9937 \
  --telemetry-url "wss://telemetry.polkadot.io/submit/ 0" \
  --execution=native \
  -lruntime=debug \
  --rpc-methods=Unsafe

Insert session keys for Grandpa and Aura to separate ports running the initial validators

curl -vH 'Content-Type: application/json' --data '{ "jsonrpc":"2.0", "method":"author_insertKey", "params":["gran", "//<SECRET>//1//grandpa", "0xedb0bfee980d12609f9641e1720ee4b2d4bee53e052c71e13580ee8f144a361c"],"id":1 }' 127.0.0.1:9933
curl -vH 'Content-Type: application/json' --data '{ "jsonrpc":"2.0", "method":"author_insertKey", "params":["aura", "//<SECRET>//1//aura", "0x1a0d59f23987acb275148a20f432bbbe81fdc950f607c2f4aa9218581590d17c"],"id":1 }' 127.0.0.1:9933

curl -vH 'Content-Type: application/json' --data '{ "jsonrpc":"2.0", "method":"author_insertKey", "params":["gran", "//<SECRET>//2//grandpa", "0x4bbcc0bb7e3f10a8ad097f00c8cd87ab647904e87c9103a46a3db20e74507bea"],"id":1 }' 127.0.0.1:9934
curl -vH 'Content-Type: application/json' --data '{ "jsonrpc":"2.0", "method":"author_insertKey", "params":["aura", "//<SECRET>//2//aura", "0xca44c0b08b439e43f439bfd8d95ea7030a1a2ff42ac1e7e54fa94080bebe0f3b"],"id":1 }' 127.0.0.1:9934

curl -vH 'Content-Type: application/json' --data '{ "jsonrpc":"2.0", "method":"author_insertKey", "params":["gran", "//<SECRET>//3//grandpa", "0xf8aaa52bc3a0b168fb1bdcfd3d4fe4220cfd893593371c4c2d21defd605ffa4e"],"id":1 }' 127.0.0.1:9935
curl -vH 'Content-Type: application/json' --data '{ "jsonrpc":"2.0", "method":"author_insertKey", "params":["aura", "//<SECRET>//3//aura", "0xde973ff7e8784a03a356cadcd7a3b6e49e0aab9d36db6c7365579dcb9debb97b"],"id":1 }' 127.0.0.1:9935

curl -vH 'Content-Type: application/json' --data '{ "jsonrpc":"2.0", "method":"author_insertKey", "params":["gran", "//<SECRET>//4//grandpa", "0x5526a6397e13b10a1d2b27114ad8322b8339cdbff9e47867cc11f500783d26ef"],"id":1 }' 127.0.0.1:9936
curl -vH 'Content-Type: application/json' --data '{ "jsonrpc":"2.0", "method":"author_insertKey", "params":["aura", "//<SECRET>//4//aura", "0xde072d2080256e333ee336527b886272170a3ab64180d22ab379fb7c513e114d"],"id":1 }' 127.0.0.1:9936

curl -vH 'Content-Type: application/json' --data '{ "jsonrpc":"2.0", "method":"author_insertKey", "params":["gran", "//<SECRET>//5//grandpa", "0xfdb309f030d63d422cdbaf079ade8fbc580aa7debf6c8a18b7db4c097ae1e854"],"id":1 }' 127.0.0.1:9937
curl -vH 'Content-Type: application/json' --data '{ "jsonrpc":"2.0", "method":"author_insertKey", "params":["aura", "//<SECRET>//5//aura", "0x98ddd9048757bf6995e430f6e58623c1b236471bdcb0d488242d3f8666acf645"],"id":1 }' 127.0.0.1:9937

Restart all of the initial validator nodes so the blocks start finalising

Run script https://github.com/DataHighway-DHX/open-polkadot-js-library/tree/main/block-delay to check if the block time for each subscribed block is the same, or if any are delayed. They should all be the same as the block time.

nvm use v16.3.0
yarn
node index.js "local"

Run tests

cargo +nightly-2021-07-30 test -p datahighway-runtime &&
cargo +nightly-2021-07-30 test -p roaming-operators &&
cargo +nightly-2021-07-30 test -p roaming-networks &&
cargo +nightly-2021-07-30 test -p roaming-organizations &&
cargo +nightly-2021-07-30 test -p roaming-network-servers &&
cargo +nightly-2021-07-30 test -p roaming-devices &&
cargo +nightly-2021-07-30 test -p roaming-routing-profiles &&
cargo +nightly-2021-07-30 test -p roaming-service-profiles &&
cargo +nightly-2021-07-30 test -p roaming-accounting-policies &&
cargo +nightly-2021-07-30 test -p roaming-agreement-policies &&
cargo +nightly-2021-07-30 test -p roaming-network-profiles &&
cargo +nightly-2021-07-30 test -p roaming-device-profiles &&
cargo +nightly-2021-07-30 test -p roaming-sessions &&
cargo +nightly-2021-07-30 test -p roaming-billing-policies &&
cargo +nightly-2021-07-30 test -p roaming-charging-policies &&
cargo +nightly-2021-07-30 test -p roaming-packet-bundles &&
cargo +nightly-2021-07-30 test -p mining-setting-token &&
cargo +nightly-2021-07-30 test -p mining-setting-hardware &&
cargo +nightly-2021-07-30 test -p mining-rates-token &&
cargo +nightly-2021-07-30 test -p mining-rates-hardware &&
cargo +nightly-2021-07-30 test -p mining-sampling-token &&
cargo +nightly-2021-07-30 test -p mining-sampling-hardware &&
cargo +nightly-2021-07-30 test -p mining-eligibility-token &&
cargo +nightly-2021-07-30 test -p mining-eligibility-hardware &&
cargo +nightly-2021-07-30 test -p mining-eligibility-proxy &&
cargo +nightly-2021-07-30 test -p mining-claims-token &&
cargo +nightly-2021-07-30 test -p mining-claims-hardware &&
cargo +nightly-2021-07-30 test -p mining-execution-token &&
cargo +nightly-2021-07-30 test -p exchange-rate

Configure runtime for faster debugging to check functionality still works

Since we want to elect the additional validator intentions, we want to the sessions and eras to go by faster, so we can change these runtime values:

pub const EPOCH_DURATION_IN_BLOCKS: BlockNumber = 1 * MINUTES;

pub const SessionsPerEra: sp_staking::SessionIndex = 1;
pub const BondingDuration: pallet_staking::EraIndex = 1;
pub const SlashDeferDuration: pallet_staking::EraIndex = 1;

For faster democracy proposals and referendums:

pub const LaunchPeriod: BlockNumber = 2 * MINUTES;
pub const VotingPeriod: BlockNumber = 2 * MINUTES;
pub const FastTrackVotingPeriod: BlockNumber = 2 * MINUTES;
pub const EnactmentPeriod: BlockNumber = 2 * MINUTES;
pub const CooloffPeriod: BlockNumber = 2 * MINUTES;

After making changes, build cargo build --release

Add additional nodes as validators

imported the Sudo account, and both stash and controller accounts for validator intentions 1-5 on brickable chain.
used Sudo > balances > setBalance to give all accounts above 10 BRI since balances aren't appearing
go to Extrinsics > staking > bond > v2 stash, 1 BRI to bond, v2 cont, and repeat for v2-5 (where v2 is the account of validator intention 2)
go to https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A9944#/rpc, choose "author > rotateKeys", then copy the hash (i.e. 0x...)
go back to https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A9944#/staking/actions, and it displays "Session Key" for v2, so click to open nodal to Set Session Key, then paste the hash from previous step
go to https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A9944#/staking/actions and click (+) Validator, choose v2 stash and v2 controller
go to https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A9944#/staking/actions and i was able to click the "Validate" button on its row, so then it's possible to click "Stop". only authority 1 was shown at the beginning.
increasing ideal no. of validators with Sudo > staking > setValidatorCount > 5
force new eras with Sudo > staking > forceNewEraAlways
initially in the Polkadot.js UI it will say you're on session 1, era 0, with validators 1 / 5, waiting 4
after waiting it changes and says you're up to session 2 and then session 3 (when changed to next Era, and when "waiting" changed to 0 at https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A9944#/staking/waiting), then all validators will be listed
it worked and all 5 were validating

Outstanding issues

[X] - balances are not appearing, even after i import Sudo account and setBalance of 10 BRI to all five authorities account addresses (both stash and controller accounts), it still only shows that 1 BRI is bonded by authority 1, and even after i add our custom_types.json at https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A9944#/settings/developer, there are no browser console errors. note: this also happens in branch 'luke/MMD-1309/update-substrate-3' it happens even after updating to latest Metadata in the Polkadot.js UI settings it must be a Polkadot.js API UI version issue. when i start with --dev and go to Polkadot.js accounts page, it displays error in UI console:

DRR: Unable to decode storage system.account: entry 0:: createType(AccountInfo):: {"nonce":"Index","consumers":"RefCount","providers":"RefCount","data":"AccountData"}:: Decoded input doesn't match input, received 0x010000000000000001000000000000000000e8890423c78a0000000000000000…0000000000000000000000000000000000000000000000000000000000000000 (80 bytes), created 0x010000000000000001000000000000000000e8890423c78a0000000000000000…0000000000000000000000000000000000000000000000000000000000000000

Update: This was fixed after we modified custom_types.json [X] - possibly an issue with democracy. using democracy to create a preimage (i.e. to increase ideal validator count to 6), and proposal, then seconding it, so it becomes a referendum. but when it's finished processing, the validator count doesn't change.

update: it's necessary for supermajority to have at least 50% of the total supply Aye or Nay (see https://github.com/polkadot-js/apps/blob/master/packages/page-democracy/src/Overview/Referendum.tsx#L66), so first Sudo > balances > setBalance > 29000000 for one of the accounts, then in democracy, create preimage using staking > setValidatorCount to 7, copy the preimage hash, then create proposal and past the preimage hash, then Second the proposal, then when it's a referenda, vote on it using 28000000 from that account, refresh the page to see updated tally of Ayes and Nays. ISSUE: even though voted with an account having at least 90% of total supply, it isn't storing the BRI tokens associated with the vote, so the referendum isn't passing Update: This was fixed after we modified custom_types.json [X] - after finish all necessary changes, generate chain spec and "raw" chain definition based on final changes in this PR for brickable, harbour, and westlake, and regenerated forkless upgrade wasm file and store in ./node/src/chain_built

DataHighway-DHX / node