Unable to finalize blocks or change session keys

[ltfschoen]

Issue definition

After running both an Alice node and a Bob node, it does not finalize blocks. i.e.

Idle (0 peers), best: #112 (0x4d42…cbd1), finalized #0 (0x3947…2716), ⬇ 0 ⬆ 0

I've tried changing the session keys as follows using insertKey for each key type (i.e. aura, babe, imonline, grandpa), but the output from each cURL request is {"jsonrpc":"2.0","result":null,"id":1}, which means it did not work.

I've been advised by @xlc that when changing validators/session keys the change only comes in effect after the block that is including the change has been finalized, so only change one at a time. Also to wait for two eras (optionally use sudo staking force era to speed it up) and make sure everything is still finalizing before making the second change. After you have everything running smoothly, you can use rotateKeys instead of insertKey.

Reproduce the issue

• Build the chain spec and raw chain definition used to run the node based on the latest code

./target/release/datahighway build-spec \
  --chain=testnet-latest > ./src/chain-spec-templates/chain_spec_testnet_latest.json

./target/release/datahighway build-spec \
  --chain ./src/chain-spec-templates/chain_spec_testnet_latest.json \
  --raw > ./src/chain-definition-custom/chain_def_testnet_v0.1.0.json

• Install latest Substrate and Subkey binary.

  curl https://getsubstrate.io -sSf | bash && \
  ./scripts/init.sh

• Remove previous chains

  ./target/release/datahighway purge-chain --dev --base-path /tmp/polkadot-chains/alice
  ./target/release/datahighway purge-chain --dev --base-path /tmp/polkadot-chains/bob

• Run a node for both Alice (bootnode) and Bob as follows:

SKIP_WASM_BUILD= ./target/release/datahighway --validator \
  --unsafe-ws-external \
  --unsafe-rpc-external \
  --rpc-cors=all \
  --base-path /tmp/polkadot-chains/alice \
  --keystore-path "/tmp/polkadot-chains/alice/keys" \
  --chain ./src/chain-definition-custom/chain_def_testnet_v0.1.0.json \
  --node-key 88dc3417d5058ec4b4503e0c12ea1a0a89be200fe98922423d4334014fa6b0ee \
  --alice \
  --rpc-port 9933 \
  --port 30333 \
  --telemetry-url ws://telemetry.polkadot.io:1024 \
  --ws-port 9944 \
  --execution=native \
  -lruntime=debug

SKIP_WASM_BUILD= ./target/release/datahighway --validator \
  --unsafe-ws-external \
  --unsafe-rpc-external \
  --rpc-cors=all \
  --base-path /tmp/polkadot-chains/bob \
  --keystore-path "/tmp/polkadot-chains/bob/keys" \
  --bootnodes /ip4/127.0.0.1/tcp/30333/p2p/QmWYmZrHFPkgX8PgMgUpHJsK6Q6vWbeVXrKhciunJdRvKZ \
  --chain ./src/chain-definition-custom/chain_def_testnet_v0.1.0.json \
  --bob \
  --rpc-port 9933 \
  --port 30334 \
  --telemetry-url ws://telemetry.polkadot.io:1024 \
  --ws-port 9944 \
  --execution=native \
  -lruntime=debug

• Generate session keys for Alice

  
  $ subkey --ed25519 inspect "//Alice"
  Secret Key URI `//Alice` is account:
  Secret seed:      0xabf8e5bdbe30c65656c0a3cbd181ff8a56294a69dfedd27982aace4a76909115
  Public key (hex): 0x88dc3417d5058ec4b4503e0c12ea1a0a89be200fe98922423d4334014fa6b0ee
  Account ID:       0x88dc3417d5058ec4b4503e0c12ea1a0a89be200fe98922423d4334014fa6b0ee
  SS58 Address:     5FA9nQDVg267DEd8m1ZypXLBnvN7SFxYwV7ndqSYGiN9TTpu

$ subkey --sr25519 inspect "//Alice"//aura Secret Key URI //Alice//aura is account: Secret seed: 0x153d8db5f7ef35f18a456c049d6f6e2c723d6c18d1f9f6c9fbee880c2a171c73 Public key (hex): 0x408f99b525d90cce76288245cb975771282c2cefa89d693b9da2cdbed6cd9152 Account ID: 0x408f99b525d90cce76288245cb975771282c2cefa89d693b9da2cdbed6cd9152 SS58 Address: 5DXMabRsSpaMwfNivWjWEnzYtiHsKwQnP4aAKB85429ZQU6v

$ subkey --sr25519 inspect "//Alice"//babe Secret Key URI //Alice//babe is account: Secret seed: 0x7bc0e13f128f3f3274e407de23057efe043c2e12d8ed72dc5c627975755c9620 Public key (hex): 0x46ffa3a808850b2ad55732e958e781146ed1e6436ffb83290e0cb810aacf5070 Account ID: 0x46ffa3a808850b2ad55732e958e781146ed1e6436ffb83290e0cb810aacf5070 SS58 Address: 5Dfo9eF9C7Lu5Vbc8LbaMXi1Us2yi5VGTTA7radKoxb7M9HT

$ subkey --sr25519 inspect "//Alice"//imonline Secret Key URI //Alice//imonline is account: Secret seed: 0xf54dc00d41d0ea7929ac00a08ed1e111eb8c35d669b011c649cea23997f5d8d9 Public key (hex): 0xee725cf87fa2d6f264f26d7d8b84b1054d2182cdcce51fdea95ec868be9d1e17 Account ID: 0xee725cf87fa2d6f264f26d7d8b84b1054d2182cdcce51fdea95ec868be9d1e17 SS58 Address: 5HTME6o2DqEuoNCxE5263j2dNzFGxspeP8wswenPA3WerfmA

$ subkey --ed25519 inspect "//Alice"//grandpa Secret Key URI //Alice//grandpa is account: Secret seed: 0x03bee0237d4847732404fde7539e356da44bce9cd69f26f869883419371a78ab Public key (hex): 0x6e2de2e5087b56ed2370359574f479d7e5da1973e17ca1b55882c4773f154d2f Account ID: 0x6e2de2e5087b56ed2370359574f479d7e5da1973e17ca1b55882c4773f154d2f SS58 Address: 5EZAkmxARDqRz5z5ojuTjacTs2rTd7WRL1A9ZeLvwgq2STA2

* Run cURL for to insert session key for each key type (i.e. "aura"), by providing the associated secret key, and associated Public key (hex) . Note that I've also tried using the Secret Seed (i.e. `0x153d8db5f7ef35f18a456c049d6f6e2c723d6c18d1f9f6c9fbee880c2a171c73`) instead of the Secret Key URI (i.e. `//Alice//aura`), but that does not work either

curl -vH 'Content-Type: application/json' --data '{ "jsonrpc":"2.0", "method":"author_insertKey", "params":["aura", "0x153d8db5f7ef35f18a456c049d6f6e2c723d6c18d1f9f6c9fbee880c2a171c73", "0x408f99b525d90cce76288245cb975771282c2cefa89d693b9da2cdbed6cd9152"],"id":1 }' localhost:9933 curl -vH 'Content-Type: application/json' --data '{ "jsonrpc":"2.0", "method":"author_insertKey", "params":["babe", "//Alice//babe", "0x46ffa3a808850b2ad55732e958e781146ed1e6436ffb83290e0cb810aacf5070"],"id":1 }' localhost:9933 curl -vH 'Content-Type: application/json' --data '{ "jsonrpc":"2.0", "method":"author_insertKey", "params":["imon", "//Alice//imonline", "0xee725cf87fa2d6f264f26d7d8b84b1054d2182cdcce51fdea95ec868be9d1e17"],"id":1 }' localhost:9933 curl -vH 'Content-Type: application/json' --data '{ "jsonrpc":"2.0", "method":"author_insertKey", "params":["gran", "//Alice//grandpa", "0x6e2de2e5087b56ed2370359574f479d7e5da1973e17ca1b55882c4773f154d2f"],"id":1 }' localhost:9933

* Problems encountered
  * Output from each cURL request is `{"jsonrpc":"2.0","result":null,"id":1}`, when it should be: `{"jsonrpc":"2.0","result":"0x...","id":1}`
    * Example output:

$ curl -vH 'Content-Type: application/json' --data '{ "jsonrpc":"2.0", "method":"author_insertKey", "params":["aura", "0x153d8db5f7ef35f18a456c049d6f6e2c723d6c18d1f9f6c9fbee880c2a171c73", "0x408f99b525d90cce76288245cb975771282c2cefa89d693b9da2cdbed6cd9152"],"id":1 }' localhost:9933

• Rebuilt URL to: localhost:9933/
• Trying ::1...
• TCP_NODELAY set
• Connection failed
• connect to ::1 port 9933 failed: Connection refused
• Trying 127.0.0.1...
• TCP_NODELAY set
• Connected to localhost (127.0.0.1) port 9933 (#0)

  POST / HTTP/1.1 Host: localhost:9933 User-Agent: curl/7.54.0 Accept: / Content-Type: application/json Content-Length: 214

• upload completely sent off: 214 out of 214 bytes < HTTP/1.1 200 OK < content-type: application/json; charset=utf-8 < content-length: 39 < date: Tue, 17 Mar 2020 03:42:35 GMT < {"jsonrpc":"2.0","result":null,"id":1}
• Connection #0 to host localhost left intact

  
  * Empty folder after running cURL: /tmp/polkadot-chains/alice/keys

[ltfschoen]

Since in GRANDPA you have authority set of size 4, it means you need 3 nodes running in order to finalize the blocks that are authored (2 nodes is insufficient to finalize the blocks).

• The output from each cURL request should be {"jsonrpc":"2.0","result":null,"id":1}, since with a successful output null is returned https://github.com/paritytech/substrate/blob/db1ab7d18fbe7876cdea43bbf30f147ddd263f94/client/rpc-api/src/author/mod.rs#L47. Also check that the following folder is not empty /tmp/polkadot-chains/alice/keys (it should now contain four keys).

(Credit: @bkchr Bastian Köcher).