Assethub not producing blocks locally.

[hbulgarini]

The paseo core team is looking to deploy the asset hub, but first we want to do a cycle of local testing.

Branch: https://github.com/paseo-network/runtimes/tree/hb-paseo-assethub-testing

Even having everything set and configured correctly in the assethub chain spec, the parachain does not produce blocks locally when it should.

Below are the steps to reproduce the error:

Build chain-spec-generator

cd chain-spec-generator ; cargo build --release --features=fast-runtime

Create raw chains specs:

./target/release/chain-spec-generator paseo-local --raw > ../chain-specs/paseo.local.raw.json

./target/release/chain-spec-generator asset-hub-paseo-local --raw   > ../chain-specs/asset-hub-local.raw.json

Start relay chain with zombienet:

 ./zombienet-macos spawn --provider native -d /tmp/relay paseo-local.toml

paseo-local.toml :

 [settings]
timeout = 1000

[relaychain]
chain_spec_path = "./paseo.local.raw.json"
default_command = "./polkadot"

  [[relaychain.nodes]]
  name = "alice"
  validator = true
  ws_port = 10000
  extra_args = ["--force-authoring -lparachain=debug --unsafe-rpc-external --rpc-cors=all"]

  [[relaychain.nodes]]
  name = "bob"
  validator = true
  extra_args = ["--force-authoring -lparachain=debug --unsafe-rpc-external --rpc-cors=all"]

  [[relaychain.nodes]]
  name = "charlie"
  validator = true
  extra_args = ["--force-authoring -lparachain=debug --unsafe-rpc-external --rpc-cors=all"]

  [[relaychain.nodes]]
  name = "dave"
  validator = true
  extra_args = ["--force-authoring -lparachain=debug --unsafe-rpc-external --rpc-cors=all"]

[types.Header]
number = "u64"
parent_hash = "Hash"
post_state = "Hash"

Once the relay chain started, start both collators:

./polkadot-parachain \
--collator \
--force-authoring \
--chain asset-hub-local.raw.json \
--base-path /tmp/parachain/alice \
--port 40333 \
--rpc-port 8844 \
--discover-local \
-- \
--execution wasm \
--chain paseo.local.raw.json \
--port 30343 \
--rpc-port 9977 

./polkadot-parachain \
--collator \
--force-authoring \
--chain asset-hub-local.raw.json \
--base-path /tmp/parachain/bob \
--port 40334 \
--rpc-port 8845 \
--discover-local \
-- \
--execution wasm \
--chain paseo.local.raw.json \
--port 30344 \
--rpc-port 9978 

Inject the following keys:

./polkadot-parachain key insert --base-path /tmp/parachain/alice \
  --chain asset-hub-local.raw.json \
  --scheme Ed25519 \
  --suri 0xd3a656b0d6ccacd7428da7fda811e6e1c89cc26035e649cbf680f7a6de8aa25e \
  --password-interactive \
  --key-type aura

./polkadot-parachain key insert --base-path /tmp/parachain/bob \
  --chain asset-hub-local.raw.json \
  --scheme Ed25519 \
  --suri 0x120870f7aeb7ace021ac3539be456dd09ba4fb456b6f3afd2c02f79c132410c9 \
  --password-interactive \
  --key-type aura

These are the keys used for both aura keys:

   bin ./polkadot-parachain key generate --scheme ed25519
Secret phrase:       conduct kitchen clump peasant acquire detail diagram balcony barrel census subway dove
  Network ID:        substrate
  Secret seed:       0xd3a656b0d6ccacd7428da7fda811e6e1c89cc26035e649cbf680f7a6de8aa25e
  Public key (hex):  0x2f430c4b3caebdb85bfe2ec034152fb2f110749b94283262bfb0226161f0b1f5
  Account ID:        0x2f430c4b3caebdb85bfe2ec034152fb2f110749b94283262bfb0226161f0b1f5
  Public key (SS58): 5D8g2Quafm4VDfsrDBxR4YYKwkeX2aKuwV7rkNDFN7bfdBS8
  SS58 Address:      5D8g2Quafm4VDfsrDBxR4YYKwkeX2aKuwV7rkNDFN7bfdBS8

➜  bin ./polkadot-parachain key generate --scheme ed25519
Secret phrase:       useful hundred vault mass vessel budget wait dutch expect mystery level quiz
  Network ID:        substrate
  Secret seed:       0x120870f7aeb7ace021ac3539be456dd09ba4fb456b6f3afd2c02f79c132410c9
  Public key (hex):  0xb6948fc8c313e24313cf6a42227be8cbd96dc71e7449beb4be35ee61a942282c
  Account ID:        0xb6948fc8c313e24313cf6a42227be8cbd96dc71e7449beb4be35ee61a942282c
  Public key (SS58): 5GC6hoCtbURddcXwtEg1rh1xZaTxyGz8urNpBBLvnwbG93Eu
  SS58 Address:      5GC6hoCtbURddcXwtEg1rh1xZaTxyGz8urNpBBLvnwbG93Eu

Please note we are using the right scheme (ed25519) taking into account the Polkadot AssetHub was launched with aura key under ed25519.

[PierreBesson]

Did you try to do this ?

1. Generate Paseo local plain chainspec
2. Edit/Check the plain chainspec to add the initial authorities in aura public keys
3. Convert the plain into raw chainspec

[al3mart]

I haven't modified the plain spec manually yet, was focused on fixing whatever might be happening on the chain spec generator side that is not producing the right output(?).

Well worth a try! Thanks for spending some time @PierreBesson, appreciate it

[BulatSaif]

1. Can you show that file you have in keystore, you should see file staring with 0x61757261 + pub key?

2. Can you also run RPC call on parachain 1: author.hasKey(publicKey, keyType) author.hasSessionKeys(sessionKeys)

for example:

author.hasKey(0x2f430c4b3caebdb85bfe2ec034152fb2f110749b94283262bfb0226161f0b1f5, 'aura')
author.hasSessionKeys(0x617572612f430c4b3caebdb85bfe2ec034152fb2f110749b94283262bfb0226161f0b1f5)

you should get true for both calls.

[al3mart]

https://github.com/paseo-network/runtimes/pull/53

This should get us going!

[al3mart]

@BulatSaif

As per the above comment I have made some changes to my local set up. Though, before that I did check those RPC calls and everything seemed correct. My node was using Alice keys, and checking for the corresponding ED25519 for its session keys was returning no errors.

At the same time I believe I must have overlooked something, when the collator was trying to produce blocks I was getting the following error in the logs:

2024-03-14 13:38:18.039 ERROR tokio-runtime-worker aura::cumulus: [Parachain] err=CannotSign("Validation error: Requested public key and public key of the loaded private key do not match. \n\n\t\t\tThis means either that the keystore password is incorrect or that the private key was stored under a wrong public key.. Key: Public(88dc3417d5058ec4b4503e0c12ea1a0a89be200fe98922423d4334014fa6b0ee (5FA9nQDV...))")

I have opened a PR that moves away from using ED25519 for the collator session keys and uses SR25519 instead. While being the usual, we wanted to keep this as close to Polkadot as possible, even mirroring these little details to provide the closes testing environment.