Hybrid Consensus Mode: Essential task `grandpa-voter` failed. Shutting down service.

[vessenes]

When following the tutorial here https://substrate.dev/docs/en/tutorials/start-a-private-network/customchain, I successfully get two nodes up and running and talking to each-other in mining mode, then proceed to try and turn on GRANDPA voting.

The documents specify that I should connect to a node and call author with keyType gran and enter the mnemonic and the public key generated by subkey; once those calls are in, I should restart the nodes and they will begin to function as finalizing nodes.

This doesn't seem to work; instead, I get:

% target/release/hybrid-consensus \
--base-path /tmp/node02 \
  --chain ./customSpecRaw.json \
  --port 30334 \
  --ws-port 9945 \
  --rpc-port 9934 \
  --telemetry-url 'wss://telemetry.polkadot.io/submit/ 0' \
  --validator \
  --rpc-methods=Unsafe \
  --name MyNode02 \
  --bootnodes /ip4/127.0.0.1/tcp/9615/p2p/12D3KooWQAmVsNzwugd1Z62YPa467iBH621cXCCeRg5H2NwGVtog
2020-08-31 16:50:57 Hybrid Consensus Node (PoW and Grandpa)
2020-08-31 16:50:57 ✌️  version 2.0.0-rc5-42ac927-x86_64-macos
2020-08-31 16:50:57 ❤️  by Substrate DevHub <https://github.com/substrate-developer-hub>, 2019-2020
2020-08-31 16:50:57 📋 Chain specification: Transformer Chain
2020-08-31 16:50:57 🏷  Node name: MyNode02
2020-08-31 16:50:57 👤 Role: AUTHORITY
2020-08-31 16:50:57 💾 Database: RocksDb at /tmp/node02/chains/transformer/db
2020-08-31 16:50:57 ⛓  Native runtime: minimal-grandpa-runtime-1 (minimal-grandpa-runtime-1.tx1.au1)
2020-08-31 16:50:57 📦 Highest known block at #289
2020-08-31 16:50:57 Using default protocol ID "sup" because none is configured in the chain specs
2020-08-31 16:50:57 🏷  Local node identity is: 12D3KooWLN9dYLXF2Coo8Dmx7Zu1KHpeQY4d62gXsrSzqf7u2xYV (legacy representation: QmYSe7ZXfAWn46yHK5QEeJtYMM9nWNZVQJc77QVauVa7Sh)
2020-08-31 16:50:57 🙌 Starting consensus session on top of parent 0xe8731b6e26e991af0996cc9170dd7e8f0807beab56864f29c144a2a9c83b4cb5
2020-08-31 16:50:57 Timeout fired waiting for transaction pool at block #289. Proceeding with production.
2020-08-31 16:50:57 🎁 Prepared block for proposing at 290 [hash: 0x3d3123065aff70540ce0f47c37f8d315916ba82c742cd69b79d42e617d54e18a; parent_hash: 0xe873…4cb5; extrinsics (1): [0x940f…2d3d]]
2020-08-31 16:50:58 🙌 Starting consensus session on top of parent 0xb5f942b91950190301044586e8b9b329887d43474a0098ed7d56a3d4fb625dea
2020-08-31 16:50:58 ✨ Imported #290 (0xb5f9…5dea)
2020-08-31 16:50:58 🎁 Prepared block for proposing at 291 [hash: 0xca0cde13e7cd4facba69d51cbeb1a242720bd8d9a511bef14c09aef43159e29a; parent_hash: 0xb5f9…5dea; extrinsics (1): [0x5d30…6741]]
2020-08-31 16:50:58 Essential task `grandpa-voter` failed. Shutting down service.
Error: Input("Essential task failed.")

How do I debug this?

[JoshOrndorff]

I'm happy to see someone using this node in this way. I'm happy to help you debug.

One quick fix comes to mind. Make sure both nodes have clean data directories before staring. One way to do this is to replace the --base-path /tmp/node02 with --tmp.

If that doesn't solve your problem, please report back with the following information.

1. The commit of the recipes you built
2. Any modification to the node you made to the node (link a repo)
3. The complete chainspec you are using
4. The complete logs of both nodes
5. The keys output of subkey unless the keys are truly secret.

cc @danforbes who maintains that tutorial

[danforbes]

Would also suggest running the node with logging: RUST_LOG=debug RUST_BACKTRACE=1 target/release/hybrid-consensus -lruntime=debug <the rest of your args>

[JoshOrndorff]

@vessenes Since you made all the effort to use our hybrid consensus node, I really want to help you get working. I just recorded this screencast of me starting up a private network the way you're describing. Hope it helps, and please report back with any problems.

https://www.twitch.tv/videos/728265359

[vessenes]

Thanks! I'm parsing through all these -- and I'll get back with updates.

[JoshOrndorff]

Okay, good luck @vessenes I'll close this issue for now since it seems to be working, but feel free to continue posting here or re-open the issue if you get stuck again.

[vessenes]

Josh, thanks for the video and help. Three things I had done that you did differently

• I had an old pull (you mention this in the video)
• I might have relied on subkey to create ed25519 keys, but I don't think I specified them directly
• I used the web API to post the keys rather than curl.

When I pulled your commit, re-authored specifying ed25519 inspection of the mnemonic, and used curl, I can now finalize with hybrid-pow. Thanks again! If you have a sense of caution/doom using the web interface to post keys, you might dig in there -- but I really don't know which of the above was the thing that got me running, or perhaps a combo.

[JoshOrndorff]

I'm really glad you got it working!

Regarding your three differences:

• It's possible that there were bugs in older versions of Recipes, but I never encountered them. I don't think that was the problem.
• You definitely need ed25519 keys, so if you didn't instruct subkey that way, this is your problem.
• Using the Apps UI to insert your keys should work just fine. The one slight tripping point is that you need to connect the UI to each node individually when inserting the keys.

[inodelisia]

@JoshOrndorff Can you please repost the video you mentioned https://www.twitch.tv/videos/728265359 Thank you!

[JoshOrndorff]

I only had it on twitch and it got pruned sorry about that. Maybe you could try to record a similar one yourself and share it (better than I did) for posterity.

[inodelisia]

@JoshOrndorff Thank you for your answer! I was trying to find a hint, because I didn't managed to start a Hybrid consensus node to produce blocks, even in --dev mode or simple Alice and Bob nodes conected using local chain. I try to figure out what am I missing... I wanna mention that I compiled an older branch of recipe repository and it's working correctly.