search

fboucquez / symbol-bootstrap

A tool that allows you to quickly configure and setup Symbol testnets and nodes.
Apache License 2.0
47 stars 27 forks source link

Wizard and Pack #190

Closed fboucquez closed 3 years ago

fboucquez commented 3 years ago

Fixes https://github.com/nemtech/symbol-bootstrap/issues/103

Please have a run by calling

symbol-bootstrap wizard

image

And follow the question.

This PR also adds the "pack" command that zips a ready-to-use node configuration that can be copied to the node machine.

mm-s commented 3 years ago

I must ctrl-c now as I am in a deadlock:

? Enter the 64 HEX private key of the VRF account. 
>> Invalid private key. It must have 64 hex characters.

Hitting enter with intent of cancelling the question, with intention of choosing another option from the prev menu. Kepps asking for the private key instead of returning to prev menu with items Generating a brand generated account and Entering a private key

mm-s commented 3 years ago

minor: Enter the Symbol host of your future node I wondered if the meaning is "Enter the hostname of your future Symbol node", hostname is unix name, DNS name, can it be IP address?, or is it the friendlyName?. Perhaps is convenient a rewrite of the question

mm-s commented 3 years ago

Last question:

? Would you like to generate the configuration now? No

Zip file mainnet-dual-node.zip size 0 MB has been created.

Symbol Bootstrap preset file 'custom-plain-text-mainnet-preset.yml' created. Keep this safe!
...

It didn't respect my negative input and should have cancelled

fboucquez commented 3 years ago

Thanks @mm-s !

Updates:

mm-s commented 3 years ago

Minor: Found this error right at the end of the process after answering all questions. It could have aborted right at the beginning.

    Error: custom-plain-text-mainnet-preset.yml already exist!!! you may want to move it somewhere else before overwriting it!
mm-s commented 3 years ago

Thanks : )

fboucquez commented 3 years ago

Thanks, Renamed the file to always be "custom-preset.yml". Moved the validation to the top. Added target folder validation

segfaultxavi commented 3 years ago

wizard does not accept the -t parameter, output must always be target. This is not consistent with the config command.

segfaultxavi commented 3 years ago

All uses of brand must be brand new, or simply new.

segfaultxavi commented 3 years ago

If I try to enter my main account's private key it gets stuck in a loop!

? How do you want to import your accounts? Private Keys: The private Keys will be generated or entered.
? How do you want to create the Main account: Entering a private key
? Enter the 64 HEX private key of the Main account (or press enter to select the option again). ****************************************************************
? How do you want to create the Main account: Entering a private key
? Enter the 64 HEX private key of the Main account (or press enter to select the option again). ****************************************************************
? How do you want to create the Main account: Entering a private key
? Enter the 64 HEX private key of the Main account (or press enter to select the option again). ****************************************************************
? How do you want to create the Main account: (Use arrow keys)
> Generating a brand account
  Entering a private key
fboucquez commented 3 years ago

Update:

I have split the wizard into the wizard and the pack command.

./bin/run pack

Workflow: 1) symbol-bootstrap wizard 2) optional, edit custom-preset.yml to further customize. 3) either symbol-bootstrap start (same machine) or symbol-bootstrap pack (a zip to deploy in the node machine)

Hi @segfaultxavi @davehodgson , please follow the current problem the wizard will also fix:

Currently, bootstrap config/start generates private keys when they are not provided in custom preset. This is fine but then if the user needs to reset for whatever reason, the keys may get lost and we end up in this issue https://nem2.slack.com/archives/C9YKR0EUX/p1616885367211100 (user messed up the upgrade process and didn't use the backup to retry upgrading/braking the secondary keys)

The current alternative is for the user to provide the private keys via a custom preset. This is doable but it has several issues: 1) The user needs to create a plain text custom preset file to then encrypt with the command (more places to leak private keys). Then remember to delete the plain text file. https://github.com/nemtech/symbol-bootstrap/blob/main/docs/presetGuides.md#private-keys-and-security. Not hard but not trivial to do. 2) User may miss one of the custom private keys going back to the problem above. For example, the offline guide only provides the main private key https://deploy-preview-606--nemtech.netlify.app/guides/network/running-a-secure-symbol-node.html 3) The user may not know how to create private keys and end up in a start/stop/start "solution". The user uses bootstrap to generate the initial private keys, then stop to copy those keys and start again with these keys in the custom preset. Similar to what's described in https://forum.nem.io/t/complete-guide-to-run-a-dual-node-on-centos-8-using-symbol-bootstrap/29268 . People have ad-hoc ways to backup private keys, which is fine but then they may not know how to provide them to bootstrap in case of a reset.

This wizard will help the user creating the right custom preset with the right encrypted keys (generated or provided) without intermediate files. It will explain to the user what to do next, how to link/enrol, and how to start a node in the current machine or how to "pack" the target to run an external machine. The custom preset created and encrypted in the dev machine will have the private keys and not the machine running the node necessarily.

I'm keen to ask the community what are the most common customization so we can add them to the wizard. Atm the wizard has prompts for:

We could add for example:

This PR doesn't change the behavior of the start/config commands. The wizard and pack are additions to bootstrap that use them.

segfaultxavi commented 3 years ago

The wizard requests information regarding the different keys. To help the user it should briefly describe what's the purpose of each key, or link to the docs.

? How do you want to create the Transport account:
> Generating a new account
  Entering a private key
segfaultxavi commented 3 years ago

This was not obvious to me when reading the description, but the wizard only works when you are offline. Therefore, I think this new command should be called offline-wizard, and it should be made clear in the instructions that it is only needed when you don't want your private keys to ever be online. For the rest of the users, standard symbol-bootstrap start on the node machine is still the recommended procedure, right?

segfaultxavi commented 3 years ago

Minor issue: On Windows the zip progress indicator didn't display very nicely (because of ANSI escape codes):

2021-04-06T08:09:21.313Z info     The docker-compose.yml file created D:\NEM\repos\symbol-bootstrap\target\docker\docker-compose.yml
\033[0G1 entries zipped!\033[0G2 entries zipped!\033[0G3 entries zipped!\033[0G4 entries zipped!\033[0G5 entries zipped!\033[0G6 entries zipped!\033[0G7 entries zipped!\033[0G8 entries zipped!\033[0G9 entries zipped!\033[0G10 entries zipped!\033[0G11 entries zipped!\033[0G12 entries zipped!\033[0G13 entries zipped!\033[0G14 entries zipped!\033[0G15 entries zipped!\033[0G16 entries zipped!\033[0G17 entries zipped!\033[0G18 entries zipped!\033[0G19 entries zipped!\033[0G20 entries zipped!\033[0G21 entries zipped!\033[0G22 entries zipped!\033[0G23 entries zipped!\033[0G24 entries zipped!\033[0G25 entries zipped!\033[0G26 entries zipped!\033[0G27 entries zipped!\033[0G28 entries zipped!\033[0G29 entries zipped!\033[0G30 entries zipped!\033[0G31 entries zipped!\033[0G32 entries zipped!\033[0G33 entries zipped!\033[0G34 entries zipped!\033[0G35 entries zipped!\033[0G36 entries zipped!\033[0G37 entries zipped!\033[0G38 entries zipped!\033[0G39 entries zipped!\033[0G40 entries zipped!\033[0G41 entries zipped!\033[0G42 entries zipped!\033[0G43 entries zipped!\033[0G44 entries zipped!\033[0G45 entries zipped!\033[0G46 entries zipped!\033[0G47 entries zipped!\033[0G48 entries zipped!\033[0G49 entries zipped!\033[0G50 entries zipped!\033[0G51 entries zipped!\033[0G52 entries zipped!\033[0G53 entries zipped!\033[0G54 entries zipped!\033[0G55 entries zipped!\033[0G56 entries zipped!\033[0G57 entries zipped!\033[0G58 entries zipped!\033[0G59 entries zipped!\033[0G60 entries zipped!\033[0G61 entries zipped!\033[0G62 entries zipped!\033[0G63 entries zipped!\033[0G64 entries zipped!\033[0G65 entries zipped!\033[0G66 entries zipped!\033[0G67 entries zipped!\033[0G68 entries zipped!\033[0G69 entries zipped!\033[0G70 entries zipped!\033[0G71 entries zipped!\033[0G72 entries zipped!\033[0G73 entries zipped!\033[0G74 entries zipped!\033[0G75 entries zipped!\033[0G76 entries zipped!\033[0G77 entries zipped!\033[0G78 entries zipped!\033[0G79 entries zipped!\033[0G80 entries zipped!\033[0G81 entries zipped!\033[0G82 entries zipped!\033[0G83 entries zipped!\033[0G84 entries zipped!\033[0G85 entries zipped!\033[0G86 entries zipped!\033[0G87 entries zipped!\033[0G88 entries zipped!\033[0G89 entries zipped!\033[0G90 entries zipped!\033[0G91 entries zipped!\033[0G92 entries zipped!\033[0G93 entries zipped!\033[0G94 entries zipped!\033[0G95 entries zipped!\033[0G96 entries zipped!\033[0G97 entries zipped!\033[0G98 entries zipped!\033[0G99 entries zipped!\033[0G100 entries zipped!\033[0G101 entries zipped!\033[0G102 entries zipped!\033[0G103 entries zipped!\033[0G104 entries zipped!\033[0G105 entries zipped!\033[0G106 entries zipped!\033[0G107 entries zipped!\033[0G108 entries zipped!\033[0G109 entries zipped!\033[0G110 entries zipped!\033[0G111 entries zipped!\033[0G112 entries zipped!\033[0G113 entries zipped!
Zip file testnet-dual-node.zip size 116 MB has been created.
segfaultxavi commented 3 years ago

The wizard command made sure that the configuration machine is offline before creating the keys, but then, when I run pack, it runs Docker which needs to download images from the Internet. Isn't this inconsistent?

fboucquez commented 3 years ago

Thanks @segfaultxavi

Key Descriptions

The wizard requests information regarding the different keys. To help the user it should briefly describe what's the purpose of each key, or link to the docs.

? How do you want to create the Transport account:
> Generating a new account
  Entering a private key

Something like this?

**Symbol bootstrap is going to resolve the key pairs the node requires for running. You can read more by going to https://docs.symbolplatform.com/concepts/cryptography.html#key-pair**

? Enter password to use to encrypt and decrypt custom presets, addresses.yml, and preset.yml files. When providing a password, private keys will be encrypted. Keep this password in a secure place! 
? How do you want to import your accounts? Private Keys: The private Keys will be generated or entered.

Main Key Pair: It holds the tokens required to give the node its importance.

**Main Key Pair: It holds the tokens required to give the node its importance.**
? How do you want to create the Main account: Generating a new account

Using account NAQPM5-T7LEQX-OAVROG-EO3I7R-XYRKZ3-DINRYR-ETA for Main key. It will be stored in your custom preset. Keep file safe!

**Transport Key Pair: It is used by nodes for secure transport over TLS.**
? How do you want to create the Transport account: Generating a new account

Using account NB747H-PGGSKC-BBER2W-LBRY6C-BZ3QNH-32VOHJ-RNQ for Transport key. It will be stored in your custom preset. Keep file safe!

**VRF Key Pair: It is required for harvesting.**
? How do you want to create the VRF account: Generating a new account

Using account NA7TMP-B6OG5W-RQOAVC-3HOHTX-7C3UX5-C3CZMJ-6HI for VRF key. It will be stored in your custom preset. Keep file safe!

**Remote Key Pair: It is used to harvest and collect the rewards on behalf of the main account in remote harvesting.**
? How do you want to create the Remote account: Generating a new account

Using account NA3T76-5SSBIA-EXD64P-KUUSLN-ZCDXCN-JDX2A5-FZY for Remote key. It will be stored in your custom preset. Keep file safe!

Offilne mode

This was not obvious to me when reading the description, but the wizard only works when you are offline. Therefore, I think this new command should be called offline-wizard, and it should be made clear in the instructions that it is only needed when you don't want your private keys to ever be online. For the rest of the users, standard symbol-bootstrap start on the node machine is still the recommended procedure, right?

The wizard is for new users and helps them have a safe and reproducible setup. It's basically the guide in the wizard for the way I see it. Atm, it forces the user to be offline but it could very much be a strong suggestion.

Pack offline mode

The wizard command made sure that the configuration machine is offline before creating the keys, but then, when I run pack, it runs Docker which needs to download images from the Internet. Isn't this inconsistent?

Yeap, I could upgrade the pack command to pull the images and strongly suggest the user to be offline mode when asking for passwords, the config generation, and zip. I have a couple of PRs

segfaultxavi commented 3 years ago

Something like this?

Your suggestions look nice. I would only change the first one, though:

Symbol bootstrap needs to provide the node with a number of key pairs (Read more at https://docs.symbolplatform.com/concepts/cryptography.html#symbol-keys).

But still, I would add a hint for the totally clueless users:

If you don't know what a key is used for, let Symbol Bootstrap generate a new one for you.

The wizard is for new users and helps them have a safe and reproducible setup. It's basically the guide in the wizard for the way I see it. Atm, it forces the user to be offline but it could very much be a strong suggestion.

Hmmmm... yeah, I think a strong suggestion makes more sense. But I am still not sure how much more secure this procedure is. I can imagine users will just unplug the network cable when bootstrap requests it, and plug it back in afterwards. Which is not the idea of an offline machine, right?

fboucquez commented 3 years ago

Hmmmm... yeah, I think a strong suggestion makes more sense.

What about?

'Bootstrap is going to ask and/or generate private keys and mnemonic phrases. It's highly recommended that this machine is offline. Is it offline? Say yes if you are offline or if you don't care.'

But I am still not sure how much more secure this procedure is. I can imagine users will just unplug the network cable when bootstrap requests it, and plug it back in afterwards. Which is not the idea of an offline machine, right?

the wizard can run completely offline once bootstrap package is installed. The pack and config commands need to pull the images before generating the configuration (due to openssl and catapult tool installation). Once we merge the other PRs it won't be a requirement anymore.

Options: 1) wizard -> ask offline -> generate keys -> pack -> ask online-> pull image -> ask offline -> generate configuration and zip 2) wizard -> pull image -> ask offline -> generate key -> pack -> generate configuration and zip

I guess even though the wizard doesn't require to pull the images, we could pull them in that command just for simplicity.

The alternative is we remove the offline suggestions from the wizard if it seems too forced until we have the other PRs merged

segfaultxavi commented 3 years ago

'Bootstrap is going to ask and/or generate private keys and mnemonic phrases. It's highly recommended that this machine is offline. Is it offline? Say yes if you are offline or if you don't care.'

Hahaha, it's rather wordy 😄 How about Symbol Bootstrap is about to start working with sensitive information (private keys or mnemonic phrases) so it is highly recommended that you disconnect from the network before continuing. And then you don't ask anything, because you don't care about the answer, right?

  1. wizard -> ask offline -> generate keys -> pack -> ask online-> pull image -> ask offline -> generate configuration and zip
  2. wizard -> pull image -> ask offline -> generate key -> pack -> generate configuration and zip

I very much prefer number 2, yeah. When everything is pre-packaged inside bootstrap then only a warning at the beginning will be needed.

fboucquez commented 3 years ago

Thanks @segfaultxavi , coud you try again? I've done the feedback fixes. Option 2 for pulling before going offline in the wizard. I think I fixed the new line issue in the pack command for windows.

fboucquez commented 3 years ago

It pulled the images right at the start, which is nice, but it failed (because I didn't have Docker running) and it just continued with a warning (It failed later during the pack command).

It seems expected. Although having a "verify" command that validates all the right things and versions are installed is my pipeline for a future PR.

https://github.com/nemtech/symbol-bootstrap/issues/119