create `rivinecg` tool: a rivine blockchain generator

[GlenDC]

For the TFB chain ticket (#534) I already started creating a template Rivine chain repo at https://github.com/threefoldtech/rivine-chain-template. It still requires a lot of manual work however to replace all the stuff, with a lot of information to be replaced duplicate information. We cannot simply make a chain repo that is ready to compile simply be providing a config file that it reads on runtime. We can however go pretty close to that. That is where the rivinecg tool comes into the picture.

Goal would be that the template repo is really files that are either already ready to use-as-is, or they end in the .template suffix and thus will be parsed using the rivinecg tool chain generation process injecting the variables provided by the chain config as is.

Commands

Generation commands we would provide:

rivinecg generate blockchain \
    --config blockchain.cfg \ 
    --directory my-awesome-chain

Will generate a new chain in the target directory. If no target directory is specified the current directory will be used on the condition that it is empty (with a single cfg file as exception). A config has to be specified and contains the variables that will configure the blockchain.

rivinecg generate config \
    --output blockchain.cfg

Generate a config that can be used to generate a blockchain with the rivinecg generate blockchain command.

rivinecg generate seed \
    -n 1

Generates a new seed as useable for Rivine-based blockchain accounts. The seed is returned as a mnemonic on the STDOUT in a single line. By default also the first address is returned, but you can return more addresses by specifying a higher amount with the -n flag.

Other command that we would provide:

rivinecg version

Returns the current version.

Blockchain Generation Config

The config used for the blockchain generation would look as follows:

template: # optional, will be defined as-is shown here if undefined
    repository: https://github.com/threefoldtech/rivine-chain-template
    version: master # can also be a release tag, but for now there is only a master branch
blockchain:
    name: rivine # required
    repository: github.com/threefoldtech/rivine # required, repo of your blockchain
    currency: # required
        unit: ROC
        precision: 9
    ports: # required
        api: 23110
        rpc: 23112
    binaries: # optional
        # name of binary client
        client: rivinec # optional, will be `<blockchain.name>c` if undefined
        # name of binary daemon
        daemon: rivindc # optional, will be `<blockchain.name>d` if undefined
    tranactions:
        default: # optional, by default it is 1
            version: 1
        minting:
            mintConditionUpdate: # required if minting defined
                version: 128 # required
            coinCreation:
                version: 129  # required
            coinDestruction: # optional, disabled if not defined
                version: 130 # required if parent defined
            requireMinerFees: No
    networks: # at least one network is required, possible networks: standard, testnet and devnet (no other name is allowed)
        standard:
            genesis: # required
                coinOutputs:
                    - value: 100000000 # expressed in one coin unit
                      condition: 01b5e42056ef394f2ad9b511a61cec874d25bebe2095682dd37455cbafed4bec154e382a23f90e
                blockStakeOutputs:
                    - value: 1000000
                      condition: 01b5e42056ef394f2ad9b511a61cec874d25bebe2095682dd37455cbafed4bec154e382a23f90e
                minting: # required only if minting transactions have been defined
                    condition: # can also be single sig, a multisig condition is expressed as follows:
                        addresses:
                            - 01434535fd01243c02c277cd58d71423163767a575a8ae44e15807bf545e4a8456a5c4afabad51
                            - 01334cf68f312026ff9df84fc023558db8624bedd717adcc9edc6900488cf6df54ac8e3d1c89a8
                            - 0149a5496fea27315b7db6251e5dfda23bc9d4bf677c5a5c2d70f1382c44357197d8453d9dfa32
                        signaturesRequired: 2
            # optional, if no defined the block creator gets the fee, otherwise this pool address will receive all tx fees
            transactionFeePool: 017267221ef1947bb18506e390f1f9446b995acfb6d08d8e39508bb974d9830b8cb8fdca788e34
            # all the options below have sane defaults for the networks: standard, testnet and devnet
            blockSizeLimit: 2e6
            arbitraryDataSizeLimit: 83
            blockCreatorFee: 1.0
            minimumTransactionFee: 0.1
            blockFrequency: 120
            maturityDelay: 144 # expressed in blocks
            medianTimestampWindow: 11
            targetWindow: 1e3
            maxAdjustmentUp: 25/10
            maxAdjustmentDown: 10/25
            futureThreshold: 1h
            extremeFutureThreshold: 2h
            stakeModifierDelay: 2000s
            blockStakeAging: 1d
            transactionPool:
                transactionSizeLimit: 16e3
                transactionSetSizeLimit: 250e3
                poolSizeLimit: 2e6 - 5e3 - 250e3
        testnet:
            genesis: # required
                coinOutputs:
                    - value: 100000000 # expressed in one coin unit
                      condition: 01fc8714235d549f890f35e52d745b9eeeee34926f96c4b9ef1689832f338d9349b453898f7e51
                blockStakeOutputs:
                    - value: 3000
                      condition: 01fc8714235d549f890f35e52d745b9eeeee34926f96c4b9ef1689832f338d9349b453898f7e51
                minting: # required only if minting transactions have been defined
                    condition: # can also be single sig, a multisig condition is expressed as follows:
                        addresses:
                            - 016148ac9b17828e0933796eaca94418a376f2aa3fefa15685cea5fa462093f0150e09067f7512
                            - 01d553fab496f3fd6092e25ce60e6f72e24b57950bffc0d372d659e38e5a95e89fb117b4eb3481
                            - 013a787bf6248c518aee3a040a14b0dd3a029bc8e9b19a1823faf5bcdde397f4201ad01aace4c9
                        signaturesRequired: 2
            # all the options below have sane defaults for the networks: standard, testnet and devnet
            blockSizeLimit: 2e6
            arbitraryDataSizeLimit: 83
            blockCreatorFee: 10.0
            minimumTransactionFee: 0.1
            blockFrequency: 120
            maturityDelay: 720
            medianTimestampWindow: 11
            targetWindow: 1e3
            maxAdjustmentUp: 25/10
            maxAdjustmentDown: 10/25
            futureThreshold: 3s
            extremeFutureThreshold: 6s
            stakeModifierDelay: 20s
            blockStakeAging: 1024s
            transactionPool:
                transactionSizeLimit: 16e3
                transactionSetSizeLimit: 250e3
                poolSizeLimit: 2e6 - 5e3 - 250e3
        devnet:
            genesis: # required
                coinOutputs:
                    - value: 100000000 # expressed in one coin unit
                      # belong to wallet with mnemonic:
                      # carbon boss inject cover mountain fetch fiber fit tornado cloth wing dinosaur proof joy intact fabric thumb rebel borrow poet chair network expire else
                      condition: 015a080a9259b9d4aaa550e2156f49b1a79a64c7ea463d810d4493e8242e6791584fbdac553e6f
                blockStakeOutputs:
                    - value: 1000000
                      condition: 015a080a9259b9d4aaa550e2156f49b1a79a64c7ea463d810d4493e8242e6791584fbdac553e6f
                minting: # required only if minting transactions have been defined
                    condition: 015a080a9259b9d4aaa550e2156f49b1a79a64c7ea463d810d4493e8242e6791584fbdac553e6f
            # all the options below have sane defaults for the networks: standard, testnet and devnet
            blockSizeLimit: 2e6
            arbitraryDataSizeLimit: 83
            blockCreatorFee: 10.0
            minimumTransactionFee: 0.1
            blockFrequency: 12
            maturityDelay: 10
            medianTimestampWindow: 11
            targetWindow: 20
            maxAdjustmentUp: 120/100
            maxAdjustmentDown: 100/120
            futureThreshold: 2m
            extremeFutureThreshold: 4m
            stakeModifierDelay: 2000s
            blockStakeAging: 1024s
            transactionPool:
                transactionSizeLimit: 16e3
                transactionSetSizeLimit: 250e3
                poolSizeLimit: 2e6 - 5e3 - 250e3

Templates

For .template files we can simply use the template format as defined by the Go std package https://golang.org/pkg/text/template/, no need to go crazier, it has all we needs. This we can apply to all Go, JS, Documentation, Makefile, script and all other files that require templating, based on the config as defined in the "Blockchain Generation Config" section.

How to create a chain

Creating a chain will than become very easy, and can be done as follows

1. create an empty (NOT initialized) repository at GitHub (or any other remote provider) as myorg/awesomechain (or however you want to name it)
2. create and go to repo: mkdir -p $GOPATH/github.com/myorg/awesomechain && cd $GOPATH/github.com/myorg/awesomechain
3. install/update rivinecg tool: go get -u github.com/threefoldtech/rivine/cmd/rivinecg
4. generate the default config, to be modified: rivinecg generate config
5. modify the generated config to suit your needs: vim blockchain.cfg
6. generate your blockchain: rivinecg generate blockchain
7. initialize your git repo: git init
8. commit your initial blockchain: git add -A && git commit -m "initial commit: blockchain codebase ready to be used"
9. vendor all the dependencies, we recommend dep but you can any tool you wish. For dep you would do it as follows: dep init
10. commit your dependencies: git add -A && git commit -m "add all blockchain code dependencies"
11. add your remote: git remote add git@github.com:myorg/awesomechain.git
12. push your codebase ready to be used by others git push origin master

You know have your blockchain codebase ready to be used by you and others, fully working out of the box.

To build in development mode (DEBUG=TRUE) you can use make install, to build in production mode you can use make install-std.

Once you have your blockchain installed you can launch your daemon anywhere using:

awesomechaind

You can control your daemon using the CLI client awesomechainc. Please consult the --help flag for both awesomechaind and awesomechainc for more information on how to use these binaries.

Please also read the README generated for your awesome chain to know how to start playing with your blockchain on a local devnet.

Conclusion

Creating a new Rivine-based blockchain becomes super easy and fast once we have this tool. I therefore recommend that we wait with task #534 (create TFB chain) and #533 (create FFT chain) until we have this tool. In fact I would wait until we have Rivine 1.2 in a stable and releasable state, so that we don't require any further updates to the TFB and FFT codebase once it is generated.

[GlenDC]

Template files will have to be worked on and committed to https://github.com/threefoldtech/rivine-chain-template repo (master branch). The tool can be worked on from a local version, on the feat/rivinecg branch.

For the CLI use https://github.com/spf13/cobra, we already use it for our other commands as well. For the blockchain config files I would say use https://github.com/spf13/viper, so we can support JSON/YAML/TOML with ease, and let's use YAML by default.

The tool should come under ./cmd/rivinecg of this repo (under the feat/rivinecg Git branch)

Tasks:

• [x] Create a working tool, that already has the rivinecg version and rivinecg generate seed commands. The version should be the same version as we use for the rest of the codebase. That version command returns similar information as you get when returning rivinec version (where relevant of course). The generate seed cmd has already been discussed in detail.
• [x] Create the rivinecg generate config cmd . The config file command already has been discussed in detail on the other page. Ensure that the config file is fully validated when loaded into memory and that errors are reported clearly. Make sure it can support reading YAML/TOML/JSON automatically.
• [x] Create a first version of the rivinecg generate blockchain, it should do the following:
  • prepare the environment (either create the specified dir, or if that location is already a dir (default the current dir), than ensure it is empty except for the allowed config file as specified by the --config (or -c) flag.
  • parse and read the config
  • download the template files from github as a raw tarball using the specified url and version
  • move all files (in a recursive manner) from the unarchived root dir to the output location (ignore the root todo folder for now, that is the folder we keep in all files that you still need to move, and if needed templatify, to the root dir (in the same file structure as the todo folder) :
    • if the file does not end in the .template suffix, just move as-is
    • if it does end with a .template suffix:
      • read it using the https://golang.org/pkg/text/template/ file, apply the config blockchain config file object as the only argument on it, and write the file to the new location to the same path (be it in a directory or not, so following the same file structure), with the same filename, but remove the .template suffix from it.
• [x] Make it work for a plain blockchain, by moving the necessary files outside the todo directory, and modifying them into .template files if required. Don't copy blindly, you can probably improve a lot in these code files and be careful. After this task is done you should be able to launch a blockchain with 1, 2 or 3 networks, but not yet with the minting extension allowed.
• [x] Support the minting transactions (2 or all 3), by also generating the needed code for the minting plugin in case the minting transactions are defined in the config file. After this task is done you should now be able to generate a blockchain with 1, 2 or 3 networks with the minting extension embedded, if needed.
• [x] add documentation, improve and test

[DylanVerstraete]

I completed the validation for the config struct, now writing some tests. Need to test following cases.

• [x] config without the blockchain property
• [x] config without the blockchain.name property
• [x] config without the blockchain.repository property
• [x] config without the blockchain.currency property
• [x] config without the blockchain.ports property
• [x] config with the same blockchain.ports.api and blockchain.ports.rpc property
• [x] config with minting property in transactions and without minting property in a network
• [x] config without the networks property
• [x] config with an empty array networks property (need to be atleast one)
• [x] config without the genesis property in a network
• [x] config with single-sig and/or multisig values in network.genesis.coinOutputs / network.genesis.blockStakeOutputs/network.genesis.minting property
• [x] config without values that are optional

[GlenDC]

Merged and ready to use. Might make some further improvements while I review the ThreefoldBonus Chain tomorrow, as ideally a generated rivine blockchain doesn't modify manually any generated files.