Istanbul network doesn't start with existing accounts which are used as the validator set in the genesis file

IronGauntlets commented 6 years ago

@markya0616 @yutelin @alanchchen @bailantaotao @tailingchen I am trying to set a private Istanbul network, however, every client I bring up has a coinbase/etherbase which I didn't create and eth.accounts[0] is not equal to eth.coinbase, because of which the coinbase/etherbase address is not in validator set and hence network just halts thus nothing proceeds. Even using --etherbase flag to set the ehterbase base doesn't change it. Can you please tell me how the etherbase is set differently to the account I have in my keystore and how do I make sure that the client is started with the correct coinbase/etherbase?How do I create a network with existing accounts where Istanbul automatically recognises the account in keystore, like in other consensus algorithms?

In the backtrace, the unauthorised addresses are which is set by the client and unlocked accounts are the desired addresses which should have been set by the client and are in the validator set.

Steps to reproduce the behaviour

Create multiple accounts with the same password since its a test network
Add the public addresses to a toml file
Use the toml file to generate extra data
Add this extra data to the genesis file
Use the genesis file to initialise nodes
Create a bootnode
Launch each node is a tmux window under the same session

I have a bash script which takes in 2 arguments: 'location of geth binary' and 'the number of nodes to bring up the network', the following creates a network for 2 nodes which are connected using a bootnode (both files need to be in the same directory):

./istprivatenetwork.sh path/to/istanbulgeth/build/bin/geth 2

The script creates a new network everytime as well as a new tmux session, so please kill the tmux session before the script is run again.

istprivatenetwork.sh:

THIS_FILE_PARENT_DIR=`dirname \`readlink -f $0\``
ISTANBUL_DIR="`pwd`/istanbultestnet"
PASSWORD_PATH="${ISTANBUL_DIR}/passwd.txt"
NUMBER_OF_NODES=$2
GETH_BIN_PATH=`readlink -f $1`
ISTANBUL_TOOLS_PATH="${GOPATH}/src/github.com/getamis/istanbul-tools"
ISTANBUL_TOOLS_GITHUB="https://github.com/getamis/istanbul-tools.git"
TMUX_SESSION_NAME="istanbul_network"
BOOTNODE_PORT=4800
PORT=4800
RPC_PORT=9501
CHAINID=1530

# Create a common password file
createPasswd() {
    echo
    echo "---------Creating ${PASSWORD_PATH}---------"
    touch "${PASSWORD_PATH}"
    echo "istanbulnetwork" > "${PASSWORD_PATH}"
    echo "---------Finished creating ${PASSWORD_PATH}---------"
    echo
}

# Create a directory for the given path
createDir() {
    DIR="$1"
    if ! [ -d  "${DIR}" ]; then
        echo "---------Creating ${DIR}---------"
        mkdir -p "${DIR}"
        echo "---------Finished creating ${DIR}---------"
    fi
}

# Bye using Geth create the account in the given dir location
createAccounts() {
    echo "---------Creating Accounts---------"
    for i in `seq 1 "${NUMBER_OF_NODES}"`
    do
    echo
        if ! [ -d  ""${ISTANBUL_DIR}/node${i}"" ]; then
            mkdir -p ""${ISTANBUL_DIR}/node${i}""
        fi
        ${GETH_BIN_PATH} --datadir "${ISTANBUL_DIR}/node${i}" --password ${PASSWORD_PATH} account new
    echo
    done
    echo "---------Finished creating Accounts---------"
}

# Create toml file which has validator addresses in them
createToml() {
    VANITY="0x00"
    NODE_ADDRESSES=("$@")

    if [ -f "${ISTANBUL_DIR}/config.toml" ]; then
        rm "${ISTANBUL_DIR}/config.toml"
    fi

    echo "vanity = \"${VANITY}\"" >> "${ISTANBUL_DIR}/config.toml"

    count=1
    for i in ${NODE_ADDRESSES[@]}
    do
        if [ ${#NODE_ADDRESSES[@]} -eq "1" ]; then
            echo "validators = [\"0x${i}\"]" >> "${ISTANBUL_DIR}/config.toml"
        elif [ ${count} -eq "1" ]; then
            echo "validators = [\"0x${i}\"," >> "${ISTANBUL_DIR}/config.toml"
        elif [ ${count} -eq ${#NODE_ADDRESSES[@]} ]; then
            echo "${string}\"0x${i}\"]" >> "${ISTANBUL_DIR}/config.toml"
        else
            echo "${string}\"0x${i}\"," >> "${ISTANBUL_DIR}/config.toml"
        fi
        count=`expr ${count} + 1`
    done
}

# Create the genesis needed to initialise the nodes
createGenesis() {
    echo
    echo "---------Creating Genesis file---------"
    # Get the address from keystore and store them in an array
    ADDRESSES=()
    for i in `ls ${ISTANBUL_DIR} | grep node`
    do
        ADDRESSES+=(`ls ${ISTANBUL_DIR}/${i}/keystore | cut -d'-' -f9`)
    done

    # Clone istanbul-tools
    if ! [ -d "${ISTANBUL_TOOLS_PATH}" ]; then
        git clone "${ISTANBUL_TOOLS_GITHUB}" "${ISTANBUL_TOOLS_PATH}"
    fi

    # Make istanbul-tools
    if ! [ -f "${ISTANBUL_TOOLS_PATH}/build/bin/istanbul" ]; then
        make -C ${ISTANBUL_TOOLS_PATH}
    fi

    # Create config.toml with the validators
    createToml ${ADDRESSES[@]}

    # Call istanbul extra encode and store in a file
    ${ISTANBUL_TOOLS_PATH}/build/bin/istanbul extra encode --config "${ISTANBUL_DIR}/config.toml" | cut -d':' -f2 | tr -d " \t\n\r" >  "${ISTANBUL_DIR}/newextradata.txt"

    # Update genesis file, the genesis.json and extra data's file location is passed in and the rest of the addresses are passed to assign some ether
    node "${THIS_FILE_PARENT_DIR}/updategenesis.js" "${ISTANBUL_DIR}/genesis.json" "${ISTANBUL_DIR}/newextradata.txt" ${CHAINID} ${ADDRESSES[@]}
    echo "---------Finished creating Genesis file---------"
    echo
}

initialiseNodes() {
    echo
    echo "---------Initialising nodes with Genesis file---------"
    for i in `ls ${ISTANBUL_DIR} | grep node`
    do
    echo
        ${GETH_BIN_PATH} --datadir "${ISTANBUL_DIR}/${i}" init "${ISTANBUL_DIR}/genesis.json"
    echo
    done
    echo "---------Finished initialising nodes with Genesis file---------"
    echo
}

createBootNodeKey() {
    bootnode -genkey "${ISTANBUL_DIR}/boot.key"
}

launchBootNode() {
    tmux new -s ${TMUX_SESSION_NAME} -n "bootnode" -d
    tmux send-keys -t "${TMUX_SESSION_NAME}:bootnode" "bootnode -nodekey \"${ISTANBUL_DIR}/boot.key\" -verbosity 9 -addr :${BOOTNODE_PORT}" C-m
    PORT=`expr ${PORT} + 1`
}

launchNodes() {
    count=1
    for i in `ls ${ISTANBUL_DIR} | grep node`
    do
        tmux new-window -t ${TMUX_SESSION_NAME}:${count} -n ${i} 
        tmux send-keys -t ${TMUX_SESSION_NAME}:${count} "${GETH_BIN_PATH} --datadir "${ISTANBUL_DIR}/${i}" --syncmode 'full' --port ${PORT} --rpcport ${RPC_PORT} --rpc --rpcaddr '0.0.0.0' --rpccorsdomain '*' --rpcapi 'personal,db,eth,net,web3,txpool,miner,istanbul' --bootnodes 'enode://`bootnode -nodekey ${ISTANBUL_DIR}/boot.key -writeaddress`@127.0.0.1:${BOOTNODE_PORT}' --networkid ${CHAINID} --gasprice '0' -unlock \"0x`ls ${ISTANBUL_DIR}/${i}/keystore | cut -d'-' -f9`\" --password ${PASSWORD_PATH} --debug --mine --minerthreads '1' --etherbase \"0x`ls ${ISTANBUL_DIR}/${i}/keystore | cut -d'-' -f9`\"" C-m
        tmux split-window -h -t ${TMUX_SESSION_NAME}:${count}
        tmux send-keys -t ${TMUX_SESSION_NAME}:${count} "sleep 10s" C-m
        tmux send-keys -t ${TMUX_SESSION_NAME}:${count} "${GETH_BIN_PATH} attach ipc:${ISTANBUL_DIR}/${i}/geth.ipc" C-m

        PORT=`expr ${PORT} + 1`
        RPC_PORT=`expr ${RPC_PORT} + 1`
        count=`expr ${count} + 1`
    done
}

### Start of the main script
if ! [ ${NUMBER_OF_NODES} -eq ${NUMBER_OF_NODES} ] || [ -z ${NUMBER_OF_NODES} ] || [ ${NUMBER_OF_NODES} -lt "1" ]; then
    echo "Please enter geth binary path as first argument and number of nodes as second"
    exit 1
fi

if [ -z ${GETH_BIN_PATH} ]; then
    echo "Please enter geth binary path as first argument and number of nodes as second"
    exit 2
fi

if [ -d "${ISTANBUL_DIR}" ]; then
    rm -rf "${ISTANBUL_DIR}"
fi

createDir "${ISTANBUL_DIR}"
cd "${ISTANBUL_DIR}"

createPasswd

createAccounts

createGenesis

initialiseNodes

createBootNodeKey

launchBootNode
launchNodes

updategenesis.js:

const fs = require('fs')
const commandLineArgs = process.argv.slice(2)
const genesisFilePath = commandLineArgs[0]
const newExtraDataFilePath = commandLineArgs[1]
const newChainId = parseInt(commandLineArgs[2])
const addresses = commandLineArgs.slice(3)

let genesis = {
    "config": {
        "homesteadBlock": 1,
        "eip150Block": 2,
        "eip150Hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
        "eip155Block": 3,
        "eip158Block": 3,
        "istanbul": {
            "epoch": 30000,
            "policy": 0
        }
    },
    "nonce": "0x0",
    "timestamp": "0x0",
    "gasLimit": "0x47b760",
    "difficulty": "0x1",
    "mixHash": "0x63746963616c2062797a616e74696e65206661756c7420746f6c6572616e6365",
    "coinbase": "0x0000000000000000000000000000000000000000",
    "number": "0x0",
    "gasUsed": "0x0",
    "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000"
}

let newExtraData = fs.readFileSync(newExtraDataFilePath, 'utf8')
let startingBalance = "0x446c3b15f9926687d2c40534fdb564000000000000"

// Update genesis.extraData
genesis.extraData = newExtraData
genesis.alloc = {}
for (var i = 0 ; i < addresses.length; i++) {
    genesis.alloc[addresses[i]] = {"balance" : startingBalance}
}
genesis.config.chainId = newChainId

fs.writeFileSync(genesisFilePath, JSON.stringify(genesis, null, 4))

Backtrace

Node1:

[backtrace]

/home/user98/Documents/github/getamis/go-ethereum/build/bin/geth --datadir /home/user98/Documents/github/eth-private-network/istanbul/istanbultestnet/node1 --syncmode 'full' --port 4801 --rpcport 9501 --rpc --rpcaddr '0.0.0.0' --rpccorsdomain '*' --rpcapi 'personal,db,eth,net,web3,txpool,miner,istanbul' --bootnodes 'enode://297e9ce0012765db9086cf3dd3efa793879b7010e7c38464ce0a05b8bcbb4babe0b1841ec68feaa9c65bfb93bb8e360f2a8d7de35eb250efa7c1b44c483bb2ee@127.0.0.1:4800' --networkid 1530 --gasprice '0' -unlock "0x6816b996ef724a6d5002ad4b9ba9f6ad4351195e" --password /home/user98/Documents/github/eth-private-network/istanbul/istanbultestnet/passwd.txt --debug --mine --minerthreads '1' --etherbase "0x6816b996ef724a6d5002ad4b9ba9f6ad4351195e"
INFO [08-08|13:07:40|cmd/utils/flags.go:856] Maximum peer count                       ETH=25 LES=0 total=25
INFO [08-08|13:07:40|node/node.go:166]       Starting peer-to-peer node               instance=Geth/v1.8.3-unstable-c7547381/linux-amd64/go1.9.3
INFO [08-08|13:07:40|ethdb/database.go:63]   Allocated cache and file handles         database=/home/user98/Documents/github/eth-private-network/istanbul/istanbultestnet/node1/geth/chaindata cache=768 handles=512
WARN [08-08|13:07:40|eth/db_upgrade.go:48]   Upgrading database to use lookup entries
INFO [08-08|13:07:40|eth/backend.go:123]     Initialised chain configuration          config="{ChainID: 1530 Homestead: 1 DAO: <nil> DAOSupport: false EIP150: 2 EIP155: 3 EIP158: 3 Byzantium: <nil> Constantinople: <nil> Engine: istanbul}"
INFO [08-08|13:07:40|eth/db_upgrade.go:118]  Database deduplication successful        deduped=0
INFO [08-08|13:07:40|eth/backend.go:146]     Initialising Ethereum protocol           versions=[64] network=1530
INFO [08-08|13:07:40|core/blockchain.go:253] Loaded most recent local header          number=0 hash=fc61f9…29aaac td=1
INFO [08-08|13:07:40|core/blockchain.go:254] Loaded most recent local full block      number=0 hash=fc61f9…29aaac td=1
INFO [08-08|13:07:40|core/blockchain.go:255] Loaded most recent local fast block      number=0 hash=fc61f9…29aaac td=1
INFO [08-08|13:07:40|core/tx_journal.go:149] Regenerated local transaction journal    transactions=0 accounts=0
INFO [08-08|13:07:40|p2p/server.go:395]      Starting P2P networking
INFO [08-08|13:07:43|p2p/discover/udp.go:238] UDP listener up                          self=enode://0aa3fca386e81c9858f998ae7c3eba4bc48dd21f042e39ebe54f2e0f4689eca9191b4174cb29074ae90b21e8412e4d29020885bd4b17745971f3b594ef230597@146.198.136.74:4801
INFO [08-08|13:07:43|p2p/server.go:742]       RLPx listener up                         self=enode://0aa3fca386e81c9858f998ae7c3eba4bc48dd21f042e39ebe54f2e0f4689eca9191b4174cb29074ae90b21e8412e4d29020885bd4b17745971f3b594ef230597@146.198.136.74:4801
INFO [08-08|13:07:43|node/node.go:397]        HTTP endpoint opened                     url=http://0.0.0.0:9501 cors=* vhosts=localhost
INFO [08-08|13:07:43|node/node.go:327]        IPC endpoint opened                      url=/home/user98/Documents/github/eth-private-network/istanbul/istanbultestnet/node1/geth.ipc
INFO [08-08|13:07:44|geth/accountcmd.go:218]  Unlocked account                         address=0x6816b996ef724a6D5002ad4b9ba9f6AD4351195E
INFO [08-08|13:07:44|core/tx_pool.go:473]     Transaction pool price threshold updated price=0
INFO [08-08|13:07:44|miner/miner.go:116]      Starting mining operation
INFO [08-08|13:07:44|miner/worker.go:500]     Commit new mining work                   number=1 txs=0 uncles=0 elapsed=62.044µs
WARN [08-08|13:07:44|miner/agent.go:108]      Block sealing failed                     err=unauthorized
ERROR[08-08|13:07:54|consensus/istanbul/core/handler.go:141] Failed to decode message from payload    address=0xEF037C337667afC4f47d5b0B7FDDaD9ec6D2D2a5 err="unauthorized address"
ERROR[08-08|13:08:06|consensus/istanbul/core/handler.go:141] Failed to decode message from payload    address=0xEF037C337667afC4f47d5b0B7FDDaD9ec6D2D2a5 err="unauthorized address"
ERROR[08-08|13:08:20|consensus/istanbul/core/handler.go:141] Failed to decode message from payload    address=0xEF037C337667afC4f47d5b0B7FDDaD9ec6D2D2a5 err="unauthorized address"
ERROR[08-08|13:08:38|consensus/istanbul/core/handler.go:141] Failed to decode message from payload    address=0xEF037C337667afC4f47d5b0B7FDDaD9ec6D2D2a5 err="unauthorized address"
ERROR[08-08|13:09:04|consensus/istanbul/core/handler.go:141] Failed to decode message from payload    address=0xEF037C337667afC4f47d5b0B7FDDaD9ec6D2D2a5 err="unauthorized address"

Node2:

[backtrace]

/home/user98/Documents/github/getamis/go-ethereum/build/bin/geth --datadir /home/user98/Documents/github/eth-private-network/istanbul/istanbultestnet/node2 --syncmode 'full' --port 4802 --rpcport 9502 --rpc --rpcaddr '0.0.0.0' --rpccorsdomain '*' --rpcapi 'personal,db,eth,net,web3,txpool,miner,istanbul' --bootnodes 'enode://297e9ce0012765db9086cf3dd3efa793879b7010e7c38464ce0a05b8bcbb4babe0b1841ec68feaa9c65bfb93bb8e360f2a8d7de35eb250efa7c1b44c483bb2ee@127.0.0.1:4800' --networkid 1530 --gasprice '0' -unlock "0xaf40683bf20cc8b61e2a0d083a926fa66291de57" --password /home/user98/Documents/github/eth-private-network/istanbul/istanbultestnet/passwd.txt --debug --mine --minerthreads '1' --etherbase "0xaf40683bf20cc8b61e2a0d083a926fa66291de57"
INFO [08-08|13:07:40|cmd/utils/flags.go:856] Maximum peer count                       ETH=25 LES=0 total=25
INFO [08-08|13:07:40|node/node.go:166]       Starting peer-to-peer node               instance=Geth/v1.8.3-unstable-c7547381/linux-amd64/go1.9.3
INFO [08-08|13:07:40|ethdb/database.go:63]   Allocated cache and file handles         database=/home/user98/Documents/github/eth-private-network/istanbul/istanbultestnet/node2/geth/chaindata cache=768 handles=512
WARN [08-08|13:07:40|eth/db_upgrade.go:48]   Upgrading database to use lookup entries
INFO [08-08|13:07:40|eth/db_upgrade.go:118]  Database deduplication successful        deduped=0
INFO [08-08|13:07:40|eth/backend.go:123]     Initialised chain configuration          config="{ChainID: 1530 Homestead: 1 DAO: <nil> DAOSupport: false EIP150: 2 EIP155: 3 EIP158: 3 Byzantium: <nil> Constantinople: <nil> Engine: istanbul}"
INFO [08-08|13:07:40|eth/backend.go:146]     Initialising Ethereum protocol           versions=[64] network=1530
INFO [08-08|13:07:40|core/blockchain.go:253] Loaded most recent local header          number=0 hash=fc61f9…29aaac td=1
INFO [08-08|13:07:40|core/blockchain.go:254] Loaded most recent local full block      number=0 hash=fc61f9…29aaac td=1
INFO [08-08|13:07:40|core/blockchain.go:255] Loaded most recent local fast block      number=0 hash=fc61f9…29aaac td=1
INFO [08-08|13:07:40|core/tx_journal.go:149] Regenerated local transaction journal    transactions=0 accounts=0
INFO [08-08|13:07:40|p2p/server.go:395]      Starting P2P networking
INFO [08-08|13:07:43|p2p/discover/udp.go:238] UDP listener up                          self=enode://8b301efdb128529b753ed41cb184374492d1c5cfcc0bf7c94bb7df7aa17a81a73a3aa772f3bbedb767f52c114004fbd51bf928bd91c6b282b51b46683adec27e@146.198.136.74:4802
INFO [08-08|13:07:43|p2p/server.go:742]       RLPx listener up                         self=enode://8b301efdb128529b753ed41cb184374492d1c5cfcc0bf7c94bb7df7aa17a81a73a3aa772f3bbedb767f52c114004fbd51bf928bd91c6b282b51b46683adec27e@146.198.136.74:4802
INFO [08-08|13:07:43|node/node.go:327]        IPC endpoint opened                      url=/home/user98/Documents/github/eth-private-network/istanbul/istanbultestnet/node2/geth.ipc
INFO [08-08|13:07:43|node/node.go:397]        HTTP endpoint opened                     url=http://0.0.0.0:9502                                                                       cors=* vhosts=localhost
INFO [08-08|13:07:44|geth/accountcmd.go:218]  Unlocked account                         address=0xaF40683bF20cC8b61e2A0D083a926fA66291dE57
INFO [08-08|13:07:44|core/tx_pool.go:473]     Transaction pool price threshold updated price=0
INFO [08-08|13:07:44|miner/miner.go:116]      Starting mining operation
INFO [08-08|13:07:44|miner/worker.go:500]     Commit new mining work                   number=1 txs=0 uncles=0 elapsed=86.813µs
WARN [08-08|13:07:44|miner/agent.go:108]      Block sealing failed                     err=unauthorized
ERROR[08-08|13:07:54|consensus/istanbul/core/handler.go:141] Failed to decode message from payload    address=0x0a38a2061ADA40B1da28E9e9E2Bbc6F0c6d668b0 err="unauthorized address"
ERROR[08-08|13:08:06|consensus/istanbul/core/handler.go:141] Failed to decode message from payload    address=0x0a38a2061ADA40B1da28E9e9E2Bbc6F0c6d668b0 err="unauthorized address"
ERROR[08-08|13:08:20|consensus/istanbul/core/handler.go:141] Failed to decode message from payload    address=0x0a38a2061ADA40B1da28E9e9E2Bbc6F0c6d668b0 err="unauthorized address"
ERROR[08-08|13:08:38|consensus/istanbul/core/handler.go:141] Failed to decode message from payload    address=0x0a38a2061ADA40B1da28E9e9E2Bbc6F0c6d668b0 err="unauthorized address"
ERROR[08-08|13:09:04|consensus/istanbul/core/handler.go:141] Failed to decode message from payload    address=0x0a38a2061ADA40B1da28E9e9E2Bbc6F0c6d668b0 err="unauthorized address"
ERROR[08-08|13:09:46|consensus/istanbul/core/handler.go:141] Failed to decode message from payload    address=0x0a38a2061ADA40B1da28E9e9E2Bbc6F0c6d668b0 err="unauthorized address"
ERROR[08-08|13:11:00|consensus/istanbul/core/handler.go:141] Failed to decode message from payload    address=0x0a38a2061ADA40B1da28E9e9E2Bbc6F0c6d668b0 err="unauthorized address"

lutianzhou001 commented 6 years ago

have u solved this problem yet?

IronGauntlets commented 6 years ago

Yes, I discovered the problem is here, the authors force set the node key as mining ether-base then this means the validators added in the genesis are not responsible for creating a block, therefore I get "unauthorized address".

Please refer here for my hack (this is not a solution) around this problem.

fixanoid commented 5 years ago

Hi all. Please open new issues at the currently updated iteration of this repo located here: https://github.com/jpmorganchase/istanbul-tools/

abhayar commented 5 years ago

@aneequesafdar I'm getting same error. How did you exactly solve this issue..?

IronGauntlets commented 5 years ago

@abhayar please use this https://github.com/clearmatics/eth-private-network/tree/istanbul_private-network/istanbul

abhayar commented 5 years ago

@aneequesafdar Thanks,I refered the same. Now coinbase & eth.account[0] are same but still getting same error

IronGauntlets commented 5 years ago

The error must be something else, it would be helpful to see your trace

getamis / istanbul-tools

Istanbul network doesn't start with existing accounts which are used as the validator set in the genesis file #112

Steps to reproduce the behaviour

Backtrace