Autogenerate Keep client accounts and configuration files

lispmeister commented 5 years ago

Context

When we want to deploy multiple Keep nodes to our local or GCS Ethereum testnet each node needs it's own Ethereum account with funding and it's own toml configuration file. We have two options to automate boostrapping these nodes after the Ethereum testnet is up:

Bake dedicated Docker images per Keep node after generating the accounts and config files.
Create a service that can provide a booting Keep node with it's funded accounts and config files via curl.

Threads

keep-client config discussion config automation approach discussion

Details

We would like the process for acquiring clients to be the same for local vs gcp.
We would like the request process for a client(s) to be configurable by N clients.
We would like the request process to include flags for boostrap-peer or not. (more on this below)
We would like an automated cleanup process that makes it easy to discard some deployed client topology.

Each keep-client needs a unique set of configuration values. For each deployed client we need to either dynamically create these values on a "one time use" keep-client docker image, or find a way to generate these values externally and inject them into a parameterized configuration on a pre-baked solution.

There are tradeoffs to each approach - which we can discuss in this issue.

A keep-client expects there to be an existing Ethereum testnet and an account with keep contracts loaded.

The following is required for each keep-client deployed:

An Ethereum account (unique to each client)
Ethereum account is unlocked
Ethereum account is funded
Ethereum account is staked
Access to the keyfile for the Ethereum account
A keep-client configuration file with the following
- Ethereum account address
- Path to Ethereum account keyfile
- Address or hostname for the Ethereum network the client is connecting to
- Contract Address for the KeepRandomBeacon
- Contract address for the KeepGroup

In addition to the above configurations we need to account for the following:

There are two types of keep peer that can be deployed, a bootstrap peer and non-bootstrap peer. Depending on which type of peer is being deployed the keep-client configuration will vary slightly.

For non-bootstrap peers you will need to include at least one bootstrap peer address, e.g.

[LibP2P]
  Peers = ["/ip4/172.17.0.2/tcp/3919/ipfs/16Uiu2HAm4dU3B5janAG34E5nYAdD6v75AMZuWKZpRybR2ieLzy4a"] 
  Port = 3920

For boostrap peers you will need to include some specific libp2p configurations, e.g.

[libp2p]
  Port = 27001
  Seed = 2

Sample keep-client configurations

Bootstrap

# This is a TOML configuration file for DKG, P2P networking and connction to Ethereum

# Provider Initialization Example

[ethereum]
    URL                = "ws://192.168.0.158:8546"
    URLRPC             = "http://192.168.0.158:8545"

[ethereum.account]
    Address            = "0xc2a56884538778bacd91aa5bf343bf882c5fb18b"
    KeyFile            = "/tmp/UTC--2018-03-11T01-37-33.202765887Z--c2a56884538778bacd91aa5bf343bf882c5fb18b"

[ethereum.ContractAddresses]
    KeepRandomBeacon = "0x639deb0dd975af8e4cc91fe9053a37e4faf37649"
    KeepGroup = "0xcf64c2a367341170cb4e09cf8c0ed137d8473ceb"

[libp2p]
    Port = 27001
    Seed = 2

non-bootstrap

# This is a TOML configuration file for DKG, P2P networking and connction to Ethereum

# Provider Initialization Example

[ethereum]
  URL                = "ws://10.102.100.190:8546"
  URLRPC             = "http://10.102.100.190:8545"

[ethereum.account]
  Address            = "0x20b613af41010cd631ad60087b1af40a197e4ca8"
  KeyFile            = "/tmp/UTC--2018-11-29T23-09-05.112323200Z--20b613af41010cd631ad60087b1af40a197e4ca8"

[ethereum.ContractAddresses]
  # Hex-encoded address of KeepRandomBeacon contract
  KeepRandomBeacon = "0xfb84b2458268f0cd28dacabe13f55c26f871b490"
  # Hex-encoded address of KeepGroup contract
  KeepGroup = "0xe5118b4836cd68a47ed5819840657d34e3fa6404"

# Non-bootstrap node connecting to an existing network.
[LibP2P]
  Peers = ["/ip4/172.17.0.2/tcp/3919/ipfs/16Uiu2HAm4dU3B5janAG34E5nYAdD6v75AMZuWKZpRybR2ieLzy4a"]
  Port = 3920

Keep token staking script

https://github.com/keep-network/keep-core/blob/master/contracts/solidity/scripts/demo.js

Brain dump on approaches (Mostly deprecated - leaving for history)

Rather than deal with seeding/funding/staking Ethereum accounts outside the scope of a keep-client configuration we should look to providing all required material as part of a request for a keep client. This would ensure that for any number of keep-clients requested were able to service the request.

A very mushy....but first came to mind idea would be to have a cloud function that on request generates an eth account / funds / stakes, then generates a config file for a single client. How we would get that into a client is still open. One idea Markus has was to shove all of that onto a volume we could mount to the container.

keep-client config approach

We can now create an and fund an arbitrary number of ethereum accounts in the eth testnet setup scripts.

Rather than create accounts on the fly, keep-clients will be attaching to one of a number of these pre-configured accounts. Eventually we'd like to move to a process where on keep-client deployment it goes and creates all the resources it needs.

At the outset (while number of eth accounts is low), we'll have a keep-client config stored in a config map, named for the eth account that is configured for this particular keep-client config.

In the kubernetes config for a client, there will be a configurable option for eth-account, which will identify which config map to pull and use.

This is rough, but will allow us to converge on a single keep-client image to deploy, which allows us to move to using CI built images and continuous deploy.

sthompson22 commented 5 years ago

I'm a bit fuzzy on the minimally viable setup to start testing. How many peers would we need? How many bootstrap peers vs regular ones?

sthompson22 commented 5 years ago

To be clear, I think we should make the keep contract population part of the ethereum setup. This means we need at least one account to be created with the testnet. Thinking that should be the only one though.

Shadowfiend commented 5 years ago

Rather than a cloud function, maybe a utility script/binary (perhaps wrapped in a container) that can spit out the config file + account file to stdin seems like it could be usable locally + in the cloud? Seems like that would be useful beyond just a docker-based testnet setup.

For testing I think we can start with 2-3 peers and just one bootstrap. Most important thing in this case is initializing the contracts with a group size that makes sense (most trivially, group size = number of peers). Then we can look at how we modify that for various scenarios, or in the cloud vs local, etc.

We would like the request process for a client(s) to be configurable by N clients.

What does this mean?

We would like an automated cleanup process that makes it easy to discard some deployed client topology.

Isn't just lighting the cluster on fire enough?

sthompson22 commented 5 years ago

What does this mean?

./spin-up-keep-clients -n 50

Where -n is number of clients

sthompson22 commented 5 years ago

Isn't just lighting the cluster on fire enough?

Probably? I really just wanted to give some high level goals in those initial bullets. Also depends on what you mean by cluster. We should be able to burn down the keep clients and eth-testnet in a cluster nuke.

Would there be cases where we want to leave the eth testnet up and only nuke the keep-clients? I would think so. This too is probably not much work, just wanted to call it out as a thing not to forget.

sthompson22 commented 5 years ago

For testing I think we can start with 2-3 peers and just one bootstrap. Most important thing in this case is initializing the contracts with a group size that makes sense (most trivially, group size = number of peers). Then we can look at how we modify that for various scenarios, or in the cloud vs local, etc.

Perfect thanks.

sthompson22 commented 5 years ago

Rather than a cloud function, maybe a utility script/binary (perhaps wrapped in a container) that can spit out the config file + account file to stdin seems like it could be usable locally + in the cloud? Seems like that would be useful beyond just a docker-based testnet setup.

Ya can see that. Was not firm on an approach at all.

Shadowfiend commented 5 years ago

Ok, some more stuff here:

Re: leaving testnet up and only nuking the clients, this would be nice but I don't think it's needed for an MVP. Having the initial pass be, “if you want fresh state, pour some gas on it and drop a match, then start over” is fine IMO.
Re: config files, I don't think the libp2p seed is going to be necessary anymore now that staking and network identities are based on Ethereum account info (I think). /cc @pdyraga
Re: funding/staking, I think we should give the images the tools to (a) generate Ethereum accounts, (b) request ETH and token funding in a defined way (as a reference, geth ships with an HTML-based faucet) and (c) stake. Then in a test environment we can run those tools in the base image, and in a production environment you just skip straight to running your node.

That said, I don't think that makes sense for the first pass. Instead, I think our first pass should start by having the base testnet setup fire up e.g. 5 accounts with set prefunded addresses and use those to feed any nodes that get booted up. We'd still need to handle token grants + staking, but that can maybe be a one-shot while booting the cluster? The goal here is first getting to a 4-5-node network as soon as possible, then getting to a 50-100-node network, then getting to an N-sized network, I think. The first one can be hardcoded everywhere, the second one can automate token granting/staking, and the last one can tackle the complexity of auto-generating and funding accounts, then token granting and staking.

Thoughts?

pdyraga commented 5 years ago

Re: config files, I don't think the libp2p seed is going to be necessary anymore now that staking and network identities are based on Ethereum account info (I think). /cc @pdyraga

The LibP2P.Seed is what currently makes the node bootstrap. If this is specified - node is a bootstrap one. If it's not specified - node is not a bootstrap one.

This is not ideal and we have it addressed in one of github issues but didn't have a chance to revisit.

sthompson22 commented 5 years ago

Re: leaving testnet up and only nuking the clients, this would be nice but I don't think it's needed for an MVP. Having the initial pass be, “if you want fresh state, pour some gas on it and drop a match, then start over” is fine IMO.

This feels reasonable, though we get some overhead on the backend without contract loading being automated as part of the process. Probably fine, it's already a script.

sthompson22 commented 5 years ago

Re: funding/staking, I think we should give the images the tools to (a) generate Ethereum accounts, (b) request ETH and token funding in a defined way (as a reference, geth ships with an HTML-based faucet) and (c) stake. Then in a test environment we can run those tools in the base image, and in a production environment you just skip straight to running your node.

This is the idea, but help my small brain. If this process is baked into the base image, for every client we would need to spin up an image yes? We were trying to avoid this.

sthompson22 commented 5 years ago

That said, I don't think that makes sense for the first pass. Instead, I think our first pass should start by having the base testnet setup fire up e.g. 5 accounts with set prefunded addresses and use those to feed any nodes that get booted up. We'd still need to handle token grants + staking, but that can maybe be a one-shot while booting the cluster? The goal here is first getting to a 4-5-node network as soon as possible, then getting to a 50-100-node network, then getting to an N-sized network, I think. The first one can be hardcoded everywhere, the second one can automate token granting/staking, and the last one can tackle the complexity of auto-generating and funding accounts, then token granting and staking.

:100:

Shadowfiend commented 5 years ago

we get some overhead on the backend without contract loading being automated as part of the process

I would say this is a better target for automation up front than the more complicated state-reset-without-burning-everything-down.

If this process is baked into the base image, for every client we would need to spin up an image yes?

Depends. If the entry point for the image is a script that generates an account + requests ETH/token + stakes, then boots up the client normally, you only need the one image. You could have two images, one with that entry point and one with a standard Keep client entry point (for more production-y envs).

pdyraga commented 5 years ago

There is one important aspect here. All those operations but funding can be done for each new client separately:

An Ethereum account (unique to each client)
Ethereum account is unlocked
~~Ethereum account is funded~~
Ethereum account is staked
Access to the keyfile for the Ethereum account
A keep-client configuration file with the following

The

Ethereum account is funded

is done with a genesis file initializing the chain and this chain initialization burns down previous configuration / data of the chain. It means, we need one place (Ethereum client image?) which will initialize the chain before clients start to stake their tokens.

lispmeister commented 5 years ago

Rather than a cloud function, maybe a utility script/binary (perhaps wrapped in a container) that can spit out the config file + account file to stdin seems like it could be usable locally + in the cloud? Seems like that would be useful beyond just a docker-based testnet setup.

@sthompson22 @Shadowfiend After some research I think the canonical way to provide each new Keep client pod with a Ethereum account and funds is to use what is called a CustomResourceDefinition and an associated manager that can provide account and funding on demand. Similar to how the cert-manager (https://github.com/jetstack/cert-manager) project provides valid certificates for HTTPS based services. That way we can just request funds in the yaml file of a Keep client pod. The manager will create and provide the equivalent of the following entry in a Keep client config file:

[ethereum.account]
    Address            = "0xc2a56884538778bacd91aa5bf343bf882c5fb18b"
    KeyFile            = "/tmp/UTC--2018-03-11T01-37-33.202765887Z--c2a56884538778bacd91aa5bf343bf882c5fb18b"

Shadowfiend commented 5 years ago

All those operations but funding

Don't forget the token grant. Once the token contract is loaded, it needs to grant tokens to particular accounts before they can stake.

That way we can just request funds in the yaml file of a Keep client pod.

Works for me, I would only add:

This bit shouldn't be a blocker or priority vs getting a small network running with hardcoded all the things.
You most likely need to handle account, funds, token, and staking. Making each of these steps optional isn't a terrible idea long term (for mainnet deployments), but isn't critical short term.
We need to think about how this is used outside of a k8s environment (i.e., make sure the config is easy to set up for someone providing external Ethereum key material). Again, that doesn't have to happen now.

keep-network / keep-core