Open h4ck3rk3y opened 1 year ago
Thanks for creating here!
Yup, the NodeID is derived from the staking key
The SubnetID is the hash of the transaction that created the Subnet. This means that if you have a wallet with 1 UTXO on the P-Chain that will fund a series of transactions, you can generate those transactions offline without issuing them and get a deterministic sequence of transactions, which will tell you the SubnetID and any BlockchainIDs that you may want to create for a test. We don't currently have example code for generating them offline, but the AvalancheGo P-Chain wallet is probably the best starting point: https://github.com/ava-labs/avalanchego/blob/master/wallet/chain/p/wallet.go#L26.
A subnet runs a set of blockchains (typically only one), and the node needs to have the VM necessary to run these blockchains available in order to validate the subnet. If you are just using Subnet-EVM, you should compile it and place it in the plugin directory before launching AvalancheGo. When you launch any subnet that runs Subnet-EVM, it will use the binary that is already in the plugin directory. If for some reason, you want to validate a new subnet that you don't have the plugin binary for when you started the node, you will need to restart to make it available to the node (good feature request would be hot reloading)
Yup
No, you need to restart it when either adding a new binary to plugin dir or when adding a new subnetID to tracked-subnets
@felipemadero could you provide context on what CreateBlockchains
is doing?
@aaronbuchwald that was super helpful!
2, 3 & 5 - I can try using the pre-computed SubnetIDs that the CLI & runner run with to get a v1 out and that can just be with subnet-evm for now; as I'll set tracked-subnets
before hand I wouldn't have to issue restarts just like the avalanche-cli
(based on my read of the code & behavior)
I'll read through the wallet code later and see if (or maybe there are docs by then) to generate the SubnetID
Will wait for @felipemadero to elaborate on what the CreateBlockchains
call does.
It is not currently possible to modify --track-subnets
on the fly. However it is possible to add new VMs that have been populated in the plugins
folder by using admin.LoadVMs. That being said, you will need to have loaded the VM before the CreateBlockchainTx
has been issued. Otherwise the chain will have been attempted to be started which will error with a VM not found error.
The full flow for launching a permissioned blockchain is:
If the goal is to launch a permissionless blockchain then:
If you want to pre-generate transactions, the easiest way will be to specify your own Client
implementation here. Specifically, you'll want to override the IssueTx call to keep the bytes that you need. If you want to make a PR to avalanchego to make this a more minimal interface (It only uses 2 of the many functions on the interface) then that would be a great change (but we would probably want to the same change to the X wallet.)
Hey Stephen! That's really thorough.
This makes sense! To makes sure I understand correctly; If I end up using a pre computed list of subnet ids; all I have to do is put the correctly named vm into the plugins
folder and then issue an admin.LoadVMs
call. This won't work if the subnet isn't tracked in the tracked-subnets
list?
This makes sense as well. I'll go further deep into it as I start implementing this and I will probably have more questions when I do so. In the meantime my understanding of the conversations so far; The sequence of operations is the following (assuming the subnet-evm) -
p.Wallet
CreateSubnetTx
foobar
even--tracked-subnets
& on restart the node will pickup the binaries in the plugins dirCreateBlockchainTx
from the same p.Wallet
created earlier; with the right VM ID, subnet ID, chainName (anything human readable). I'm not sure what fxIDs
are and perhaps I can leave them empty for now? I also have to pass a genesis.json for the state of the new chain thats being created. I can probably use this one here with right addresses for initial allocation. At this point the chain is running and the nodes that have the right vm & --tracked-subnets
are already validating the subnet.At this point I have a choice I can either use AddSubnetValidatorTxs or launch a permission less blockchain and then use AddPermissionLessValidatorTx (based on my understanding this is the path the CLI takes). I could stop too as I have a chain with some validators.
For permission less(elastic?) Blockchain
TransformSubnetTx
this takes in a lot of arguments. Are these sensible defaults?AddPermissionlessValidatorTx
. The signer is empty. I can take inspiration from here. These validators need to have tracked-subnets and the VM set too.For permissioned blockchain to add new validators
Some questions that I might not have asked above
avalanche subnet deploy
stops at step 7? And at step 7 I have a fully running subnet with enough validators? I need to make sure that I start with at least 5 nodes when I start it all.I'll have to decide whether I go the pre-generated transactions route or the route above. I am biasing towards generate on runtime route as I have dived deep enough into it I feel. If I do create a minimal interface I'll make sure I PR back to the repository.
Thanks for being so patient with me. I have been programming for years but I am fairly new to blockchain dev & Avalanche.
Hi Gyanendra
Regarding the original figured-out point 2: pre-computed subnet ids are used mainly for CLI implementation (for fast local subnet deploy). They belong to a network snapshot that comes packaged into CLI (https://github.com/ava-labs/avalanche-cli/blob/main/assets/bootstrapSnapshot.tar.gz), and are not available in a network started from scratch without using such snapshot.
I think there was a confusion on this point, as it is different what the CLI does (use a subnet id for a subnet that was already created and is contained in a network snapshot) from what a custom wallet enables to do (pre generate a subnet id without creating a real subnet, by simulating UTXO consumption and construct the tx without issuing it to the blockchain - as aaron mention in his comment to 2, and stephen extends in relation to pre-generating transactions)
Regarding the original questions:
1)
An avalanchego network needs to be restarted if:
In the specific case of adding a new vm, no need to restart as stephen already told, just need to call LoadVMs as network runner does in https://github.com/ava-labs/avalanche-network-runner/blob/80262bb1beafb633d7485dd8b57a9680ae8de9cc/local/blockchain.go#L307 (and I recommend following that code of installCustomChains to try to go deeper into what is needed to be done to set up a subnet)
2
The CreateBlockchains function of ANR creates the requested blockchains, each of one defined by the following blockchain parameters:
vm name
genesis
optional chain config (and optional per node specific chain config)
optional network upgrade
optional blockchain alias
optional subnet id
optional subnet parameters
optional list of nodes that will participate in the subnet
optional subnet config
Where:
If the subnet id is defined, then a blockchain is created (see createBlockchainsTxs) under that previously generated subnet (a subnet could had been previously generated by using network runner's CreateSubnets)
If the subnet id is not defined, a new subnet will we generated, will all network nodes as participants and no subnet config as default, unless optional subnet parameters are passed in. In the special case of receiving an optional list of nodes as participants where some of the nodes do not exist, first the new nodes will be added to the network, and then used as participants for the desired subnet. After all of that, the blockchain is created over the new subnet.
If blockchain config files, or network upgrade files are given, then they are first generared in the right locations, and then the network will be restarted previously to create the blockchains.
Regarding your second comment in the thread, yup you can try to use the network snapshot that comes with the CLI by calling LoadSnapshot on network runner, and then you will have those predefined subnet ids available and can call network runner's CreateBlockchains to create the subnets without restart (or issue a CreateBlockchainTx as Stephen told).
Regarding using a wallet to pre generate subnet ids: I think you probably don't want to first pre generate a subnet id by using a wallet, and you just want to create a subnet from scratch.
Hey Stephen! That's really thorough.
Node Configuration
This makes sense! To makes sure I understand correctly; If I end up using a pre computed list of subnet ids; all I have to do is put the correctly named vm into the
plugins
folder and then issue anadmin.LoadVMs
call. This won't work if the subnet isn't tracked in thetracked-subnets
list?
You can't just use a pre computed list of subnet ids, you can either use a network snapshot with already have subnet ids and tracked-subnets defined, or you need to do all the steps: create a subnet, restart the network with updated tracked-subnets, and then create the blockchain. LoadVMs will work to load the VMs on plugin dirs, but you need the subnet to be tracked for the vms start working.
Wallet issuance
This makes sense as well. I'll go further deep into it as I start implementing this and I will probably have more questions when I do so. In the meantime my understanding of the conversations so far; The sequence of operations is the following (assuming the subnet-evm) -
- I launch a private testnet and have some pre-funded accounts in that testnet. Make sure the nodes are validator nodes & are fully bootstrapped. I can try to use the default genesis.json(but I will have to copy the signer/staker keys) of the avalanche-network-runner or use this to generate a valid genesis JSON
- I transfer some AVAX from the C chain address to the pChain address of one of the genesis addresses
You have funds already available to a local network on PChain by using the EWOQ address as in network runner code
- Using the address from step 2 I create a
p.Wallet
- Using the wallet created in Step 3 I issue a
CreateSubnetTx
- I place the rightly named(VMID) binary of the chosen VM in the nodes of the testnet in the plugins folder that is tracked by the Avalanche Go Client. The VMID of the binary just has to match the VMID supplied and can be
foobar
even- I then restart the avalanchego nodes with the right id in
--tracked-subnets
& on restart the node will pickup the binaries in the plugins dir- I issue a
CreateBlockchainTx
from the samep.Wallet
created earlier; with the right VM ID, subnet ID, chainName (anything human readable). I'm not sure whatfxIDs
are and perhaps I can leave them empty for now? I also have to pass a genesis.json for the state of the new chain thats being created. I can probably use this one here with right addresses for initial allocation. At this point the chain is running and the nodes that have the right vm &--tracked-subnets
are already validating the subnet.
You can leave fxIDs empty for now. those are for feature extensions to be run by the vm. For genesis you can use a genesis created by CLI when locally deploying a subnet, to be found on ~/.avalanche-cli/subnets/SUBNET_NAME/genesis.json
At this point I have a choice I can either use AddSubnetValidatorTxs or launch a permission less blockchain and then use AddPermissionLessValidatorTx (based on my understanding this is the path the CLI takes). I could stop too as I have a chain with some validators.
You will not have any subnet validators until you issue an AddSubnetValidatorTx and you did not issue any in the previous points. It is not enough to set tracked-subnets, you need to issue an add subnet validator tx for each node you want to be a validator of the subnet.
For permission less(elastic?) Blockchain
Yup elastic
- I create a new asset on the X-Chain. I can perhaps take inspiration from here to get some of the values right
- I transfer some of that asset from the XChain to the P-Chain
- I issue a
TransformSubnetTx
this takes in a lot of arguments. Are these sensible defaults?- I issue a
AddPermissionlessValidatorTx
. The signer is empty. I can take inspiration from here. These validators need to have tracked-subnets and the VM set too.For permissioned blockchain to add new validators
- I issue a AddSubnetValidatorTxs; with a reasonable weight & the right subnet id. These validators need to have tracked-subnets and the VM set too.
Yon can also first issue an add subnet validator tx for those validators, and then restart then with proper tracked subnets and vms.
Some questions that I might not have asked above
avalanche subnet deploy
stops at step 7? And at step 7 I have a fully running subnet with enough validators? I need to make sure that I start with at least 5 nodes when I start it all.
avalanche subnet deploy
stops at step 7, but it previously added all 5 default local network nodes as subnet validators. it does that even before restarting the network.
I'll have to decide whether I go the pre-generated transactions route or the route above. I am biasing towards generate on runtime route as I have dived deep enough into it I feel. If I do create a minimal interface I'll make sure I PR back to the repository.
Probably you may want to just follow the route you described.
Thanks for being so patient with me. I have been programming for years but I am fairly new to blockchain dev & Avalanche.
Thank you so much @felipemadero
Your comments are very clear, insightful and super helpful. I think I have a fairly good understanding of how subnets launch now.
I'll try to have an n Node
subnet that runs on Docker/K8S out next week. I'll write about my learnings in the process. I'll reach out you folks in case I run into any troubles there but for now I feel equipped.
EWOQ addresses in the above post refer to the addresses here - https://github.com/ava-labs/avalanchego/blob/master/genesis/genesis_local.go#L18-L21 which come prefunded with a lot of tokens
Update from conversation with Aaron on Wednesday - the VMID is a cb58 hash and can be computed using either of
Hey!
@aaronbuchwald Thanks again for your time last week :)
I have added a bunch of stuff to the avalanche-package since and now this launches subnets! You can launch subnets with how many nodes you want using {"node_count": N}
; where N is the number of nodes you want. Otherwise it launches 5 nodes by default
The time for the docker run of avalanchego to expose 9651 and Kurtosis is very similar; around 12s on my machine.
I am tracking a lot of improvements that need to be made
srEXiWaHuhNyGwPUi444Tu47ZEDwxTWrbQiuD7FmgSAQ6X7Dy
on 1.10.1 to the new VMID in the same folder, but I reckon this would be different for 1.10.2; so have to remove this hardcoding and othersIn the meantime with Kurtosis installed you can give it a spin using
kurtosis run github.com/kurtosis-tech/avalanche-package '{"ephemeral_ports": false, "node_count": 5}'
This would spin up a 5 node cluster but you should be able to put a random number as well. By default this runs against Docker. I can send you instructions for K8S if you want to try that out too!
Note without the "ephemeral_ports": false
argument; the containers will get ephemeral ports so the chain-rpc-url in the end will be valid but only inside the enclave and not from the outside world. Working with product to print the public ephemeral port at the end without the ephemeral_ports
flag.
Thank you @felipemadero @StephenButtolph for your help too!
Hey! I have added a few more things
I am running into some weirdness though - up to 20 nodes its fine but if try starting a cluster with any more nodes there's some weirdness - here's the steps i take for permissioned subnets
kurtosis run . '{"node_count": 30}
')--tracked-subnets
With 30 nodes it(main.log) was initially failing with this when I did the vm installation after the after the AddSubnetValidatorTxs
but then I moved it to before the wallet transactions and that went away. Now it fails with this
From the main.log
[06-09|12:03:08.001] INFO chains/manager.go:319 creating chain {"subnetID": "2aPkmYuVqCBMGnyDBUsKExmD6SZa7JSH8sw8ekcsQ41nTj2qrp", "chainID": "NAL3xKipcGFTxKrzQZAtXStmZqN7QCqh2AZ88j9iZqmvtgtUA", "vmID": "tGBrM7iZGgNZvqPiwD9oD716rVRR9PiB6BFuG3ot3SP54ie8K"}
[06-09|12:03:13.322] ERROR chains/manager.go:349 error creating chain {"subnetID": "2aPkmYuVqCBMGnyDBUsKExmD6SZa7JSH8sw8ekcsQ41nTj2qrp", "chainID": "NAL3xKipcGFTxKrzQZAtXStmZqN7QCqh2AZ88j9iZqmvtgtUA", "chainAlias": "NAL3xKipcGFTxKrzQZAtXStmZqN7QCqh2AZ88j9iZqmvtgtUA", "vmID": "tGBrM7iZGgNZvqPiwD9oD716rVRR9PiB6BFuG3ot3SP54ie8K", "error": "error while creating vm: handshake failed: vm process not found"}
the stack trace says not committed
in the wallet
error occurred while adding validators: an error occurred while adding node '16' as validator: not committed
Do you know what might be going on here @aaronbuchwald @felipemadero ? This is running against avaplatform/avalanchego:v1.10.1-Subnet-EVM-master on Docker on my M1 Mac
Update - Alright I see where the error was coming from my logic here
While doing the issue validator transaction I had hardcoded a start time delay of 1 minute from the beginning of the loop. I think we'd run a transaction with a start timestamp in the past and that would fail.
If I setup startTime per node instead of having a global one that I use; I don't get the error anymore. Still having some troubles getting things into a healthy state as I run into
ID": "NAL3xKipcGFTxKrzQZAtXStmZqN7QCqh2AZ88j9iZqmvtgtUA", "chainAlias": "NAL3xKipcGFTxKrzQZAtXStmZqN7QCqh2AZ88j9iZqmvtgtUA", "vmID": "tGBrM7iZGgNZvqPiwD9oD716rVRR9PiB6BFuG3ot3SP54ie8K", "error": "error while creating vm: handshake failed: vm process not found"}
per node on restart but that seems different from the above issue. Will investigate. Maybe I should do add validator, restart validator per validator instead.
Alright! I was able to fix what was broken and the 100 node demo on k8s runs fine :) You will be able to specify how much memory + cpu something runs with
Here's the demo https://www.loom.com/share/f4e8bfeb4c304903b9b5f8736df0591d
This doc should get you started - https://docs.kurtosis.com/k8s/
@ashucoder9 Hi, can you please provide an update on this? Wondering if it can be closed.
Is there a document that you have that explains the process of subnet creation? All articles I can find on the internet use the avalanche network runner. I have seen some articles on ChainStack and others that refer to deprecated calls like
keystore.CreateUser
orplatform.CreateSubnet
I have been going through the avalanche-cli & the network runner and so far I have figured out the following
--tracked-subnets
. I am happy with using pre-computed SubnetIDs but I am curious if you can link me to the process of creating these SubnetIDsWhat I am still stuck on
Thanks @aaronbuchwald for suggesting I create an issue here!