[Feature Requests] geth --dev --fork

[BlinkyStitt]

There are so many DeFi projects now that getting them all setup inside a devnet is complicated and very fragile.

A pattern that is growing in popularity is using ganache-cli --fork. This lets us deploy our own contracts and test transactions in a private test network built on top of another network's state (usually mainnet).

While ganache-cli --fork has been a great help in getting my tests running quickly, over the last few weeks I have been running into several bugs in ganache that are making it difficult to fully test my contracts. Being a production client, I think that having a --fork flag in geth would be a lot more reliable way to test things.

Expected behaviour

There are a few ways that I can see this working, but one way could look something like this:

geth --fork mainnet@$BLOCKNUM --fork-unlock ANYADDRESS --fork-fund SOMEADDRESS

This would launch a node that fast syncs mainnet up to block $BLOCKNUM. Once it reaches that block, it hard forks to switch to a POA archive node with itself as the only signer. Blocks could either be mined on a timer or mined whenever a pending transaction is received.

It should also be easy to connect a second geth node to this forked node. This will make it simple to test various load-balancing strategies.

The switch to POA (or some other non-POW mining) is important since following mainnet's difficulty would make mining very difficult.

The node should also unlock any address and accept transactions for them without valid signatures. Similar to funding accounts in genesis.json, --fork should also be able fund addresses with configurable amounts of ETH. This will let us modify the mainnet contracts however we might need for our testing.

Actual behaviour

It is currently possible to start geth with --dev, but that is a fresh chain without any mainnet state.

I don't see any way to import mainnet blocks into this --dev network.

Also, if anyone can think of a way to do this today without having to modify geth, please let me know.

[holiman]

This sounds like several features. I haven't used ganache myself for years, so I'm not sure what --fork does more in-depth, but this is what I can read out from your descriptions:

1. This would launch a node that fast syncs mainnet up to block $BLOCKNUM

That would be neat, but unfortunately that's not doable, due to the nature of fast-sync. It might be possible, in some circumstances, if you happen to find an archive node which has old state. Otherwise, fast-sync syncs to whatever is the 'highest' block among your peers, and even that is not stable -- we have to move the 'pivot' point while the sync is ongoing, since our peer is pruning state as time goes by, and we can't rely on state older than 128 blocks to be available.

For full-sync, this is trivial though.

2. Once it reaches that block, it hard forks to switch to a POA archive node with itself as the only signer.

That's a legit feature-request, switching from PoW to PoA. Not supported today.

3. It should also be easy to connect a second geth node to this forked node. This will make it simple to test various load-balancing strategies.

Not a feature-request, this should already be possible.

4. The node should also unlock any address and accept transactions for them without valid signatures.

That's not doable today, and I don't believe it's a feature we'd accept. That means we'd need to have a special type of transaction RLP-encoding where we encode the sender also, which today is never encoded anywhere (it's derived from the signature). The change would touch a lot of core areas, and add lots of complexity.

5. Similar to funding accounts in genesis.json, --fork should also be able fund addresses with configurable amounts of ETH. This will let us modify the mainnet contracts however we might need for our testing.

I guess that could be done, but it's a non-trivial feature to add. Especially if it should be done in a way that other nodes (parity/besu et al) want's to be able to partake, then we'd need a cross-client consensus definition of how such a distribution of ether should happen. Sounds easier to just add a faucet and do the disbursements from that faucet. We'd still have to "insert" a faucet at block N, but that is only one address.

[BlinkyStitt]

This looks similar to what I'm doing: https://github.com/trufflesuite/ganache-core/issues/526#issuecomment-591367152

[BlinkyStitt]

Thanks for the reply!

This sounds like several features.

Yes, this is definitely a very large request. But I think the benefits of using a production client for development and testing make it worthwhile. I also didn't want to open multiple interconnected issues until we figured out exactly what this would look like. One place to talk this through seemed best for now.

1. This would launch a node that fast syncs mainnet up to block $BLOCKNUM

That would be neat, but unfortunately that's not doable, due to the nature of fast-sync. It might be possible, in some circumstances, if you happen to find an archive node which has old state. Otherwise, fast-sync syncs to whatever is the 'highest' block among your peers, and even that is not stable -- we have to move the 'pivot' point while the sync is ongoing, since our peer is pruning state as time goes by, and we can't rely on state older than 128 blocks to be available.

Luckily, I have an archive node; Using ganache-fork against a fast synced node starts throwing errors after 128 blocks.

My main goal was the automatic switching. I'd like a developer to only have to run one geth command, wait a short while, and then be able to develop.

1. Once it reaches that block, it hard forks to switch to a POA archive node with itself as the only signer.

That's a legit feature-request, switching from PoW to PoA. Not supported today.

I was hoping this could use some existing code for the ETH 2.0 switch, but that works differently and so won't help us.

For now, I'm looking into recompiling geth to have no difficulty to mine blocks.

1. It should also be easy to connect a second geth node to this forked node. This will make it simple to test various load-balancing strategies.

Not a feature-request, this should already be possible.

Possible yes, but I want to make sure it's easy and well documented.

1. The node should also unlock any address and accept transactions for them without valid signatures.

That's not doable today, and I don't believe it's a feature we'd accept. That means we'd need to have a special type of transaction RLP-encoding where we encode the sender also, which today is never encoded anywhere (it's derived from the signature). The change would touch a lot of core areas, and add lots of complexity.

That's too bad. I wonder how ganache is doing their --unlock, since they allow unlocking any account.

1. Similar to funding accounts in genesis.json, --fork should also be able fund addresses with configurable amounts of ETH. This will let us modify the mainnet contracts however we might need for our testing.

I guess that could be done, but it's a non-trivial feature to add. Especially if it should be done in a way that other nodes (parity/besu et al) want's to be able to partake, then we'd need a cross-client consensus definition of how such a distribution of ether should happen. Sounds easier to just add a faucet and do the disbursements from that faucet. We'd still have to "insert" a faucet at block N, but that is only one address.

A faucet would work, too. I just need some way to have a bunch of ETH for testing. For now, I think I'll just mine a bunch of low difficulty blocks and use the mining rewards.

[holiman]

ganache's reason for existence is to allow testing of advanced contract flows and events. Whereas geth's primary reason is to sync ethereum mainnet. If it can be used to do testing, that's great, but it's not the primary focus.

These features would be great, but we cannot implement all of this in geth -- it's too complicated and interferes with some core functioning of geth.

I'm not sure what (if anything) we can do on our end to make lives easier for dapp=devs who wants to use geth.

[holiman]

While ganache-cli --fork has been a great help in getting my tests running quickly, over the last few weeks I have been running into several bugs in ganache that are making it difficult to fully test my contracts. Being a production client, I think that having a --fork flag in geth would be a lot more reliable way to test things.

I think it would be better to focus on getting ganache to work, as they're already pursuing this feature.

[krasi-georgiev]

@holiman does this still stand difficult with the current codebase? I use hardhat forks, but would be much nicer if I can use go libraries for the tests

Currently in the tests I spin a hardhat fork using exec.Command but this is fragile. It would be nicer to have something like

import github.com/ethereum/go-ethereum/fork

client:= fork.New()

https://github.com/cryptoriums/packages/blob/main/hardhat/hardhat.go#L33

[root0x0]

any information about this feature. It's really convenient for testing geth new feature.

[ChinW]

Same idea, how about simulated backend, will this achieve same effect within cost-effectively steps?

[trajan0x]

tbh, rpc wrapper would be great for simulated backend