Specify blockchain testing methods in RPC

[axic]

With the availability of various test tools, different methods for accomplishing the same testing features were implemented.

cpp-ethereum (complete list):

• test_setChainParams(genesis) - resets the blockchain to the provided genesis
• test_mineBlocks(n) - mine n blocks
• test_modifyTimestamp(time) - change timestamp
• test_addBlock(block)
• test_rewindToBlock(number) - revert to the block number

go-ethereum (complete list and probably outdated here):

• admin_importChain(path-to-file) - Imports the blockchain from a marshalled binary format. Note that the blockchain is reset (to genesis) before the imported blocks are inserted to the chain.
• admin_exportChain(path-to-file) - Exports the blockchain to the given file in binary format.
• admin_sleepBlocks(n) - Sleeps for n blocks.
• debug_setHead(number) - Sets the current head of the blockchain to the block referred to by blockNumber.

It has a lot more under debug.

parity: ?

testrpc:

• evm_snapshot: Snapshot the state of the blockchain at the current block. Takes no parameters. Returns the integer id of the snapshot created.
• evm_revert: Revert the state of the blockchain to a previous snapshot. Takes a single parameter, which is the snapshot id to revert to. If no snapshot id is passed it will revert to the latest snapshot. Returns true.
• evm_increaseTime: Jump forward in time. Takes one parameter, which is the amount of time to increase in seconds. Returns the total time adjustment, in seconds.
• evm_mine: Force a block to be mined. Takes no parameters. Mines a block independent of whether or not mining is started or stopped.

It would be nice defining a common set implemented by the clients.

[axic]

Similarities:

• go:admin_importChain equals in behaviour to cpp:test_setChainParams
• go:debug_setHead = cpp:test_rewindToBlock
• testrpc:evm_increaseTime = cpp:test_modifyTimestamp (though the latter can jump back in time too)
• testrpc:evm_mine= cpp:test_mineBlocks(1)

[axic]

Testeth filling process (for "state tests"):

0. retrieve client signature (name, version, etc?)
1. set genesis block
2. start chain with said genesis
3. submit 1 (or more) transaction
4. mine 1 block without proof of work
5. retrieve state result (state hash and list of all accounts)

[axic]

Some proposals from the call with @karalabe @winsvega @pirapira @cdetrio:

• Define a simple genesis format which can be implemented by all clients (to be used by setChainParams)
• Add an override option to addBlock which allows specifying header fields (used to create "invalid" blocks in "blockchain tests")
• Use this header to addBlock to change the timestamp

[axic]

Current genesis formats:

• Go: https://github.com/ethereum/go-ethereum#defining-the-private-genesis-state
• C++: https://github.com/ethereum/solidity/blob/develop/test/RPCSession.cpp#L219-L249

[axic]

Current work in progress changes from @winsvega (https://github.com/ethereum/cpp-ethereum/pull/4859):

• test_getClientInfo
• test_getPostState
• test_addTransaction

Also a more detailed list for all the methods porposed by @winsvega: https://github.com/ethereum/retesteth/issues

[winsvega]

the method formats are not final.

[pipermerriam]

We'd like to propose the following spec which I believe is a superset of all if not most of the functionality exposed by the endpoints so far discussed in this issue.

Use cases which are explicitly intended to be supported.

• External integration testing (e.g. web3 style libraries).
• Testing using the JSON fixture tests using RPC.
• Construction of JSON fixture tests using RPC.

Existing JSON-RPC endpoints

The test_ endpoints would be expected to be used in conjunction with the existing JSON-RPC endpoints, specifically, those in the eth_ namespace, but likely others such as web3_clientVersion. The test_ endpoints allow meta operations which cannot be performed over the traditional JSON-RPC APIs.

Anything like normal transaction sending, reading account balances or storage can and should still be done over the traditional endpoints.

Endpoints

• test_resetGenesis TBD define format for genesis params (use BlockchainTests format as base). -> wipes any chain state when called.
• test_importBlock or test_importRawBlock or test_importRLPBlock?????, does NOT bypass validation/verification.
• test_snapshot - Exact copy of State and Chain database up to last mined/finalized block. -> id
• test_revert - Revert to snapshot denoted by id. Discards any pending state like unmined transactions.
• test_mineBlock - bypass TBD header validation. extraData/coinbase/timestamp
• test_knownAccounts - returns enumerated list of account addresses.
• test_exportFixture - returns a BlockchainTest formatted json body.

test_resetGenesis

Resets the chain to its genesis state, or the genesis state provided in the JSON-RPC params.

• test_resetGenesis() -> resets to existing genesis state.
• test_resetGenesis(genesis_info) -> resets to the genesis state specified by genesis_info

We propose using the format from the BlockchainTest fixtures for the genesis_info. This should also allow specifying of the network parameter.

test_importBlock

Imports the given block to the chain. This does not bypass any chain validation rules so blocks must be valid according to the active fork rules.

• test_importBlock(block_data)

We propose block_data be the RLP encoded block as this is unambiguous, and allows for submission of invalid blocks to check that they are rejected.

test_snapshot and test_revert

test_snapshot would perform an exact copy of state and chain databases up to last mined/finalized block, returning an identifier which can be later used with test_revert to revert back to this saved snapshot.

We propose that any pending state like unmined transactions would not be preserved in these operations.

test_mineBlocks

Used for instant block mining. Accepts a list of header overrides (see below) and mines a block for each provided header, bypassing proof-of-work and some other validation checks (see below), allowing for instant on-demand mining.

• test_mineBlocks([{}]) -> mines a single block.
• test_mineBlocks([{}, {}, {}]) -> mines 3 blocks.
• `test_mineBlocks([{'timestamp': 12345}, {'timestamp': 67890}]) -> mines 2 blocks with the given timestamps.

We propose that the headers be supplied as a list of dictionaries. This endpoint should support the following fields to be overridden.

• timestamp
• coinbase
• extraData

When provided, timestamp field should bypass normal validation rules to allow for arbitrary modification of newly mined block's timestamps.

Implementations may choose to allow other fields to be overridden.

In addition, we suggest an optional extra parameter to allow newly mined blocks to be properly sealed with proof-of-work via CPU mining. This allows for generation of valid blocks which can later be exported to generate fixture tests.

test_knownAccounts

Returns a list of account addresses which are present in the state database.

The intention behind this endpoint is that while it is possible to check the account state against a known state over the JSON-RPC, if the current account state has extra entries, it will not be possible to discover what they are without the ability to enumerate all account addresses.

test_exportFixture

Returns a BlockchainTest formatted fixture.

[karalabe]

@axic @winsvega we've been discussing the above interface with @pipermerriam yesterday. This spec doesn't allow concurrent chains and does overwrite the current chain of the node (so it requires a custom flag to start). The benefit however is that it reuses the same RPC interface as everything else uses. That might be a good idea long term since we would have constant regression tests against the RPC, opposed to testing via an isolated API.

I'd still like to retain some ability to run potentially multiple chains in the same binary. Perhaps we could add some form of test_startChain / test_stopChain which configures a new in-memory testing instance and also opens up new RPC interfaces for it. That would allow us to retain the exact same functionality RPC wise but still support multiple concurrent tests. Not sure exactly how much work that would involve, just dropping it as an idea.

[winsvega]

Speaking of different chains. Another way is to start two instances of a client with different configs and ask one client to provide a block then import it into another. But this could be done within one instance with chain index of some kind. (guess its harder to implement)

I am currently fixing existing state tests and developing a minimum set of rpc methods required to fill a state test. the state test format will be slightly changed. some fields from env section might not be needed.

after this I plan to rework blockchain test filling through RPC.

[karalabe]

The issue with starting instances is the long boot time. Maybe it's "just" 2-3 seconds, but if you want to run repeating tests, those will add up. Would be imho nicer not to have to spin up entire processes all the time.

The only two options I see it is:

• Track "chain ids" via these methods, but then we can't use the normal RPC APIs for anything.
• Launch new RPC endpoints for each chain, a bit heavier, but we can use the existing RPC APIs.

A further problem with starting multiple processes is that then you either start a new process for each test to make it clean (but that's what hive does already, so there's no point); or you make some chain run in the host process and start secondary chains in new processes, but that will just be a sync/setup nightmare. I think we need to handle all test chains uniformly, so either all in the same process, or all in separate processes.

My fav for now is launching RPC endpoints.

[winsvega]

I was thinking to start different RPC. so multiple clients with multiple RPC sockets at the same time running tests in paralel. A setChainParams called once per test. if the config is not changed then rewindBlock method is used to disable transactions.

There should be options to optimize it. like use memoryDB for chain. starting the chain faster. cpp client has like 1-2 sec delay.

We have hive, but we don't have testeth anymore.

[folsen]

In Parity this isn't really how we do testing at all, which is why we don't have any test RPCs what-so-ever yet. I think the main worry from us is exposing test-stuff to the mainnet, so we'd probably try to hide these behind a compiler-flag so the default build doesn't include it at all.

The question then beyond that is just what we can do without significantly changing the codebase. We don't really want to code in a backdoor to skip block validation unless we can be 100% sure it can be excluded or at least isolated in such a way that we can guarantee that it can't be exploited.

Asked @tomusdrw about this and his off-the-top-of-his-head response was:

test_knownAccounts <- requires fat db test_exportFixture <- the format is pretty complex, might require some db-level capturing test_mineBlock <- means that we create some fake block without PoW and then disable seal checking in the entire client? test_snapshot <- seems fine test_revert <- will be troublesome in terms of caches test_resetGenesis <- nightmarish, since we load genesis state in constructors

With the additional comment that resetGenesis could probably just be done by restarting the client. There was even the idea floated to implement this by having a completely separate binary that just talks to parity to provide the abstraction layer (some stuff would probably still have to be added to parity itself).

This is all sort of speculation though until we actually try to do it.

[gnidan]

(cc @benjamincburns @seesemichaelj: we should follow along here for Ganache)