Implement a system testing environment that can be utilized for high-level black-box testing

[ggoranov]

Problem:

There is not a testing mechanism where a developer can automate the testing of issues like https://github.com/decred/dcrwallet/issues/1297

Proposal:

• A new testing environment would be able to automatically

  • Create from scratch a simnet of N nodes on any supported platforms. (Each dcrd node is nested into a docker container)
  • Specify all test configuration per each node.
  • Execute a set of actions
  • Accumulate and monitor all nodes traces
  • Perform hard-coded checks
  • Close the simnet

Advantages:

• Such an environment would help to
  • automate any complex use-cases to avoid regressions
  • demonstrate a running howto for any newly developed feature
  • (eventually) these tests execution can be integrated as a regular event in CI

Example:

Issue: dcrwallet/issues/1297

Pseudo code:

////////////////////////////////////////////////////
// Initialization 
simnet := NewSimnet( LINUX /*platform*/ )

wallet := createMiningWallet(simnet)
wallet.debugLevel  = "trace"

node := createSimnetNode(simnet, num)
    node.config.spv = false
    node.config.miningAddr = wallet.Addr

//NB By default any rpc credentials are handled by the environment

// Create additional 2 nodes
for num := 0: num < 2; num++ {
        node := createSimnetNode(simnet, num)
    node.config.spv = false
    node.config.debugLevel = "trace"
    }

// this will ..
//  create docker containers,
//  build dcrd/dcrdwallet from master or local branch
//  set configurations per node
//  Run all dcrd and dcrwallet instances

simnet.start();

////////////////////////////////////////////////////
// Definition

// Mine 1 block only (the  premine block )
simnet.mine( 1 /*node_id*/,  1 /*blocks count*/ )

// Check if no coinbase transaction was noticed by the wallet
CHECK_IS_TRUE( simnet.miningWallet.Balance() == 0 )

// Mine 1 blocks only 
simnet.mine( 1 /*node_id*/,  1 /*blocks count*/ )

// Check if second coinbase transaction was noticed by the wallet
CHECK_IS_TRUE( simnet.miningWallet.Balance() == COINBASE_REWARD )

////////////////////////////////////////////////////
//  Clean-up
simnet.stop();

[ggoranov]

Seems that similar idea was already implemented in https://github.com/btcsuite/btcsim. So not sure if it makes sense in the context of Decred. @davecgh Your input here would be helpful.

[davecgh]

I suggest taking a look at rpctest which already effectively provides the capability. It's also worth noting that @JFixby has been working to split it up into more reusable modules and clean it up a bit.

[davecgh]

For example, see the RPC server test file setup.

[ggoranov]

@davecgh Seems you've already made progress on that idea.

[JFixby]

I confirm I have something in progress in this direction.