Manage Bitcoin/Omni Core nodes within bitcoin-spock (start and stop daemon)

[dexX7]

Sooner or later we won't get around it, I believe, and have to handle node statup internally.

This became very clear when I tested @zathras-crypto's UI branch with two seperated QT clients, resulting in two manual starts and a lot of switching between manual commands and spock tests.

[msgilligan]

That's what the com.msgilligan.bitcoin.BitcoinDaemon class was intended to be a step towards. But if you google around there are lots of warnings and complaints about subprocess management from the JVM.

[dexX7]

Ah, great hint, thanks. The Groovy process management also looks nice to handle. I quickly did a search and there are indeed some negative comments. Might still be worth a try or would you consider it as a lost cause?

[msgilligan]

It's definitely worth a try. But I don't relish the idea of spending a bunch of time on it and then deciding it is a lost cause.

Personally, I would focus on finding the best Java solution and then add Groovy syntax sugar if it helps. For example there is Apache Commons Exec and jnr-process from @headius.

Another possibility would be to use the Gradle Exec task though that would not allow us to set things up on test-by-test basis.

[dexX7]

Ah, thanks for the refererence. Let's start thinking on a high level. Let me quickly show schematically how the Core RPC test framework works by some example code mixed with some pseudo code and comments, so the idea is more clear.

The base framework:

class BitcoinTestFramework(object):
    """Base class for all tests"""

    def setup_chain(self):
        """Initializes datadirs for n nodes, creates config files, ...

        Usually either from a copy of a cache where some blocks are pre-mined or 
        where pristine datadirs are created. As per default, those "working" dirs
        are temporary and stored in /tmp/testXXXX"""
        pass

    def setup_network(self):
        """Starts processes and connects nodes"""
        pass

    def run_test(self):
        """Test entry point, to be overwritten be the actual test"""
        pass

    def main(self):
        """Program entry point"""
        # startup argument handling

        try:
            self.setup_chain()
            self.setup_network()
            self.run_test()

        except Exception as e:
            # handle exceptions, sets exit status flag, extra logging, ...
            pass

        # shutdown nodes

        if not self.options.nocleanup:
            # clean datadirs
            pass

        # set exit status, ...

An actual test:

class SimpleCoinbaseExampleTest(BitcoinTestFramework):

    def setup_chain(self):
        """Overwrites base method and handles initialization for 1 fresh node"""
        initialize_chain_clean(self.options.tmpdir, 1)

    def setup_network(self, split=False):
        """Overwrites base method and starts 1 node with an extra argument"""
        extra_args = [['-debug=logtimestamps']]
        self.nodes = start_nodes(1, self.options.tmpdir, extra_args)
        # no inter-connection of nodes required, because we only have one node

    def run_test(self):
        """Simple demonstration of a test to show coins must be 100 blocks old
        to be spendable"""

        # mine 200 blocks
        self.nodes[0].setgenerate(True, 200)

        # check that there are 200 blocks
        assert_equal(self.nodes[0].getblockcount(), 200)

        # check that there are 5000 spendable BTC
        assert_equal(self.nodes[0].getbalance(), 5000.0)

Using the test from the command line, say for example from ~/bitcoin/qa/rpc-tests where the bitcoind binary is available as ~/bitcoin/src/bitcoind:

./coinbase_example_test.py --srcdir=../../src

The base class provides some defaults, for example in Core 4 temporary nodes are created, connected and each pre-mines 50 blocks.

This example should not serve as blueprint, but provides an overview of where this could go.

I suggest we start with a minimal version that is able to simply start a node, running in server mode when executing the test suite, and stops the node when finished with the tests. Subclasses for regtest, mainnet, and maybe testnet would then further specify additional configuration options such as regtest=1.

Pretty similar to what we already do here in integ/groovy/foundation/omni/BaseRegTestSpec.groovy, but with minimal refinements and a layer on top or as part of, that replaces the startup bash scripts, and moves the process and datadir creation inside the test framework.

I believe this should be feasible, and hope the example didn't create the impression of implying a major refactoring or extension of the current test framework.

It basically comes down to getting rid of the bash scripts by doing it "in house" as first step.

[msgilligan]

BTW, @casey-bowman has created casey-bowman/pb-regtest which has 3 Groovy classes which are used to start and stop a Bitcoin Core instance.

His deleteRegtest() method deletes the RegTest directory before every test Spec execution. (Casey, we're currently doing this with Bash scripts that run before the entire suite of tests.)

Note also that his BitcoinServer.post() method implements an RPC client in about 25 lines of code. He's using groovyx.net.http.HTTPBuilder which provides a convenient API, JSON mapping, etc. Whereas our core client is pure Java and has no dependencies besides the JDK and Jackson (for JSON mapping) and is designed to be lightweight enough for Android. We have also focused on providing strongly typed RPC wrapper methods and have provided error handling. But it's cool to see how easy it is to talk to JSON-RPC with a simple Groovy solution. Nice work, Casey.

@casey-bowman make sure to look at our DynamicRPCClient.groovy which uses methodMissing to implement any RPC method dynamically. (We are not using this in our tests as we are trying to use/implement/test strongly-typed method wrappers, but provided this subclass for reference and completeness. We may also make it a Groovy trait so it can be added to any RPCClient subclass)

[msgilligan]

Note that Java 9 (now in release candidate stage, I believe) includes a significantly improved process API: http://openjdk.java.net/jeps/102

We could try using that.

[headius]

I never noticed this mention of jnr-process back then, but it still does provide the simplest path for true native process control on *nix. Even the Java 9 ehnancements really only give you access to pids and process trees, and still don't give you native, selectable Channel implementations for the stdio streams, making it incredibly cumbersome to drive interactive subprocesses.

Happy to include features into jnr-process if someone wants or can help out (Win32 CreateProcess support would be a big plus).

[msgilligan]

Thanks for the update @headius !

We've been managing with bash scripts and CI YAML files to start the Bitcoin Core / Omni Core daemon.

I'm thinking that using Docker (possibly with Test Containers) might be the best solution for this project's functional/integration testing needs (particularly since Docker allows integration testing of the Linux daemon on macOS, Windows, etc and handles installation and configuration of the daemon)

A jnr-process implementation for Bitcoin Core would also be useful for many users and now that ConsensusJ is split off as a separate library -- as a more general-purpose Bitcoin (and derivates) JVM library -- it would make sense to implement a jnr-process implementation over there.