Implemented and deploy regression testing system be able to verify changes

[vogel76]

1. Initial test suite shall be generated as described in the issue #1986
2. Then automatic testing machinery (described in #1567) shall be used to continuously run such tests.
3. Dedicated massive verification shall be done for APIs which semantic is more complex (similary to #2039) for each API being subject of done changes.

[vogel76]

This task is blocked until following PRs are not merged: https://github.com/steemit/steem/pull/1989 https://github.com/steemit/steem/pull/1666

[mvandeberg]

Before we progress with full on regression testing of APIs, I want some answers to questions.

We want to record API queries that we see on live and results to use a test vectors. (#1986) However, it is unlikely that those test vectors will ever be useful because blockchain state changes so frequently. How are we going to ensure we can automatically dump meaningful test vectors?

The answer seems simple. We need to test against the same state. What happens when we have an update the requires reindexing? We will need to generate a new state file up to some specific block. However, it is likely that if we are changing reindex semantics that fields may be added/removed from objects, invalidating the tests. So how can we ever prove using this system that a new version passes?

The intent behind #1567 was to create a set of unit tests to test the API stack, not the results of the APIs themselves.

While I do see value in checking API results of services, such as Hivemind and steemd to ensure the new service is working properly, there are a number of bugs with the plugins that currently exist that we have no plans to fix because the plugin is effectively deprecated already. (ex #2025) Purposefully, the API results between Hivemind and Steemd will not match and it will take developer resources to maintain a deprecated plugin in order to do so.

There are a number of hurdles to make this type of regression testing useful without being undermined by its own complexity. What exactly is the use case that you see for this test suite?

[vogel76]

• tests must operate always on prepared instance (and reference instance) ie replied up to given block
• we plan to do dedicated regression tests mostly for things being reimplemented (so broken obsolete plugins really don't care us).
• if any complex test for dedicated scenario must be written, mentioned python tool allows it. But number of such tests will be always limited, just to cover corner cases. Even with produced pattern maintaining shall be not troublesome.
• since planned (persistent-layer) changes will modify just one layer, the logic done by new and old version of steemd must always match. No patterns, just runtime queries to compare live results from reference and testing instance
• initial set of generated tests (from calls done by condenser) is easy to maintain when changes are expected - just throw away and regenerate.

[mvandeberg]

This is being prioritized for the RocksDB Account History plugin to verify the implementation is correct.

We will discuss expanding this system to more APIs after this initial implementation.