search

stacks-network / stacks-core

The Stacks blockchain implementation
https://docs.stacks.co
GNU General Public License v3.0
3.01k stars 664 forks source link

Property Testing for PoX-2 Contract #3845

Open kantai opened 1 year ago

kantai commented 1 year ago

As a first step for getting property testing in place for boot contracts, I think using the known-issues from PoX-2 (which precipitated the 2.1-2.4 forks) forms a quantifiable target for baseline property tests of the pox-2 contract (which could then be applied to pox-3). You can think of this like a meta-regression test: a bug is known, that should have been caught, but instead of a "normal" regression test, where you just add a unit test or likewise that covers the scenario, this issue is about creating property tests that would surface the issue in a reasonable amount of time.

The PoX-2 Bug

Ultimately, the stacks-increase bug and the also-addressed "swimming in multiple pools" bug are smart contract bugs. They should be caught using techniques that are applicable to any smart contracts. Expanding tests like the contract_tests module in the stacks-blockchain repo or the hirosystems/stacks-2-1-testing tests (which use Stacks DevnetJS) should be able to catch these bugs. However, those tests are all hand written. Instead, the style of testing that should be applied here is something along the lines of property testing (or quickcheck).

There are multiple options for applying property testing, but the most promising is using clarinet's normal test execution combined with tools that @moodmosaic started working on which enable property testing for clarity contracts.

The first hurdle for this approach is that it requires altering Clarinet's handling of "special" contracts like pox-* so that lockups can be sufficiently tested. This PR (https://github.com/hirosystems/clarinet/pull/1074) achieves this in Clarinet, so as long as clarinet can be built with that, progress on this issue could be made.

moodmosaic commented 1 year ago

module in the stacks-blockchain repo or the hirosystems/stacks-2-1-testing tests (which use Stacks DevnetJS)

@kantai

kantai commented 1 year ago

What is the advantage of stacks-devnet-js over plain Clarinet.test?

stacks-devnet-js runs an actual stacks-node, pointed at a bitcoin regtest network, meaning that its closer to a "real" setting. From clarity's perspective, this mostly means that the get-burn-info methods return more realistic data. But it also means that if you want to write end-to-end tests for integrations with the stacks-node's RPC endpoints or test actual transactions, you can do that. Plain clarinet.test is instead just directly invoking the clarity VM and simulating the bitcoin information.

What scenarios can be tested with stacks-devnet-js that can't be tested via Clarinet.test?

For PoX, you can test the actual assignment of slots and the "auto-unlock if stacking threshold isn't met" mechanism via devnet-js, but you cannot via Clarinet.test. Without my branch of Clarinet, you used to be unable to test the locking behavior of PoX, but with my branch, you're now able to test the locking behavior in plain Clarinet.

For sBTC, you can test interactions with regtest bitcoin transactions via devnet.js (similar to testing on testnet), but you couldn't do that in plain Clarinet.

moodmosaic commented 1 year ago

@kantai, thank you for your comments. I looked into the stacks-e2e-testing repo but run into a few issues (linked above).

kantai commented 1 year ago

Sorry for the slow response @moodmosaic. I think that when it comes to the property testing, it would be best to try to use the clarinet runtime itself (using the branch that supports pox locks), rather than stacks-devnet-js. stacks-e2e-testing used devnet-js because the branch that supported pox locks didn't exist at the time. You should be able to look at the tests in stacks-e2e-testing to help write tests that use the normal clarinet testing environment. The tests in this directory are relevant to PoX: https://github.com/hirosystems/stacks-e2e-testing/tree/main/tests/integration/pox

moodmosaic commented 1 year ago

to help write tests that use the normal clarinet testing environment

Those tests seem to use devnet-js, here for example. What is the ultimate goal? To be able to just use Clarinet.test?

kantai commented 1 year ago

Yes, all the tests in that repo use devnet-js -- I linked those tests so you can see some of the ways that the PoX contract has been tested today. The ultimate goal of this project is to write property tests for the PoX-2 contract (which those linked tests aren't) using the clarinet test environment.