[Research]: Persistent Segregated sBTC Testnet

[AshtonStephens]

Completing the issue description and arriving at a conclusion is the deliverable of this issue.

Research - Persistent Segregated sBTC Testnet

This ticket holds the research relating to creating and maintaining a persistent segregated sBTC Testnet and how it can be structured to be the most useful for sBTC-v1.

1. Summary

We will be setting up a Segregated Stacks Testnet with the following components in AWS:

• Bitcoin Follower with Electrum server
• Stacks Node
• Mempool Frontend
• Mempool Backend
• Hiro API
• Stacks Explorer

The longer-term scope of this Testnet is to be a "CI/CD" style deployment where updates & new features will be automatically integrated. Ideally, as new code is pushed into Github, our Testnet should automatically pull & integrate changes.

In the medium term, however, we will have to manually update the Stacks code versions in the Testnet (e.g "v1.0.0" -> "v1.0.1").

The DevOps code will be written in AWS CDK and possibly docker-compose for specific components, if necessary.

The Design Document of this will be linked here shortly: TODO()

2. Context & Relevance

The release of Nakamoto and the eventual future release of sBTC will require us to stress-test the validity and resilience of our code; connecting other feature sets (outlined in section [1] above) to our Stacks code is critical to ensuring overall correctness.

We want to possess our own dedicated testnet for this specific reason.

This discussion will only pretain to an Individual Node Setup for now (no HA / Autoscaling). By default, we will be running a Signer alongside our Stacks Node in order to be able to run sBTC with our latest Nakamoto release.

3. Research

3.1.0 Proposed Research Conclusions

Note: The full detailed breakdown of this will be covered in the Design Document linked above. This is an abridged version

We will run each one of the components in Section [1], as much as we can, in one node each. The node distribution is outlined below:

Group No#	Instance Type	Specs	Optimization	Workload(s)
1	c7i.4xlarge	16vCPU, 32Gb Memory	Compute	Bitcoin Follower, Mempool Backend, Mempool Frontend, Electrum Server
2	r5.xlarge	4vCPU, 32Gb Memory	Compute	Stacks Node w/ Signer
3	r5.large	4vCPU, 16Gb Memory	Memory	Hiro API
4	t3.xlarge	2vCPU, 16Gb Memory	General	Stacks Explorer

#

Each single instance will be equipped with its own security group in order to only expose ports which are absolutely necessary (like exposing the P2P port for the Stacks Node while keeping the Stacks Signer ports publicly inaccessible )

#

3.1.1 Possible Insights on sBTC behavior from Running this Testnet

This Testnet could reveal or point towards the following:

• During withdrawal, how well will sBTC handle Bitcoin forks
• What if our .sbtc contract calls get stuck in the Bitcoin mempool
• If we increase the amount of Signers we run to the sBTC v1 scenario of 7, then how long will a withdrawal really take?
• Being able to see the following Signer metrics (https://github.com/Trust-Machines/sbtc-v1/issues/35):
  • Display of BTC pegged into the Stacks ecosystem (TVL).
  • Tracking unique addresses holding sBTC.
  • Comprehensive transaction history for sBTC.
  • Monitoring minting and burning of sBTC tokens.
• The performance of the Deposit API (we haven't outlined the specific metrics for this)
• If Replace-by-fee (RBF) properly works
• Handling the event of a Signer's key being compromised
• Testing out double spend attacks (DSA), and denial of service (DOS) (https://github.com/Trust-Machines/sbtc-v1/issues/23) and observing if sBTC can detect malicious behavior
• Simulating coordinator failure (periodic VRF w/ constant selection)

3.2 External Resources

Electrum: https://github.com/Blockstream/electrs?tab=readme-ov-file Mempool Backend: https://github.com/mempool/mempool/tree/master/backend Mempool Frontend: https://github.com/mempool/mempool/tree/master/frontend Stacks Core: https://github.com/stacks-network/stacks-core Hiro API: https://github.com/hirosystems/stacks-blockchain-api Stacks Explorer: https://github.com/hirosystems/explorer

3.3 Areas of Ambiguity

• If the Deposit("Signer") API a separate service, should we run it in its own AWS Instance?
• Should we also run another seperate instance as a Stacks Signer so that we can test out the .sbtc withdrawal flow as outlined in https://github.com/Trust-Machines/sbtc-v1/discussions/16
• Other than what is discussed in https://github.com/Trust-Machines/sbtc-v1/discussions/35, what other specific metrics will the Signers emit?
• How will we be stress testing our sBTC code? What will be the criteria for success/failure?
• Roughly, what will sBTC really lean into? (high cpu, high mem etc.??)
• Maybe we should run these programs as docker containers instead of a direct install, since it becomes super easy to automate version changes (possibly integrate into ArgoCD)
• If we do use Docker containers, then we should really look into Kubernetes since it really allows us to run the ArgoCD CI/CD tool

---

Closing Checklist

• [ ] The takeaway from this issue is clearly documented in the description of this ticket.
• [ ] Everyone necessary has reviewed the resolution and agrees with the takeaways.
• [ ] This ticket has or links all the information necessary to familiarize a contributor with the topic and how it was resolved.

[AshtonStephens]

Among other details, this issue should identify the most useful details that the segregated testnet would help identy during development and the way that the system should generally be structured to achieve that.

Once that's done we'll create a design ticket that will either follow the regular format or point to an external design doc.

[hstove]

Is it simpler if we stand up a Bitcoin testnet node / electrum / etc, and then use that in our segregated testnet? Then, we can re-use that Bitcoin node for other things if needed. Maybe what I'm saying is basically what is outlined here.

[hstove]

Maybe we should run these programs as docker containers instead of a direct install, since it becomes super easy to automate version changes (possibly integrate into ArgoCD)

One challenge here is that it's often not possible to do a simple "reboot" when there are consensus-breaking changes. For example, if we change the contracts, we need to fully restart the environment. We might not want to do that for every change. But if we only want to do that for a "beta release" branch, CI can make sense.

[radicleart]

my thoughts for reference;

1. is there a possibility to re-use the docker regtest environment that was developed for sbtc testing last October - this was a similar set of containers/nodes configured on regtest.
2. integration testing on regtest with docker is essential
3. manual testing on an environment that is as close as possible to mainnet is also important
4. i'd like to know (from core nakamoto team) how helpful the interim nakamoto testnet turned out to be as a basis to inform this work ?

[AshtonStephens]

4. i'd like to know (from core nakamoto team) how helpful the interim nakamoto testnet turned out to be as a basis to inform this work ?

Working on lockstacks you can either use the hosted testnet OR use a local environment. Using a Testnet is a lot easier if you're testing flows that take time.

[SundarGowtham]

It seems that we would use this persistent testnet not only to see/test some time or traffic related scenarios regarding sBTC, but also validating some integration tests running for longer.

• For example we can test out the full flow (end-to-end) of Deposit & Withdraw and see if it works properly longer term
• Checking Correctness of Deposit API
• Simulate multiple Signers

But during these flows some very key operational metrics were suggested to be measured:

• Memory and CPU consumption when idle vs when initating a signing round
• Wall clock time
• The amount of messages exchanged between the signers
• Time taken as as a function of signers [f(s) with s = No# of signers]
• Time taken for a block signing round
• DKG time
• How well does sBTC operate during possible simulated outages
• No# of requests halting/getting rejected for no exact reason
• Simulate attacks like DOS or Minting Attacks

Please feel free to add more key behaviors or metrics

[netrome]

In addition to this, it's very important for us to have a working devnet and integration testing environment available for us to be able to develop effectively. We've discussed this a few times in our calls, so I have created this issue to track it:

https://github.com/Trust-Machines/sbtc-v1/issues/46

Would love to discuss and prioritize this. I believe there are many potential synergies between the devnet setup and a persistent testnet.

[hstove]

I would agree to prioritizing devnet more than peristent testnet, especially earlier on in development.

One problem with a persistent testnet for testing high traffic is that it's harder to set up and re-run the environment. For example, if you're writing some kind of performance test, you need to have a bunch of accounts setup - this is easier to do with a "bootup" script than to need to modify the accounts on persistent testnet. You might also want to simulate different block times, which isn't something we'd want to do on-the-fly in a persistent testnet, because you might mess up someone else that is using it.

That last point maybe sums it up - if you're trying to test something that will mess up other users on the testnet (like moving the majority of funds), a devnet is better.

Also, if you have those kind of tests on a devnet, it's very easy to write an integration test for it. You wouldn't want to target integration tests on a persistent testnet for the same reasons written above.

[AshtonStephens]

We moved off of a persistent segregated sBTC Testnet and are going with the local dev environment first. Closing as not planned but we may one day come back to this.

Thanks @SundarGowtham!