Add persistent data options

[wileyj]

Use the following template to create your pull request

Description

This PR adds the ability of stacks-blockchain to use a persistent data directory at the default location of ./persistent-data/stacks-blockchain/<mode>, and ./persistent-data/postgres. Other changes are simplifying the sample.env file, standardizing some env vars and removing unused vars. All changes are documented in the Readme Lastly - this PR adds the ability to import v1 bns-data (currently not enabled). This involves downloading historical v1 data from a static location and extracting the files that are used to store the data in postgres via the API. This required a change to the docker-compose run script, so there is now a shell script to download/verify the files (further work can potentially be done to verify the sha256 of each file) before starting the API

tagging @MarvinJanssen for visibility from the foundation.

Type of Change

• [x] New feature
• [ ] Bug fix
• [ ] API reference/documentation update
• [x] Other

Does this introduce a breaking change?

Sort of - restarting postgres/stacks-node will cause a full resync of all data. However, afterwards all restarts should be much faster since the data will be persistent across reboots in the ./persistent-data directory. Note that the ./persistent-data/bns-data will also need to be downloaded and extracted to import V1 data (not currently active in mainnet) - this step adds some extra time to the initial startup process (as long as these files are not deleted, they won't be re-downloaded).

Are documentation updates required?

Changes documented here: https://github.com/blockstack/stacks-local-dev/blob/mainnet-persist/README.md

[wileyj]

ping @MarvinJanssen

[wileyj]

Looks good to me apart from the one point regarding mounting files.

Why do we create separate branches per mode: mocknet/testnet/mainnet + persist / non-persist? We could also split it over multiple compose files.

docker-compose -f mainnet.yaml up

docker-compose -f testnet.yaml up

Or even layer them:

docker-compose -f mainnet.yaml -f persist.yaml up

Etc. Let me know what you think. I can make those changes. We can define separate envs or another context switch to keep the same level of operability without having to manage a bunch of branches.

Definitely could do that - but that would also require specific changes to the .env files. You're kind of getting into the whole reason this repo exists - it wasn't originally supposed to do what it's used for today. it was designed literally for local mocknet dev work. it had to be repurposed quickly to provide mainnet features.

the next update, perhaps we can work together on consolidating. the idea of relying on the end-user to use the correct files on startup i think wouldn't be the best, but pehaps a wrapper could make this easier, i.e. ./launch testnet

[MarvinJanssen]

Definitely could do that - but that would also require specific changes to the .env files. You're kind of getting into the whole reason this repo exists - it wasn't originally supposed to do what it's used for today. it was designed literally for local mocknet dev work. it had to be repurposed quickly to provide mainnet features.

the next update, perhaps we can work together on consolidating. the idea of relying on the end-user to use the correct files on startup i think wouldn't be the best, but pehaps a wrapper could make this easier, i.e. ./launch testnet

Totally! It kind of evolved beyond that, maybe it would be worth renaming the repo at some point.

I like the idea of a wrapper script that will invoke the right docker-compose files. The composability makes it very versatile. (Not to mention docker-compose.override.yaml.) You could also separate the env files via compose and give each mode their own.

Contrived example for mocknet.yaml:

  follower:
    image: ${STACKS_IMAGE}
    container_name: ${STACKS_FOLLOWER_NAME}
    env_file:
      - common.env
      - mocknet.env

Regarding networks/DNS, those will be inherited by compose files down the chain, where the ones on the right override any parameters (if present). We can also explicitly set a network name.

It can be worth it to have a call and discuss the different ways we could go about it before we start consolidating for the next update.

[wileyj]

Definitely could do that - but that would also require specific changes to the .env files. You're kind of getting into the whole reason this repo exists - it wasn't originally supposed to do what it's used for today. it was designed literally for local mocknet dev work. it had to be repurposed quickly to provide mainnet features. the next update, perhaps we can work together on consolidating. the idea of relying on the end-user to use the correct files on startup i think wouldn't be the best, but pehaps a wrapper could make this easier, i.e. ./launch testnet

Totally! It kind of evolved beyond that, maybe it would be worth renaming the repo at some point.

I like the idea of a wrapper script that will invoke the right docker-compose files. The composability makes it very versatile. (Not to mention docker-compose.override.yaml.) You could also separate the env files via compose and give each mode their own.

Contrived example for mocknet.yaml:

  follower:
    image: ${STACKS_IMAGE}
    container_name: ${STACKS_FOLLOWER_NAME}
    env_file:
      - common.env
      - mocknet.env

Regarding networks/DNS, those will be inherited by compose files down the chain, where the ones on the right override any parameters (if present). We can also explicitly set a network name.

It can be worth it to have a call and discuss the different ways we could go about it before we start consolidating for the next update.

Sure! you have my email, we can schedule there