search

stacks-network / stacks-blockchain-docker

Stacks-blockchain with API using docker compose
GNU General Public License v3.0
27 stars 37 forks source link

Private testnet not working #43

Closed dorucioclea closed 3 years ago

dorucioclea commented 3 years ago

Hi there, I'm trying to use your setup for private testnet but unfortunately it is not working.

Initially I had an issue related to a wallet not being created which I fixed by adding this container execution: 

bitcoind-createwallet:
    image: "byrnedo/alpine-curl"
    depends_on:
      - bitcoin
    restart: on-failure
    command: |
      -f -u blockstack:blockstacksystem --data-binary '{"jsonrpc": "1.0", "id": "c", "method": "createwallet", "params": ["testwallet"]}' -H 'content-type: text/plain;' http://bitcoin:18443/
    networks:
      - stacks-blockchain

What this does is create a wallet called testwallet in order to avoid the wallet not being created error which this previously gave me

Next, I slightly modified the bitcoin configuration to look like this

server=1
regtest=1
disablewallet=0
rpcallowip=0.0.0.0/0
rpcallowip=::/0
rpcuser=blockstack
rpcpassword=blockstacksystem
txindex=1
listen=1
rpcserialversion=0
rpcworkqueue=100

[regtest]
bind=0.0.0.0:28444
rpcbind=0.0.0.0:28443
rpcport=28443

Notably, the only change I did was to add disiablewallet=0

Now I'm getting the error below:

image

A couple of questions here:

Thank you

wileyj commented 3 years ago

We'll keep this simple to start with - I can confirm the wallet issue on my end, so we'll focus on that first. I'll test some potential fixes and get back to you, and then we can discuss the remainder of your questions - the wallet issue is definitely the most important here.

This network is also something relatively new we added, born from an internal hackathon - so it's definitely a work in progress and we're still ironing things out.

pavitthrap commented 3 years ago

Regarding "We are using the puppet-chain version of bitcoin node but I noticed that this is not the version used for the stack-blockchain-api scripts", could you share the version of bitcoin core that is being used? Should look like the following line in the logs: bitcoin | 2021-06-08T15:47:41Z Bitcoin Core version v0.20.99.0-831675c8d (release build)

For versions of bitcoin core earlier than 0.21.0, a default wallet should automatically be created (so I wouldn't expect you would need a call to create wallet at all).

edit: I was incorrect, by version 0.20.99.0, a default wallet was not being created automatically. The master branch has a commit updating the stacks-follower-node to be a later version (which is compatible with bitcoin version 0.21.0) (https://github.com/blockstack/stacks-local-dev/commit/075a9ac10415470ae16dd6c3d098f3b06a8f1771)

dorucioclea commented 3 years ago

Thank you for the quick response. Not at the computer atm, will provide a more detailed log tomorrow 👌

wileyj commented 3 years ago

Regarding "We are using the puppet-chain version of bitcoin node but I noticed that this is not the version used for the stack-blockchain-api scripts", could you share the version of bitcoin core that is being used? Should look like the following line in the logs: bitcoin | 2021-06-08T15:47:41Z Bitcoin Core version v0.20.99.0-831675c8d (release build)

For versions of bitcoin core earlier than 0.21.0, a default wallet should automatically be created (so I wouldn't expect you would need a call to create wallet at all).

The version here should be identical since it's using a docker image

pavitthrap commented 3 years ago

In that case, logs for the wallet not being created would be helpful (so removing the call to create wallet & the disable wallet line).

dorucioclea commented 3 years ago

Sure. So I'm running exactly what's on master rn, no modifications.

Bitcoin node log: 

2021-06-09T12:33:02Z Bitcoin Core version v0.20.99.0-831675c8d (release build)

Bitcoin starting logs:

2021-06-09T12:33:02Z Bitcoin Core version v0.20.99.0-831675c8d (release build)

2021-06-09T12:33:02Z Validating signatures for all blocks.

2021-06-09T12:33:02Z Setting nMinimumChainWork=0000000000000000000000000000000000000000000000000000000000000000

2021-06-09T12:33:02Z Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation

2021-06-09T12:33:02Z Using RdSeed as additional entropy source

2021-06-09T12:33:02Z Using RdRand as an additional entropy source

2021-06-09T12:33:02Z Default data directory /root/.bitcoin

2021-06-09T12:33:02Z Using data directory /root/.bitcoin/regtest

2021-06-09T12:33:02Z Config file: /etc/bitcoin/bitcoin.conf

2021-06-09T12:33:02Z Config file arg: listen="1"

2021-06-09T12:33:02Z Config file arg: regtest="1"

2021-06-09T12:33:02Z Config file arg: rpcallowip="0.0.0.0/0"

2021-06-09T12:33:02Z Config file arg: rpcallowip="::/0"

2021-06-09T12:33:02Z Config file arg: rpcpassword=****

2021-06-09T12:33:02Z Config file arg: rpcserialversion="0"

2021-06-09T12:33:02Z Config file arg: rpcuser=****

2021-06-09T12:33:02Z Config file arg: rpcworkqueue="100"

2021-06-09T12:33:02Z Config file arg: server="1"

2021-06-09T12:33:02Z Config file arg: txindex="1"

2021-06-09T12:33:02Z Config file arg: [regtest] bind="0.0.0.0:28444"

2021-06-09T12:33:02Z Config file arg: [regtest] rpcbind=****

2021-06-09T12:33:02Z Config file arg: [regtest] rpcport="28443"

2021-06-09T12:33:02Z Command-line arg: conf="/etc/bitcoin/bitcoin.conf"

2021-06-09T12:33:02Z Command-line arg: datadir="/root/.bitcoin"

2021-06-09T12:33:02Z Command-line arg: debuglogfile=false

2021-06-09T12:33:02Z Command-line arg: pid="/run/bitcoind.pid"

2021-06-09T12:33:02Z Using at most 125 automatic connections (1048576 file descriptors available)

2021-06-09T12:33:02Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements

2021-06-09T12:33:02Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements

2021-06-09T12:33:02Z Script verification uses 15 additional threads

2021-06-09T12:33:02Z scheduler thread start

2021-06-09T12:33:02Z WARNING: the RPC server is not safe to expose to untrusted networks such as the public internet

2021-06-09T12:33:02Z HTTP: creating work queue of depth 100

2021-06-09T12:33:02Z Config options rpcuser and rpcpassword will soon be deprecated. Locally-run instances may remove rpcuser to use cookie-based auth, or may be replaced with rpcauth. Please see share/rpcauth for rpcauth auth generation.

2021-06-09T12:33:02Z HTTP: starting 4 worker threads

2021-06-09T12:33:02Z Using wallet directory /root/.bitcoin/regtest/wallets

2021-06-09T12:33:02Z init message: Verifying wallet(s)...

2021-06-09T12:33:02Z init message: Loading banlist...

2021-06-09T12:33:02Z ERROR: DeserializeFileDB: Failed to open file /root/.bitcoin/regtest/banlist.dat

2021-06-09T12:33:02Z Invalid or missing banlist.dat; recreating

2021-06-09T12:33:02Z SetNetworkActive: true

2021-06-09T12:33:02Z Using /16 prefix for IP bucketing

2021-06-09T12:33:02Z Cache configuration:

2021-06-09T12:33:02Z * Using 2.0 MiB for block index database

2021-06-09T12:33:02Z * Using 56.0 MiB for transaction index database

2021-06-09T12:33:02Z * Using 8.0 MiB for chain state database

2021-06-09T12:33:02Z * Using 384.0 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)

2021-06-09T12:33:02Z init message: Loading block index...

2021-06-09T12:33:02Z Switching active chainstate to Chainstate [ibd] @ height -1 (null)

2021-06-09T12:33:02Z Opening LevelDB in /root/.bitcoin/regtest/blocks/index

2021-06-09T12:33:02Z Opened LevelDB successfully

2021-06-09T12:33:02Z Using obfuscation key for /root/.bitcoin/regtest/blocks/index: 0000000000000000

2021-06-09T12:33:02Z LoadBlockIndexDB: last block file = 0

2021-06-09T12:33:02Z LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=0, size=0, heights=0...0, time=1970-01-01...1970-01-01)

2021-06-09T12:33:02Z Checking all blk files are present...

2021-06-09T12:33:02Z Initializing databases...

2021-06-09T12:33:02Z Pre-allocating up to position 0x1000000 in blk00000.dat

2021-06-09T12:33:02Z Opening LevelDB in /root/.bitcoin/regtest/chainstate

2021-06-09T12:33:02Z Opened LevelDB successfully

2021-06-09T12:33:02Z Wrote new obfuscate key for /root/.bitcoin/regtest/chainstate: 75d6e28d0a728167

2021-06-09T12:33:02Z Using obfuscation key for /root/.bitcoin/regtest/chainstate: 75d6e28d0a728167

2021-06-09T12:33:02Z init message: Rewinding blocks...

2021-06-09T12:33:02Z block index 72ms

2021-06-09T12:33:02Z Opening LevelDB in /root/.bitcoin/regtest/indexes/txindex

2021-06-09T12:33:02Z Opened LevelDB successfully

2021-06-09T12:33:02Z Using obfuscation key for /root/.bitcoin/regtest/indexes/txindex: 0000000000000000

2021-06-09T12:33:02Z txindex thread start

2021-06-09T12:33:02Z txindex is enabled

2021-06-09T12:33:02Z txindex thread exit

2021-06-09T12:33:02Z loadblk thread start

2021-06-09T12:33:02Z UpdateTip: new best=0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206 height=0 version=0x00000001 log2_work=1.000000 tx=1 date='2011-02-02T23:16:42Z' progress=1.000000 cache=0.0MiB(0txo)

2021-06-09T12:33:02Z block tree size = 1
2021-06-09T12:33:02Z nBestHeight = 0

2021-06-09T12:33:02Z Failed to open mempool file from disk. Continuing anyway.

2021-06-09T12:33:02Z loadblk thread exit

2021-06-09T12:33:02Z torcontrol thread start

2021-06-09T12:33:02Z Bound to 0.0.0.0:28444

2021-06-09T12:33:02Z Bound to 127.0.0.1:18445

2021-06-09T12:33:02Z init message: Loading P2P addresses...

2021-06-09T12:33:02Z ERROR: DeserializeFileDB: Failed to open file /root/.bitcoin/regtest/peers.dat

2021-06-09T12:33:02Z Invalid or missing peers.dat; recreating

2021-06-09T12:33:02Z ERROR: DeserializeFileDB: Failed to open file /root/.bitcoin/regtest/anchors.dat

2021-06-09T12:33:02Z 0 block-relay-only anchors will be tried for connections.

2021-06-09T12:33:02Z init message: Starting network threads...

2021-06-09T12:33:02Z dnsseed thread start

2021-06-09T12:33:02Z net thread start

2021-06-09T12:33:02Z 0 addresses found from DNS seeds

2021-06-09T12:33:02Z addcon thread start

2021-06-09T12:33:02Z init message: Done loading

2021-06-09T12:33:02Z opencon thread start

2021-06-09T12:33:02Z dnsseed thread exit

2021-06-09T12:33:02Z msghand thread start

Bootstrapping chain

INFO: detected DYNAMIC_GENESIS_TIMESTAMP, will set the genesis timestamp to 1623241984

2021-06-09T12:33:04Z CreateNewBlock(): block weight: 896 txs: 0 fees: 0 sigops 400

2021-06-09T12:33:04Z Pre-allocating up to position 0x100000 in rev00000.dat

2021-06-09T12:33:04Z Leaving InitialBlockDownload (latching to false)

2021-06-09T12:33:04Z UpdateTip: new best=000bc41da11a396eb3c2a891e6bb52cf4bda0f393e4e970ef7770404898d44b2 height=1 version=0x20000000 log2_work=2.000000 tx=2 date='2021-06-09T12:33:04Z' progress=1.000000 cache=0.0MiB(1txo)
................................

Stacks-node-follower error:

stacks-node-follower    | ERRO [1623242134.265502] [testnet/stacks-node/src/burnchains/bitcoin_regtest_controller.rs:619] [main] Bitcoin RPC failure: error listing utxos Network("Bitcoin RPC: status(500) != success, body is \'Object({\"error\": Object({\"code\": Number(-18), \"message\": String(\"No wallet is loaded. Load a wallet using loadwallet or create a new one with createwallet. (Note: A default wallet is no longer automatically created)\")}), \"id\": String(\"stacks\"), \"result\": Null})\'")

Furthermore, stacks-blockchain-api logs:

{"message":"Pg connection failed, retrying in 2000ms..","level":"error","timestamp":"2021-06-09T12:33:04.159Z"}

{"message":"Stacks 1.0 token offering data import started","level":"info","timestamp":"2021-06-09T12:33:07.050Z"}

{"message":"Stacks 1.0 token offering data import completed","level":"info","timestamp":"2021-06-09T12:33:08.908Z"}

{"message":"Notice: full BNS functionality requires 'BNS_IMPORT_DIR' to be set.","level":"warn","timestamp":"2021-06-09T12:33:08.909Z"}

{"message":"Event observer listening at: http://0.0.0.0:3700","level":"info","timestamp":"2021-06-09T12:33:08.916Z"}

Notice how the proxy and api (port 3999) is not starting

dorucioclea commented 3 years ago

As shown above, after the stacks-node-follower errors out, it will remain in this state, as it cannot move past the missing wallet issue. My understanding is (correct me if I'm wrong please) that newer bitcoin builds stopped creating the (test) wallet by default

There are a few recent interesting issues on bitcoin's repository related to this, one of them being:

Furthermore, I noticed that on stacks-blockchain-api the bitcoin version used in their docker-compose is:

"ruimarinho/bitcoin-core:0.19.1-alpine"

Included permalink to their docker-compose setup:

https://github.com/blockstack/stacks-blockchain-api/blob/cba848585f2d372345aca9a35ae984b5dd7ab6d3/docker-compose.dev.bitcoind.yml#L4

Interestingly, they do not use this as they have a docker-compose override file which will always override the stacks configuration to mocknet.

Is there a reason we're using this particular bitcoin docker image and are not aligned with the devops scripts from stacks-blockchain-api and/or stacks-blockchain

For example, stacks-blockchain has a different set of configurations for local:

https://github.com/blockstack/stacks-blockchain/tree/master/testnet/stacks-node/conf

pavitthrap commented 3 years ago

To address the wallet issue - for me, running the following commands on master (no further modifications) work: ./manage.sh private-testnet pull ./manage.sh private-testnet up If you haven't already tried it, could you reset to the head of master and try these commands (just so we can both be using the exact same config/ image)

dorucioclea commented 3 years ago

To address the wallet issue - for me, running the following commands on master (no further modifications) work: ./manage.sh private-testnet pull ./manage.sh private-testnet up If you haven't already tried it, could you reset to the head of master and try these commands (just so we can both be using the exact same config/ image)

I'm on latest version but will do another docker pull just to be safe when I get back to a computer.

Another quick question : when you run the testnet does the api start (port 3999) with the condigs on master?

Thank you 👌

wileyj commented 3 years ago

Is there a reason we're using this particular bitcoin docker image and are not aligned with the devops scripts from stacks-blockchain-api and/or stacks-blockchain

yes! the docker-compose you're referencing is very outdated (and really shouldn't be used at this point). the version of bitcoin used in this repo is the same version that we use internally at Hiro.

As for the configs, what's included here are samples based on what's in the stacks-blockchain repo. We try to keep them more or less identical, but it's impossible to keep them as 1:1 duplications.

dorucioclea commented 3 years ago

Alrignt, thank you ! 🔥

So, got back to the computer and did

./manage.sh private-testnet pull (which does docker-compose -f ./configurations/common.yaml -f ./configurations/private-testnet.yaml pull)

I was already up-to-date, latest changes from the master branch, no changes whatsoever on my part. I have the same error.

At this point, the only thing that could differ in your local stack are the configs. Could you please check that the 3 configuration files are up-to-date on master?

My configuration files versions are as follow:

Thank you 😄

wileyj commented 3 years ago

sorry for the delayed responses here....been dealing with a lot of sponsored tx's i've had to resend for the btc domains. Once I work through that list, i'll circle back and devote more time here

dorucioclea commented 3 years ago

sorry for the delayed responses here....been dealing with a lot of sponsored tx's i've had to resend for the btc domains. Once I work through that list, i'll circle back and devote more time here

Hey, no worries, and thank you for your help 😄 sorry for the avalanche of questions on my side but I'm currently learning stacks/clarity and want to have a clear picture of it all + a completely-ish private devenv so I could run my tests and interact with contracts from the explorer ui 🔥

dorucioclea commented 3 years ago

Is there a reason we're using this particular bitcoin docker image and are not aligned with the devops scripts from stacks-blockchain-api and/or stacks-blockchain

yes! the docker-compose you're referencing is very outdated (and really shouldn't be used at this point). the version of bitcoin used in this repo is the same version that we use internally at Hiro.

As for the configs, what's included here are samples based on what's in the stacks-blockchain repo. We try to keep them more or less identical, but it's impossible to keep them as 1:1 duplications.

Thank you for sheding some light on the subject. So:

Thank you 👌

wileyj commented 3 years ago

Yes, this repo can be considered the "source of truth" for orchestration. @MarvinJanssen and myself do our best to keep it updated with stacks-blockchain/stacks-blockchain-api changes as well (if something in the configs change, we do our best to get them here).

I can say that the configs in the source repos have been a low priority, and what you see in this repo is based on the running systems at Hiro and what's in the base repos. ideally, they would be close to identical - but there will always be some drift.

For now, i can confirm that using blockstack/stacks-blockchain image other than blockstack/stacks-blockchain:feat-miner-control will result in the bitcoin wallet issue (this was the branch we created during hackathon to address that error, and the resulting image is what we found to work. not sure where that branch went but i'll work with @pavitthrap to look into it).

to keep this message short, i'll reply again with some steps i'd like for you to take to try and resolve or possibly to get more data to troubleshoot here.

wileyj commented 3 years ago

First, ensure you have the latest code from the master branch. either git clone again, or do a pull and stash/remove any local changes. (if you've modified any of the .toml files, best to remove them or rename them to .toml.save).

Next, we'll remove any images locally that we'll be using:

docker rmi blockstack/stacks-blockchain:feat-miner-control
docker rmi blockstack/stacks-blockchain:latest
docker rmi blockstack/explorer:latest
docker rmi blockstack/bitcoind:puppet-chain
docker rmi postgres:alpine
docker rmi blockstack/stacks-blockchain-api:latest

Next, ensure nothing else is occupying the expected ports:

$ nc -vz localhost 20443
nc: connectx to localhost port 20443 (tcp) failed: Connection refused
nc: connectx to localhost port 20443 (tcp) failed: Connection refused
$ nc -vz localhost 20444
nc: connectx to localhost port 20444 (tcp) failed: Connection refused
nc: connectx to localhost port 20444 (tcp) failed: Connection refused
$ nc -vz localhost 3999
nc: connectx to localhost port 3999 (tcp) failed: Connection refused
nc: connectx to localhost port 3999 (tcp) failed: Connection refused
$ nc -vz localhost 3000
nc: connectx to localhost port 3000 (tcp) failed: Connection refused
nc: connectx to localhost port 3000 (tcp) failed: Connection refused

Next, pull the images we'll be using:

$ ./manage.sh private-testnet upgrade
Running: docker-compose -f ./configurations/common.yaml -f ./configurations/private-testnet.yaml pull 
Pulling bitcoin              ... done
Pulling postgres             ... done
Pulling stacks-node-api      ... done
Pulling explorer             ... done
Pulling stacks-node-follower ... done

Start the services:

$ ./manage.sh private-testnet up
Running: docker-compose -f ./configurations/common.yaml -f ./configurations/private-testnet.yaml up -d
Docker Compose is now in the Docker CLI, try `docker compose up`

Creating network "stacks-blockchain" with driver "bridge"
Creating postgres ... done
Creating bitcoin  ... done
Creating explorer        ... done
Creating stacks-node-api ... done
Creating stacks-node-follower ... done
Brought up private-testnet, use './manage.sh private-testnet logs' to follow log files.

$ docker ps
CONTAINER ID   IMAGE                                             COMMAND                  CREATED              STATUS              PORTS                                                                   NAMES
43e45bdb8058   blockstack/stacks-blockchain:feat-miner-control   "/bin/stacks-node st…"   About a minute ago   Up 44 seconds       0.0.0.0:20443-20444->20443-20444/tcp, :::20443-20444->20443-20444/tcp   stacks-node-follower
34602f89cf6d   blockstack/stacks-blockchain-api:latest           "docker-entrypoint.s…"   About a minute ago   Up About a minute   0.0.0.0:3999->3999/tcp, :::3999->3999/tcp                               stacks-node-api
df48e1e7521b   blockstack/explorer:latest                        "docker-entrypoint.s…"   2 minutes ago        Up About a minute   0.0.0.0:3000->3000/tcp, :::3000->3000/tcp                               explorer
e2c35cd3dfa9   postgres:alpine                                   "docker-entrypoint.s…"   2 minutes ago        Up About a minute   5432/tcp                                                                postgres
4069f6e8f0ed   blockstack/bitcoind:puppet-chain                  "/sbin/tini -- /dock…"   2 minutes ago        Up 2 minutes        18443/tcp, 28444/tcp

And finally, check the logs of all containers: ./manage.sh private-testnet logs

You should see log entries like this after a bit:

...
bitcoin                 | 2021-06-11T21:38:38Z [default wallet] AddToWallet ca38081dda807fef8d7652750d9c0c450ff7788cc8ad798bff577cb04a63b638  new
bitcoin                 | Assembled block 338. Will be accepting incoming transactions for the next 7000ms, then assemble block 339.
bitcoin                 | 2021-06-11T21:38:45Z CreateNewBlock(): block weight: 904 txs: 0 fees: 0 sigops 400
bitcoin                 | 2021-06-11T21:38:45Z UpdateTip: new best=1804475290894b8cfc1fbd9cf03d62a973764fa018689e38316c4f72627b532f height=338 version=0x30000000 log2_work=9.405141 tx=339 date='2021-06-11T21:38:45Z' progress=1.000000 cache=0.0MiB(338txo)
bitcoin                 | 2021-06-11T21:38:45Z [default wallet] AddToWallet 63d190ac0934b5fb933d90963cb02266d76a56980427835b3a95e8fb72f8127c  new
stacks-node-follower    | INFO [1623447525.845309] [src/chainstate/stacks/index/marf.rs:1185] [main] Batching insertions in MARF: 23% (77127 out of 335331)
stacks-node-follower    | INFO [1623447527.917987] [src/chainstate/stacks/index/marf.rs:1185] [main] Batching insertions in MARF: 24% (80480 out of 335331)
...
wileyj commented 3 years ago

@pavitthrap I was able to reproduce the wallet issue using the stacks-blockchain image master and develop tag, let's discuss what's going on there.

wileyj commented 3 years ago

@dorucioclea - the code from that feature branch image has been merged into develop, but the image for develop was out of date. i've just updated the image, will give it a shot using that one shortly.

@pavitthrap do we have an idea of when those changes will be merged into main ?

dorucioclea commented 3 years ago

Hey, thank you for the good news @wileyj I will give it a try with the develop image then 😄

dorucioclea commented 3 years ago

I see blockstack/stacks-blockchain:develop was updated 13 hours ago so will use that one and let you know. Thank you 🎉

pavitthrap commented 3 years ago

@wileyj I would expect this commit to make it into the next release candidate (so probably next month sometime).

dorucioclea commented 3 years ago

@wileyj @pavitthrap I tried with both blockstack/stacks-blockchain:develop and blockstack/stacks-blockchain:feat-miner-control with the config files currently in master - no changes whatsoever

However I still have the aforementioned issue.

I have tried: 

docker rmi blockstack/stacks-blockchain:feat-miner-control
docker rmi blockstack/stacks-blockchain:latest
docker rmi blockstack/explorer:latest
docker rmi blockstack/bitcoind:puppet-chain
docker rmi postgres:alpine
docker rmi blockstack/stacks-blockchain-api:latest
$ ./manage.sh private-testnet upgrade
Running: docker-compose -f ./configurations/common.yaml -f ./configurations/private-testnet.yaml pull 
Pulling bitcoin              ... done
Pulling postgres             ... done
Pulling stacks-node-api      ... done
Pulling explorer             ... done
Pulling stacks-node-follower ... done

However I still have the error: 

stacks-node-follower    | ERRO [1623862680.986874] [testnet/stacks-node/src/burnchains/bitcoin_regtest_controller.rs:619] [main] Bitcoin RPC failure: error listing utxos Network("Bitcoin RPC: status(500) != success, 
body is \'Object({\"error\": Object({\"code\": Number(-18), \"message\": String(\"No wallet is loaded. Load a wallet using loadwallet or create 
a new one with createwallet. (Note: A default wallet is no longer automatically created)\")}), \"id\": String(\"stacks\"), \"result\": Null})\'"

I don't know what to do. My only idea is that if the same code/configuration with the same steps works seamlessly for you, then the configuration files we use must be different. I will have a look on the configuration files in the stacks-blockchain repository. But this is the only thing that could be different in our workflow.

Any thoughts? Thank you 😄

wileyj commented 3 years ago

And this output is from a clean working copy? i.e. there are no local changes? what do the bitcoin container logs show?

wileyj commented 3 years ago

@dorucioclea I'm still unable to reproduce...here's a sample exec on my local machine. For this test, i started with a clean working directory (no local changes), on the master branch. I had also cleaned out all containers and images so everything had to be pulled from remote. I've also added some line breaks in my terminal output so it's easier to read.

Please note the docker images --digests command I ran - it might be worthwhile to confirm your image digests match what i'm seeing.

for reference, this is the version of Docker i'm running:

03:05:56 {master} ~/Git/blockstack/public/stacks-local-dev$ docker -v
Docker version 20.10.6, build 370c289
02:46:28 (master) ~/Git/blockstack/public/stacks-local-dev$ gitinfo 
* remote origin
  Fetch URL: https://github.com/blockstack/stacks-local-dev
  Push  URL: https://github.com/blockstack/stacks-local-dev
  HEAD branch: master
  Remote branches:
    broken-wallet          tracked
    composer_composability tracked
    kucoin                 tracked
    mainnet                tracked
    mainnet-persist        tracked
    master                 tracked
    mocknet                tracked
    mocknet-persist        tracked
    okcoin                 tracked
    private-testnet        tracked
    testnet                tracked
  Local branches configured for 'git pull':
    broken-wallet          merges with remote broken-wallet
    composer_composability merges with remote composer_composability
    kucoin                 merges with remote kucoin
    link-test              merges with remote link-test
    mainnet                merges with remote mainnet
    mainnet-persist        merges with remote mainnet-persist
    master                 merges with remote master
    mocknet                merges with remote mocknet
    mocknet-persist        merges with remote mocknet-persist
    okcoin                 merges with remote okcoin
    private-testnet        merges with remote private-testnet
    testnet                merges with remote testnet
  Local refs configured for 'git push':
    broken-wallet          pushes to broken-wallet          (up to date)
    composer_composability pushes to composer_composability (up to date)
    kucoin                 pushes to kucoin                 (up to date)
    mainnet                pushes to mainnet                (up to date)
    mainnet-persist        pushes to mainnet-persist        (up to date)
    master                 pushes to master                 (up to date)
    mocknet                pushes to mocknet                (up to date)
    mocknet-persist        pushes to mocknet-persist        (up to date)
    okcoin                 pushes to okcoin                 (up to date)
    private-testnet        pushes to private-testnet        (up to date)
    testnet                pushes to testnet                (up to date)

02:46:54 (master) ~/Git/blockstack/public/stacks-local-dev$ git log -1
commit 3bd9c6e6f33a81a6c850fa434abfa0480dcd0f91 (HEAD -> master, origin/master, origin/HEAD)
Merge: 56909d9 5bdac27
Author: wileyj <2847772+wileyj@users.noreply.github.com>
Date:   Mon Jun 7 11:52:03 2021 -0400

    Merge pull request #39 from Akirtovskis/patch-1

    docs: remove $ from cmd so it is easier to paste in terminal

02:46:58 (master) ~/Git/blockstack/public/stacks-local-dev$ docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

02:47:06 (master) ~/Git/blockstack/public/stacks-local-dev$ docker images
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE

02:47:08 (master) ~/Git/blockstack/public/stacks-local-dev$ ./manage.sh private-testnet upgrade
Running: docker-compose -f ./configurations/common.yaml -f ./configurations/private-testnet.yaml pull 
Pulling bitcoin              ... done
Pulling postgres             ... done
Pulling stacks-node-api      ... done
Pulling stacks-node-follower ... done
Pulling explorer             ... done

02:57:36 (master) ~/Git/blockstack/public/stacks-local-dev$ docker images
REPOSITORY                         TAG                  IMAGE ID       CREATED       SIZE
blockstack/explorer                latest               a68f69cee53c   2 weeks ago   1.91GB
blockstack/stacks-blockchain-api   latest               9b482165ce2c   4 weeks ago   643MB
postgres                           alpine               0ad8200ddbe2   4 weeks ago   160MB
blockstack/stacks-blockchain       feat-miner-control   f2e0e5bf8bed   5 weeks ago   322MB
blockstack/bitcoind                puppet-chain         04b0e113f846   6 weeks ago   87MB

02:57:52 (master) ~/Git/blockstack/public/stacks-local-dev$ docker images --digests
REPOSITORY                         TAG                  DIGEST                                                                    IMAGE ID       CREATED       SIZE
blockstack/explorer                latest               sha256:e02059b971278537eb8b05f9329ff0f14f3fdba3306ba9cddc3c05c2a6c642a3   a68f69cee53c   2 weeks ago   1.91GB
blockstack/stacks-blockchain-api   latest               sha256:0e5f43e1b6c9c6eb9d2894940f170d1ba127999b98b2cb944272eb09fdcacc61   9b482165ce2c   4 weeks ago   643MB
postgres                           alpine               sha256:448f2cee7f081887456211d16cdbace69b387b10cfffedb638f69d7b9a27490b   0ad8200ddbe2   4 weeks ago   160MB
blockstack/stacks-blockchain       feat-miner-control   sha256:eeb7bb50ad9337d9b76ef2118010a0697c082e1e9a21ce86e182f7b593607579   f2e0e5bf8bed   5 weeks ago   322MB
blockstack/bitcoind                puppet-chain         sha256:0235a5fcfdcbfa756a9d00724d0604e52d2932171ab4045143d79de308ef9435   04b0e113f846   6 weeks ago   87MB

02:57:55 (master) ~/Git/blockstack/public/stacks-local-dev$ ./manage.sh private-testnet up
Running: docker-compose -f ./configurations/common.yaml -f ./configurations/private-testnet.yaml up -d
Docker Compose is now in the Docker CLI, try `docker compose up`

Creating network "stacks-blockchain" with driver "bridge"
Creating bitcoin  ... done
Creating explorer        ... done
Creating postgres ... done
Creating stacks-node-api ... done
Creating stacks-node-follower ... done
Brought up private-testnet, use './manage.sh private-testnet logs' to follow log files.

attaching the log output as well in tar.gz format so you can see what my logs are showing. output.tar.gz

wileyj commented 3 years ago

The only theory i have here is that there is something off with the bitcoin container you're using - either something is cached, or it wasn't removed maybe. otherwise, thepuppet-chain binary on that image, combined with the feat/miner-control stacks-node image should resolve the no wallet error. there is also nothing inherently special about the bitcoin image - the Dockerfile is here: https://github.com/blockstack/stacks-local-dev/blob/master/Dockerfiles/Dockerfile

dorucioclea commented 3 years ago

Hmm alright, so I made sure I'm on latest master, no changes.

Furthermore, I cleaned up everything like so: 

docker-compose -f ./configurations/common.yaml -f ./configurations/private-testnet.yaml down --rmi all --volumes

docker rmi $(docker images | grep blockstack | tr -s ' ' | cut -d ' ' -f 3)

docker rmi $(docker images | grep stacks | tr -s ' ' | cut -d ' ' -f 3)

docker rmi $(docker images | grep bitcoin | tr -s ' ' | cut -d ' ' -f 3)

My output ( the important part of it )


bitcoin                 | 2021-06-17T07:59:32Z Using obfuscation key for /root/.bitcoin/regtest/chainstate: 57769f302243abea
bitcoin                 | 2021-06-17T07:59:32Z init message: Rewinding blocks...
bitcoin                 | 2021-06-17T07:59:32Z  block index              53ms
bitcoin                 | 2021-06-17T07:59:32Z Opening LevelDB in /root/.bitcoin/regtest/indexes/txindex
bitcoin                 | 2021-06-17T07:59:32Z Opened LevelDB successfully
bitcoin                 | 2021-06-17T07:59:32Z Using obfuscation key for /root/.bitcoin/regtest/indexes/txindex: 0000000000000000
bitcoin                 | 2021-06-17T07:59:32Z txindex thread start
bitcoin                 | 2021-06-17T07:59:32Z txindex is enabled
bitcoin                 | 2021-06-17T07:59:32Z txindex thread exit
bitcoin                 | 2021-06-17T07:59:32Z loadblk thread start
bitcoin                 | 2021-06-17T07:59:32Z UpdateTip: new best=0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206 height=0 version=0x00000001 log2_work=1.000000 tx=1 date='2011-02-02T23:16:42Z' progress=1.000000 cache=0.0MiB(0txo)
bitcoin                 | 2021-06-17T07:59:32Z block tree size = 1
bitcoin                 | 2021-06-17T07:59:32Z nBestHeight = 0
bitcoin                 | 2021-06-17T07:59:32Z Failed to open mempool file from disk. Continuing anyway.
bitcoin                 | 2021-06-17T07:59:32Z loadblk thread exit
bitcoin                 | 2021-06-17T07:59:32Z torcontrol thread start
bitcoin                 | 2021-06-17T07:59:32Z Bound to 0.0.0.0:28444
bitcoin                 | 2021-06-17T07:59:32Z Bound to 127.0.0.1:18445
bitcoin                 | 2021-06-17T07:59:32Z init message: Loading P2P addresses...
bitcoin                 | 2021-06-17T07:59:32Z ERROR: DeserializeFileDB: Failed to open file /root/.bitcoin/regtest/peers.dat
bitcoin                 | 2021-06-17T07:59:32Z Invalid or missing peers.dat; recreating
bitcoin                 | 2021-06-17T07:59:33Z ERROR: DeserializeFileDB: Failed to open file /root/.bitcoin/regtest/anchors.dat
bitcoin                 | 2021-06-17T07:59:33Z 0 block-relay-only anchors will be tried for connections.
bitcoin                 | 2021-06-17T07:59:33Z init message: Starting network threads...
bitcoin                 | 2021-06-17T07:59:33Z net thread start
bitcoin                 | 2021-06-17T07:59:33Z dnsseed thread start
bitcoin                 | 2021-06-17T07:59:33Z 0 addresses found from DNS seeds
bitcoin                 | 2021-06-17T07:59:33Z dnsseed thread exit
bitcoin                 | 2021-06-17T07:59:33Z addcon thread start
bitcoin                 | 2021-06-17T07:59:33Z msghand thread start
bitcoin                 | 2021-06-17T07:59:33Z opencon thread start
bitcoin                 | 2021-06-17T07:59:33Z init message: Done loading
stacks-node-follower    | INFO [1623916774.343527] [testnet/stacks-node/src/run_loop/neon.rs:147] [main] Miner node: checking UTXOs at address: mtFzK54XtpktHj7fKonFExEPEGkUMsiXdy
stacks-node-follower    | WARN [1623916774.344113] [testnet/stacks-node/src/run_loop/neon.rs:150] [main] Error when creating wallet: Network("Bitcoin RPC: connection failed - Os { code: 111, kind: ConnectionRefused, message: \"Connection refused\" }")
stacks-node-follower    | ERRO [1623916774.344360] [testnet/stacks-node/src/burnchains/bitcoin_regtest_controller.rs:619] [main] Bitcoin RPC failure: error listing utxos Network("Bitcoin RPC: connection failed - Os { code: 111, kind: ConnectionRefused, message: \"Connection refused\" }")
stacks-node-api         | {"message":"Stacks 1.0 token offering data import started","level":"info","timestamp":"2021-06-17T07:59:35.132Z"}
bitcoin                 | Bootstrapping chain
bitcoin                 | INFO: detected DYNAMIC_GENESIS_TIMESTAMP, will set the genesis timestamp to 1623916775
bitcoin                 | 2021-06-17T07:59:35Z CreateNewBlock(): block weight: 896 txs: 0 fees: 0 sigops 400
bitcoin                 | 2021-06-17T07:59:35Z Pre-allocating up to position 0x100000 in rev00000.dat
bitcoin                 | 2021-06-17T07:59:35Z Leaving InitialBlockDownload (latching to false)
bitcoin                 | 2021-06-17T07:59:35Z UpdateTip: new best=47b75cc9fcf17f73556bcda2155f48ccc29e190fc92e00222caa05b16514f121 height=1 version=0x20000000 log2_work=2.000000 tx=2 date='2021-06-17T07:59:35Z' progress=1.000000 cache=0.0MiB(1txo)

The important thing here is that it seems that stacks-node-follower attempts to create the wallet before bitcoin finishes creating blocks.

That gave me an idea:

I first started ONLY the bitcoin image 

docker-compose -f ./configurations/common.yaml -f ./configurations/private-testnet.yaml up -d bitcoin

Waited a 30 seconds then started the rest of the stack: 

docker-compose -f ./configurations/common.yaml -f ./configurations/private-testnet.yaml up

I can confirm this works now but it doesn't work if I start all the services at once. That makes me think this is a timing issue. What do you think?

dorucioclea commented 3 years ago

I'm thinking of adding a healthcheck on bitcoin so I could properly use depends-on docker-compose directive. Or is there any other way to fix the timing issue ?

Other than that, if I first start the bitcoin node, wait for it to initialize and then start the rest of the services, everything works smooth 💯

dorucioclea commented 3 years ago

Alright, @wileyj so this did the trick for me:

When specifying the bitcoin image definition, I added a healthcheck:


bitcoin:
    image: blockstack/bitcoind:puppet-chain
    container_name: bitcoin
    restart: on-failure
    volumes:
      - ./private-testnet:/etc/bitcoin
    expose:
      - 18443
      - 28444
    environment:
      - DYNAMIC_GENESIS_TIMESTAMP=1
      - RUST_BACKTRACE=1
    networks:
      - stacks-blockchain
    healthcheck:
      test: ["CMD", "curl", "-f", "-u", "blockstack:blockstacksystem", "http://bitcoin:18443"]
      interval: 30s
      timeout: 10s
      retries: 10

And then made the stacks-node-follower container dependent of bitcoin container being healthy like this:


stacks-node-follower:
    image: blockstack/stacks-blockchain:feat-miner-control
    container_name: stacks-node-follower
    restart: on-failure
    ports:
      - ${STACKS_FOLLOWER_RPC_PORT_LOCAL:-20443}:20443
      - ${STACKS_FOLLOWER_P2P_PORT_LOCAL:-20444}:20444
    networks:
      - stacks-blockchain
    volumes:
      - ./private-testnet:/src/stacks-node
    depends_on:
      - stacks-node-api
      - bitcoin

This fixed it and is actually a best practice when dealing with timing issues between containers

I could prepare a PR for this, maybe someone else has this issue. Issues with containers starting sooner on some machines than others can happen due to difference in docker performance (how many cores are assigned to WSL2 and so on))

Later edit:

False alarm, healthcheck doesn't fix this issue completely. However, running bitcoin, waiting like 10 seconds and then running the rest of services does the trick 👌 Is there a way to include some sort of retrying mechanism in stacks-blockchain for when bitcoin responds with connection refused on wallet creation? 🤔

wileyj commented 3 years ago 
The important thing here is that it seems that stacks-node-follower attempts to create the wallet before bitcoin finishes creating blocks.

this is very interesting.....I think we can address this on our end, to be fair though it's something we thought possible but had never seen it in practice.

You are correct though, the timing absolutely matters - we'll have to look at what we can do here to slow things down for folks with faster systems, haha. I'll read through the rest of your comments in a bit, but i think we should loop in @MarvinJanssen as well here to spitball some ideas.

thanks again for your help diagnosing this!

dorucioclea commented 3 years ago

Hey, glad to help.

I've attached the full docker-compose logfile in case this would help provide more clarity 😄

logs.zip

wileyj commented 3 years ago

Outstanding, thanks! i'll do my best to have a fix for this by monday, but i've been swamped as of late so don't hold me to it.

dorucioclea commented 3 years ago

No worries and thank you, I'd also love to get involved, as I'm currently learning all I can about the platform

dorucioclea commented 3 years ago

Hey guys, ended up fixing it from docker-compose by using the wonderful wait utility from here: https://github.com/ufoscout/docker-compose-wait

Essencially waiting for bitcoin:18443 to become available and then waiting a bit longer just to make sure

My container docker-compose configuration looks like: 


stacks-blockchain:
      build:
        context: ./provisioning/stacks-blockchain
      restart: on-failure
      environment:
        STACKS_EVENT_OBSERVER: stacks-blockchain-api:3700
        XBLOCKSTACK_DEBUG: 1
        RUST_BACKTRACE: 1
        WAIT_HOST_CONNECT_TIMEOUT: 6
        WAIT_BEFORE: 5
        WAIT_HOSTS: bitcoin:18443
        WAIT_AFTER: 15
        WAIT_SLEEP_INTERVAL: 1
      ports:
        - "20443:20443"
        - "20444:20444"
      depends_on:
        - stacks-blockchain-api
      networks:
        - stacks-blockchain

I ended up wrapping blockstack/stacks-blockchain:develop-stretch into my own dockerfile so I could copy the wait utility. Confirmed this works. Looking forward to the fix on stacks-blockchain but for now I've got my functional setup 😄

Included an overly simplified private testnet setup here: https://github.com/dorucioclea/stacks-dev-env

wileyj commented 3 years ago

Thanks, and nice work! i'm not sure we want to have a modified stacks-blockchain image as part of this repo, i've been looking at ways to incorporate what that "wait" function does without having to modify the underlying images. I have 2 possible solutions, but need time to test - essentially a docker health check, or loop over nc until bitcoin is up (replicating what you've done, but without having to create a new image)

wileyj commented 3 years ago

bitcoin:18443

@MarvinJanssen thoughts on using the changes here from @dorucioclea vs. what i've added here: https://github.com/blockstack/stacks-local-dev/commit/958e78f71883319a6e3382bd6d0025a8630717f1

dorucioclea commented 3 years ago

Tbh this looks nice to me, minimally invasive @wileyj great work 🎉

MarvinJanssen commented 3 years ago

@wileyj I think your approach makes the most sense for now.

wileyj commented 3 years ago

Thanks all! i'm closing this for now - the image used here is a bit old, but once a new master branch is merged in stacks-blockchain (coming soon), we can update to use the latest image for this configuration.