Closed dorucioclea closed 3 years ago
We'll keep this simple to start with - I can confirm the wallet issue on my end, so we'll focus on that first. I'll test some potential fixes and get back to you, and then we can discuss the remainder of your questions - the wallet issue is definitely the most important here.
This network is also something relatively new we added, born from an internal hackathon - so it's definitely a work in progress and we're still ironing things out.
Regarding "We are using the puppet-chain version of bitcoin node but I noticed that this is not the version used for the stack-blockchain-api scripts", could you share the version of bitcoin core that is being used? Should look like the following line in the logs:
bitcoin | 2021-06-08T15:47:41Z Bitcoin Core version v0.20.99.0-831675c8d (release build)
For versions of bitcoin core earlier than 0.21.0, a default wallet should automatically be created (so I wouldn't expect you would need a call to create wallet at all).
edit: I was incorrect, by version 0.20.99.0, a default wallet was not being created automatically. The master branch has a commit updating the stacks-follower-node to be a later version (which is compatible with bitcoin version 0.21.0) (https://github.com/blockstack/stacks-local-dev/commit/075a9ac10415470ae16dd6c3d098f3b06a8f1771)
Thank you for the quick response. Not at the computer atm, will provide a more detailed log tomorrow π
Regarding "We are using the puppet-chain version of bitcoin node but I noticed that this is not the version used for the stack-blockchain-api scripts", could you share the version of bitcoin core that is being used? Should look like the following line in the logs:
bitcoin | 2021-06-08T15:47:41Z Bitcoin Core version v0.20.99.0-831675c8d (release build)
For versions of bitcoin core earlier than 0.21.0, a default wallet should automatically be created (so I wouldn't expect you would need a call to create wallet at all).
The version here should be identical since it's using a docker image
In that case, logs for the wallet not being created would be helpful (so removing the call to create wallet & the disable wallet line).
Sure. So I'm running exactly what's on master rn, no modifications.
Bitcoin node log:
2021-06-09T12:33:02Z Bitcoin Core version v0.20.99.0-831675c8d (release build)
Bitcoin starting logs:
2021-06-09T12:33:02Z Bitcoin Core version v0.20.99.0-831675c8d (release build)
2021-06-09T12:33:02Z Validating signatures for all blocks.
2021-06-09T12:33:02Z Setting nMinimumChainWork=0000000000000000000000000000000000000000000000000000000000000000
2021-06-09T12:33:02Z Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation
2021-06-09T12:33:02Z Using RdSeed as additional entropy source
2021-06-09T12:33:02Z Using RdRand as an additional entropy source
2021-06-09T12:33:02Z Default data directory /root/.bitcoin
2021-06-09T12:33:02Z Using data directory /root/.bitcoin/regtest
2021-06-09T12:33:02Z Config file: /etc/bitcoin/bitcoin.conf
2021-06-09T12:33:02Z Config file arg: listen="1"
2021-06-09T12:33:02Z Config file arg: regtest="1"
2021-06-09T12:33:02Z Config file arg: rpcallowip="0.0.0.0/0"
2021-06-09T12:33:02Z Config file arg: rpcallowip="::/0"
2021-06-09T12:33:02Z Config file arg: rpcpassword=****
2021-06-09T12:33:02Z Config file arg: rpcserialversion="0"
2021-06-09T12:33:02Z Config file arg: rpcuser=****
2021-06-09T12:33:02Z Config file arg: rpcworkqueue="100"
2021-06-09T12:33:02Z Config file arg: server="1"
2021-06-09T12:33:02Z Config file arg: txindex="1"
2021-06-09T12:33:02Z Config file arg: [regtest] bind="0.0.0.0:28444"
2021-06-09T12:33:02Z Config file arg: [regtest] rpcbind=****
2021-06-09T12:33:02Z Config file arg: [regtest] rpcport="28443"
2021-06-09T12:33:02Z Command-line arg: conf="/etc/bitcoin/bitcoin.conf"
2021-06-09T12:33:02Z Command-line arg: datadir="/root/.bitcoin"
2021-06-09T12:33:02Z Command-line arg: debuglogfile=false
2021-06-09T12:33:02Z Command-line arg: pid="/run/bitcoind.pid"
2021-06-09T12:33:02Z Using at most 125 automatic connections (1048576 file descriptors available)
2021-06-09T12:33:02Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
2021-06-09T12:33:02Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
2021-06-09T12:33:02Z Script verification uses 15 additional threads
2021-06-09T12:33:02Z scheduler thread start
2021-06-09T12:33:02Z WARNING: the RPC server is not safe to expose to untrusted networks such as the public internet
2021-06-09T12:33:02Z HTTP: creating work queue of depth 100
2021-06-09T12:33:02Z Config options rpcuser and rpcpassword will soon be deprecated. Locally-run instances may remove rpcuser to use cookie-based auth, or may be replaced with rpcauth. Please see share/rpcauth for rpcauth auth generation.
2021-06-09T12:33:02Z HTTP: starting 4 worker threads
2021-06-09T12:33:02Z Using wallet directory /root/.bitcoin/regtest/wallets
2021-06-09T12:33:02Z init message: Verifying wallet(s)...
2021-06-09T12:33:02Z init message: Loading banlist...
2021-06-09T12:33:02Z ERROR: DeserializeFileDB: Failed to open file /root/.bitcoin/regtest/banlist.dat
2021-06-09T12:33:02Z Invalid or missing banlist.dat; recreating
2021-06-09T12:33:02Z SetNetworkActive: true
2021-06-09T12:33:02Z Using /16 prefix for IP bucketing
2021-06-09T12:33:02Z Cache configuration:
2021-06-09T12:33:02Z * Using 2.0 MiB for block index database
2021-06-09T12:33:02Z * Using 56.0 MiB for transaction index database
2021-06-09T12:33:02Z * Using 8.0 MiB for chain state database
2021-06-09T12:33:02Z * Using 384.0 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
2021-06-09T12:33:02Z init message: Loading block index...
2021-06-09T12:33:02Z Switching active chainstate to Chainstate [ibd] @ height -1 (null)
2021-06-09T12:33:02Z Opening LevelDB in /root/.bitcoin/regtest/blocks/index
2021-06-09T12:33:02Z Opened LevelDB successfully
2021-06-09T12:33:02Z Using obfuscation key for /root/.bitcoin/regtest/blocks/index: 0000000000000000
2021-06-09T12:33:02Z LoadBlockIndexDB: last block file = 0
2021-06-09T12:33:02Z LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=0, size=0, heights=0...0, time=1970-01-01...1970-01-01)
2021-06-09T12:33:02Z Checking all blk files are present...
2021-06-09T12:33:02Z Initializing databases...
2021-06-09T12:33:02Z Pre-allocating up to position 0x1000000 in blk00000.dat
2021-06-09T12:33:02Z Opening LevelDB in /root/.bitcoin/regtest/chainstate
2021-06-09T12:33:02Z Opened LevelDB successfully
2021-06-09T12:33:02Z Wrote new obfuscate key for /root/.bitcoin/regtest/chainstate: 75d6e28d0a728167
2021-06-09T12:33:02Z Using obfuscation key for /root/.bitcoin/regtest/chainstate: 75d6e28d0a728167
2021-06-09T12:33:02Z init message: Rewinding blocks...
2021-06-09T12:33:02Z block index 72ms
2021-06-09T12:33:02Z Opening LevelDB in /root/.bitcoin/regtest/indexes/txindex
2021-06-09T12:33:02Z Opened LevelDB successfully
2021-06-09T12:33:02Z Using obfuscation key for /root/.bitcoin/regtest/indexes/txindex: 0000000000000000
2021-06-09T12:33:02Z txindex thread start
2021-06-09T12:33:02Z txindex is enabled
2021-06-09T12:33:02Z txindex thread exit
2021-06-09T12:33:02Z loadblk thread start
2021-06-09T12:33:02Z UpdateTip: new best=0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206 height=0 version=0x00000001 log2_work=1.000000 tx=1 date='2011-02-02T23:16:42Z' progress=1.000000 cache=0.0MiB(0txo)
2021-06-09T12:33:02Z block tree size = 1
2021-06-09T12:33:02Z nBestHeight = 0
2021-06-09T12:33:02Z Failed to open mempool file from disk. Continuing anyway.
2021-06-09T12:33:02Z loadblk thread exit
2021-06-09T12:33:02Z torcontrol thread start
2021-06-09T12:33:02Z Bound to 0.0.0.0:28444
2021-06-09T12:33:02Z Bound to 127.0.0.1:18445
2021-06-09T12:33:02Z init message: Loading P2P addresses...
2021-06-09T12:33:02Z ERROR: DeserializeFileDB: Failed to open file /root/.bitcoin/regtest/peers.dat
2021-06-09T12:33:02Z Invalid or missing peers.dat; recreating
2021-06-09T12:33:02Z ERROR: DeserializeFileDB: Failed to open file /root/.bitcoin/regtest/anchors.dat
2021-06-09T12:33:02Z 0 block-relay-only anchors will be tried for connections.
2021-06-09T12:33:02Z init message: Starting network threads...
2021-06-09T12:33:02Z dnsseed thread start
2021-06-09T12:33:02Z net thread start
2021-06-09T12:33:02Z 0 addresses found from DNS seeds
2021-06-09T12:33:02Z addcon thread start
2021-06-09T12:33:02Z init message: Done loading
2021-06-09T12:33:02Z opencon thread start
2021-06-09T12:33:02Z dnsseed thread exit
2021-06-09T12:33:02Z msghand thread start
Bootstrapping chain
INFO: detected DYNAMIC_GENESIS_TIMESTAMP, will set the genesis timestamp to 1623241984
2021-06-09T12:33:04Z CreateNewBlock(): block weight: 896 txs: 0 fees: 0 sigops 400
2021-06-09T12:33:04Z Pre-allocating up to position 0x100000 in rev00000.dat
2021-06-09T12:33:04Z Leaving InitialBlockDownload (latching to false)
2021-06-09T12:33:04Z UpdateTip: new best=000bc41da11a396eb3c2a891e6bb52cf4bda0f393e4e970ef7770404898d44b2 height=1 version=0x20000000 log2_work=2.000000 tx=2 date='2021-06-09T12:33:04Z' progress=1.000000 cache=0.0MiB(1txo)
................................
Stacks-node-follower error:
stacks-node-follower | ERRO [1623242134.265502] [testnet/stacks-node/src/burnchains/bitcoin_regtest_controller.rs:619] [main] Bitcoin RPC failure: error listing utxos Network("Bitcoin RPC: status(500) != success, body is \'Object({\"error\": Object({\"code\": Number(-18), \"message\": String(\"No wallet is loaded. Load a wallet using loadwallet or create a new one with createwallet. (Note: A default wallet is no longer automatically created)\")}), \"id\": String(\"stacks\"), \"result\": Null})\'")
Furthermore, stacks-blockchain-api
logs:
{"message":"Pg connection failed, retrying in 2000ms..","level":"error","timestamp":"2021-06-09T12:33:04.159Z"}
{"message":"Stacks 1.0 token offering data import started","level":"info","timestamp":"2021-06-09T12:33:07.050Z"}
{"message":"Stacks 1.0 token offering data import completed","level":"info","timestamp":"2021-06-09T12:33:08.908Z"}
{"message":"Notice: full BNS functionality requires 'BNS_IMPORT_DIR' to be set.","level":"warn","timestamp":"2021-06-09T12:33:08.909Z"}
{"message":"Event observer listening at: http://0.0.0.0:3700","level":"info","timestamp":"2021-06-09T12:33:08.916Z"}
Notice how the proxy and api (port 3999) is not starting
As shown above, after the stacks-node-follower
errors out, it will remain in this state, as it cannot move past the missing wallet
issue. My understanding is (correct me if I'm wrong please) that newer bitcoin builds stopped creating the (test) wallet by default
There are a few recent interesting issues on bitcoin's repository related to this, one of them being:
Furthermore, I noticed that on stacks-blockchain-api
the bitcoin version used in their docker-compose is:
"ruimarinho/bitcoin-core:0.19.1-alpine"
Included permalink to their docker-compose
setup:
Interestingly, they do not use this as they have a docker-compose override file
which will always override the stacks configuration to mocknet
.
Is there a reason we're using this particular bitcoin docker image and are not aligned with the devops scripts from stacks-blockchain-api
and/or stacks-blockchain
For example, stacks-blockchain
has a different set of configurations for local:
https://github.com/blockstack/stacks-blockchain/tree/master/testnet/stacks-node/conf
To address the wallet issue - for me, running the following commands on master (no further modifications) work:
./manage.sh private-testnet pull
./manage.sh private-testnet up
If you haven't already tried it, could you reset to the head of master and try these commands (just so we can both be using the exact same config/ image)
To address the wallet issue - for me, running the following commands on master (no further modifications) work:
./manage.sh private-testnet pull
./manage.sh private-testnet up
If you haven't already tried it, could you reset to the head of master and try these commands (just so we can both be using the exact same config/ image)
I'm on latest version but will do another docker pull just to be safe when I get back to a computer.
Another quick question : when you run the testnet does the api start (port 3999) with the condigs on master?
Thank you π
Is there a reason we're using this particular bitcoin docker image and are not aligned with the devops scripts from
stacks-blockchain-api
and/orstacks-blockchain
yes! the docker-compose you're referencing is very outdated (and really shouldn't be used at this point). the version of bitcoin used in this repo is the same version that we use internally at Hiro.
As for the configs, what's included here are samples based on what's in the stacks-blockchain repo. We try to keep them more or less identical, but it's impossible to keep them as 1:1 duplications.
Alrignt, thank you ! π₯
So, got back to the computer and did
./manage.sh private-testnet pull
(which does docker-compose -f ./configurations/common.yaml -f ./configurations/private-testnet.yaml pull
)
I was already up-to-date, latest changes from the master branch, no changes whatsoever on my part. I have the same error.
At this point, the only thing that could differ in your local stack are the configs. Could you please check that the 3 configuration files are up-to-date on master?
My configuration files versions are as follow:
/configurations/private-testnet/Config.toml
with content identical to https://github.com/blockstack/stacks-local-dev/blob/3bd9c6e6f33a81a6c850fa434abfa0480dcd0f91/configurations/private-testnet/Config.toml.sample#L1/configurations/private-testnet/bitcoin.conf
with content identical to https://github.com/blockstack/stacks-local-dev/blob/3bd9c6e6f33a81a6c850fa434abfa0480dcd0f91/configurations/private-testnet/bitcoin.conf.sample#L1/configurations/private-testnet/puppet-chain.toml
with content identical to https://github.com/blockstack/stacks-local-dev/blob/3bd9c6e6f33a81a6c850fa434abfa0480dcd0f91/configurations/private-testnet/puppet-chain.toml.sample#L1Thank you π
sorry for the delayed responses here....been dealing with a lot of sponsored tx's i've had to resend for the btc
domains. Once I work through that list, i'll circle back and devote more time here
sorry for the delayed responses here....been dealing with a lot of sponsored tx's i've had to resend for the
btc
domains. Once I work through that list, i'll circle back and devote more time here
Hey, no worries, and thank you for your help π sorry for the avalanche of questions on my side but I'm currently learning stacks/clarity and want to have a clear picture of it all + a completely-ish private devenv so I could run my tests and interact with contracts from the explorer ui π₯
Is there a reason we're using this particular bitcoin docker image and are not aligned with the devops scripts from
stacks-blockchain-api
and/orstacks-blockchain
yes! the docker-compose you're referencing is very outdated (and really shouldn't be used at this point). the version of bitcoin used in this repo is the same version that we use internally at Hiro.
As for the configs, what's included here are samples based on what's in the stacks-blockchain repo. We try to keep them more or less identical, but it's impossible to keep them as 1:1 duplications.
Thank you for sheding some light on the subject. So:
stacks-blockchain-api
. I will try my best at cross-referencing them, maybe a combination of what's in here and what's in there will help me π₯ Thank you π
Yes, this repo can be considered the "source of truth" for orchestration. @MarvinJanssen and myself do our best to keep it updated with stacks-blockchain/stacks-blockchain-api changes as well (if something in the configs change, we do our best to get them here).
I can say that the configs in the source repos have been a low priority, and what you see in this repo is based on the running systems at Hiro and what's in the base repos. ideally, they would be close to identical - but there will always be some drift.
For now, i can confirm that using blockstack/stacks-blockchain
image other than blockstack/stacks-blockchain:feat-miner-control
will result in the bitcoin wallet issue (this was the branch we created during hackathon to address that error, and the resulting image is what we found to work. not sure where that branch went but i'll work with @pavitthrap to look into it).
to keep this message short, i'll reply again with some steps i'd like for you to take to try and resolve or possibly to get more data to troubleshoot here.
First, ensure you have the latest code from the master branch. either git clone
again, or do a pull
and stash/remove any local changes. (if you've modified any of the .toml
files, best to remove them or rename them to .toml.save
).
Next, we'll remove any images locally that we'll be using:
docker rmi blockstack/stacks-blockchain:feat-miner-control
docker rmi blockstack/stacks-blockchain:latest
docker rmi blockstack/explorer:latest
docker rmi blockstack/bitcoind:puppet-chain
docker rmi postgres:alpine
docker rmi blockstack/stacks-blockchain-api:latest
Next, ensure nothing else is occupying the expected ports:
$ nc -vz localhost 20443
nc: connectx to localhost port 20443 (tcp) failed: Connection refused
nc: connectx to localhost port 20443 (tcp) failed: Connection refused
$ nc -vz localhost 20444
nc: connectx to localhost port 20444 (tcp) failed: Connection refused
nc: connectx to localhost port 20444 (tcp) failed: Connection refused
$ nc -vz localhost 3999
nc: connectx to localhost port 3999 (tcp) failed: Connection refused
nc: connectx to localhost port 3999 (tcp) failed: Connection refused
$ nc -vz localhost 3000
nc: connectx to localhost port 3000 (tcp) failed: Connection refused
nc: connectx to localhost port 3000 (tcp) failed: Connection refused
Next, pull the images we'll be using:
$ ./manage.sh private-testnet upgrade
Running: docker-compose -f ./configurations/common.yaml -f ./configurations/private-testnet.yaml pull
Pulling bitcoin ... done
Pulling postgres ... done
Pulling stacks-node-api ... done
Pulling explorer ... done
Pulling stacks-node-follower ... done
Start the services:
$ ./manage.sh private-testnet up
Running: docker-compose -f ./configurations/common.yaml -f ./configurations/private-testnet.yaml up -d
Docker Compose is now in the Docker CLI, try `docker compose up`
Creating network "stacks-blockchain" with driver "bridge"
Creating postgres ... done
Creating bitcoin ... done
Creating explorer ... done
Creating stacks-node-api ... done
Creating stacks-node-follower ... done
Brought up private-testnet, use './manage.sh private-testnet logs' to follow log files.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
43e45bdb8058 blockstack/stacks-blockchain:feat-miner-control "/bin/stacks-node stβ¦" About a minute ago Up 44 seconds 0.0.0.0:20443-20444->20443-20444/tcp, :::20443-20444->20443-20444/tcp stacks-node-follower
34602f89cf6d blockstack/stacks-blockchain-api:latest "docker-entrypoint.sβ¦" About a minute ago Up About a minute 0.0.0.0:3999->3999/tcp, :::3999->3999/tcp stacks-node-api
df48e1e7521b blockstack/explorer:latest "docker-entrypoint.sβ¦" 2 minutes ago Up About a minute 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp explorer
e2c35cd3dfa9 postgres:alpine "docker-entrypoint.sβ¦" 2 minutes ago Up About a minute 5432/tcp postgres
4069f6e8f0ed blockstack/bitcoind:puppet-chain "/sbin/tini -- /dockβ¦" 2 minutes ago Up 2 minutes 18443/tcp, 28444/tcp
And finally, check the logs of all containers:
./manage.sh private-testnet logs
You should see log entries like this after a bit:
...
bitcoin | 2021-06-11T21:38:38Z [default wallet] AddToWallet ca38081dda807fef8d7652750d9c0c450ff7788cc8ad798bff577cb04a63b638 new
bitcoin | Assembled block 338. Will be accepting incoming transactions for the next 7000ms, then assemble block 339.
bitcoin | 2021-06-11T21:38:45Z CreateNewBlock(): block weight: 904 txs: 0 fees: 0 sigops 400
bitcoin | 2021-06-11T21:38:45Z UpdateTip: new best=1804475290894b8cfc1fbd9cf03d62a973764fa018689e38316c4f72627b532f height=338 version=0x30000000 log2_work=9.405141 tx=339 date='2021-06-11T21:38:45Z' progress=1.000000 cache=0.0MiB(338txo)
bitcoin | 2021-06-11T21:38:45Z [default wallet] AddToWallet 63d190ac0934b5fb933d90963cb02266d76a56980427835b3a95e8fb72f8127c new
stacks-node-follower | INFO [1623447525.845309] [src/chainstate/stacks/index/marf.rs:1185] [main] Batching insertions in MARF: 23% (77127 out of 335331)
stacks-node-follower | INFO [1623447527.917987] [src/chainstate/stacks/index/marf.rs:1185] [main] Batching insertions in MARF: 24% (80480 out of 335331)
...
@pavitthrap I was able to reproduce the wallet issue using the stacks-blockchain image master
and develop
tag, let's discuss what's going on there.
@dorucioclea - the code from that feature branch image has been merged into develop
, but the image for develop was out of date. i've just updated the image, will give it a shot using that one shortly.
@pavitthrap do we have an idea of when those changes will be merged into main
?
Hey, thank you for the good news @wileyj I will give it a try with the develop
image then π
I see blockstack/stacks-blockchain:develop
was updated 13 hours ago so will use that one and let you know. Thank you π
@wileyj I would expect this commit to make it into the next release candidate (so probably next month sometime).
@wileyj @pavitthrap I tried with both blockstack/stacks-blockchain:develop
and blockstack/stacks-blockchain:feat-miner-control
with the config files currently in master
- no changes whatsoever
However I still have the aforementioned issue
.
I have tried:
docker rmi blockstack/stacks-blockchain:feat-miner-control
docker rmi blockstack/stacks-blockchain:latest
docker rmi blockstack/explorer:latest
docker rmi blockstack/bitcoind:puppet-chain
docker rmi postgres:alpine
docker rmi blockstack/stacks-blockchain-api:latest
$ ./manage.sh private-testnet upgrade
Running: docker-compose -f ./configurations/common.yaml -f ./configurations/private-testnet.yaml pull
Pulling bitcoin ... done
Pulling postgres ... done
Pulling stacks-node-api ... done
Pulling explorer ... done
Pulling stacks-node-follower ... done
However I still have the error:
stacks-node-follower | ERRO [1623862680.986874] [testnet/stacks-node/src/burnchains/bitcoin_regtest_controller.rs:619] [main] Bitcoin RPC failure: error listing utxos Network("Bitcoin RPC: status(500) != success,
body is \'Object({\"error\": Object({\"code\": Number(-18), \"message\": String(\"No wallet is loaded. Load a wallet using loadwallet or create
a new one with createwallet. (Note: A default wallet is no longer automatically created)\")}), \"id\": String(\"stacks\"), \"result\": Null})\'"
I don't know what to do. My only idea is that if the same code/configuration with the same steps works seamlessly for you, then the configuration files we use must be different
. I will have a look on the configuration files in the stacks-blockchain repository.
But this is the only thing that could be different in our workflow.
Any thoughts? Thank you π
And this output is from a clean working copy? i.e. there are no local changes? what do the bitcoin container logs show?
@dorucioclea I'm still unable to reproduce...here's a sample exec on my local machine. For this test, i started with a clean working directory (no local changes), on the master branch. I had also cleaned out all containers and images so everything had to be pulled from remote. I've also added some line breaks in my terminal output so it's easier to read.
Please note the docker images --digests
command I ran - it might be worthwhile to confirm your image digests match what i'm seeing.
for reference, this is the version of Docker i'm running:
03:05:56 {master} ~/Git/blockstack/public/stacks-local-dev$ docker -v
Docker version 20.10.6, build 370c289
02:46:28 (master) ~/Git/blockstack/public/stacks-local-dev$ gitinfo
* remote origin
Fetch URL: https://github.com/blockstack/stacks-local-dev
Push URL: https://github.com/blockstack/stacks-local-dev
HEAD branch: master
Remote branches:
broken-wallet tracked
composer_composability tracked
kucoin tracked
mainnet tracked
mainnet-persist tracked
master tracked
mocknet tracked
mocknet-persist tracked
okcoin tracked
private-testnet tracked
testnet tracked
Local branches configured for 'git pull':
broken-wallet merges with remote broken-wallet
composer_composability merges with remote composer_composability
kucoin merges with remote kucoin
link-test merges with remote link-test
mainnet merges with remote mainnet
mainnet-persist merges with remote mainnet-persist
master merges with remote master
mocknet merges with remote mocknet
mocknet-persist merges with remote mocknet-persist
okcoin merges with remote okcoin
private-testnet merges with remote private-testnet
testnet merges with remote testnet
Local refs configured for 'git push':
broken-wallet pushes to broken-wallet (up to date)
composer_composability pushes to composer_composability (up to date)
kucoin pushes to kucoin (up to date)
mainnet pushes to mainnet (up to date)
mainnet-persist pushes to mainnet-persist (up to date)
master pushes to master (up to date)
mocknet pushes to mocknet (up to date)
mocknet-persist pushes to mocknet-persist (up to date)
okcoin pushes to okcoin (up to date)
private-testnet pushes to private-testnet (up to date)
testnet pushes to testnet (up to date)
02:46:54 (master) ~/Git/blockstack/public/stacks-local-dev$ git log -1
commit 3bd9c6e6f33a81a6c850fa434abfa0480dcd0f91 (HEAD -> master, origin/master, origin/HEAD)
Merge: 56909d9 5bdac27
Author: wileyj <2847772+wileyj@users.noreply.github.com>
Date: Mon Jun 7 11:52:03 2021 -0400
Merge pull request #39 from Akirtovskis/patch-1
docs: remove $ from cmd so it is easier to paste in terminal
02:46:58 (master) ~/Git/blockstack/public/stacks-local-dev$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
02:47:06 (master) ~/Git/blockstack/public/stacks-local-dev$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
02:47:08 (master) ~/Git/blockstack/public/stacks-local-dev$ ./manage.sh private-testnet upgrade
Running: docker-compose -f ./configurations/common.yaml -f ./configurations/private-testnet.yaml pull
Pulling bitcoin ... done
Pulling postgres ... done
Pulling stacks-node-api ... done
Pulling stacks-node-follower ... done
Pulling explorer ... done
02:57:36 (master) ~/Git/blockstack/public/stacks-local-dev$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
blockstack/explorer latest a68f69cee53c 2 weeks ago 1.91GB
blockstack/stacks-blockchain-api latest 9b482165ce2c 4 weeks ago 643MB
postgres alpine 0ad8200ddbe2 4 weeks ago 160MB
blockstack/stacks-blockchain feat-miner-control f2e0e5bf8bed 5 weeks ago 322MB
blockstack/bitcoind puppet-chain 04b0e113f846 6 weeks ago 87MB
02:57:52 (master) ~/Git/blockstack/public/stacks-local-dev$ docker images --digests
REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE
blockstack/explorer latest sha256:e02059b971278537eb8b05f9329ff0f14f3fdba3306ba9cddc3c05c2a6c642a3 a68f69cee53c 2 weeks ago 1.91GB
blockstack/stacks-blockchain-api latest sha256:0e5f43e1b6c9c6eb9d2894940f170d1ba127999b98b2cb944272eb09fdcacc61 9b482165ce2c 4 weeks ago 643MB
postgres alpine sha256:448f2cee7f081887456211d16cdbace69b387b10cfffedb638f69d7b9a27490b 0ad8200ddbe2 4 weeks ago 160MB
blockstack/stacks-blockchain feat-miner-control sha256:eeb7bb50ad9337d9b76ef2118010a0697c082e1e9a21ce86e182f7b593607579 f2e0e5bf8bed 5 weeks ago 322MB
blockstack/bitcoind puppet-chain sha256:0235a5fcfdcbfa756a9d00724d0604e52d2932171ab4045143d79de308ef9435 04b0e113f846 6 weeks ago 87MB
02:57:55 (master) ~/Git/blockstack/public/stacks-local-dev$ ./manage.sh private-testnet up
Running: docker-compose -f ./configurations/common.yaml -f ./configurations/private-testnet.yaml up -d
Docker Compose is now in the Docker CLI, try `docker compose up`
Creating network "stacks-blockchain" with driver "bridge"
Creating bitcoin ... done
Creating explorer ... done
Creating postgres ... done
Creating stacks-node-api ... done
Creating stacks-node-follower ... done
Brought up private-testnet, use './manage.sh private-testnet logs' to follow log files.
attaching the log output as well in tar.gz
format so you can see what my logs are showing.
output.tar.gz
The only theory i have here is that there is something off with the bitcoin container you're using - either something is cached, or it wasn't removed maybe. otherwise, thepuppet-chain
binary on that image, combined with the feat/miner-control
stacks-node image should resolve the no wallet error.
there is also nothing inherently special about the bitcoin image - the Dockerfile is here: https://github.com/blockstack/stacks-local-dev/blob/master/Dockerfiles/Dockerfile
Hmm alright, so I made sure I'm on latest master, no changes.
Furthermore, I cleaned up everything like so:
docker-compose -f ./configurations/common.yaml -f ./configurations/private-testnet.yaml down --rmi all --volumes
docker rmi $(docker images | grep blockstack | tr -s ' ' | cut -d ' ' -f 3)
docker rmi $(docker images | grep stacks | tr -s ' ' | cut -d ' ' -f 3)
docker rmi $(docker images | grep bitcoin | tr -s ' ' | cut -d ' ' -f 3)
My output ( the important part of it )
bitcoin | 2021-06-17T07:59:32Z Using obfuscation key for /root/.bitcoin/regtest/chainstate: 57769f302243abea
bitcoin | 2021-06-17T07:59:32Z init message: Rewinding blocks...
bitcoin | 2021-06-17T07:59:32Z block index 53ms
bitcoin | 2021-06-17T07:59:32Z Opening LevelDB in /root/.bitcoin/regtest/indexes/txindex
bitcoin | 2021-06-17T07:59:32Z Opened LevelDB successfully
bitcoin | 2021-06-17T07:59:32Z Using obfuscation key for /root/.bitcoin/regtest/indexes/txindex: 0000000000000000
bitcoin | 2021-06-17T07:59:32Z txindex thread start
bitcoin | 2021-06-17T07:59:32Z txindex is enabled
bitcoin | 2021-06-17T07:59:32Z txindex thread exit
bitcoin | 2021-06-17T07:59:32Z loadblk thread start
bitcoin | 2021-06-17T07:59:32Z UpdateTip: new best=0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206 height=0 version=0x00000001 log2_work=1.000000 tx=1 date='2011-02-02T23:16:42Z' progress=1.000000 cache=0.0MiB(0txo)
bitcoin | 2021-06-17T07:59:32Z block tree size = 1
bitcoin | 2021-06-17T07:59:32Z nBestHeight = 0
bitcoin | 2021-06-17T07:59:32Z Failed to open mempool file from disk. Continuing anyway.
bitcoin | 2021-06-17T07:59:32Z loadblk thread exit
bitcoin | 2021-06-17T07:59:32Z torcontrol thread start
bitcoin | 2021-06-17T07:59:32Z Bound to 0.0.0.0:28444
bitcoin | 2021-06-17T07:59:32Z Bound to 127.0.0.1:18445
bitcoin | 2021-06-17T07:59:32Z init message: Loading P2P addresses...
bitcoin | 2021-06-17T07:59:32Z ERROR: DeserializeFileDB: Failed to open file /root/.bitcoin/regtest/peers.dat
bitcoin | 2021-06-17T07:59:32Z Invalid or missing peers.dat; recreating
bitcoin | 2021-06-17T07:59:33Z ERROR: DeserializeFileDB: Failed to open file /root/.bitcoin/regtest/anchors.dat
bitcoin | 2021-06-17T07:59:33Z 0 block-relay-only anchors will be tried for connections.
bitcoin | 2021-06-17T07:59:33Z init message: Starting network threads...
bitcoin | 2021-06-17T07:59:33Z net thread start
bitcoin | 2021-06-17T07:59:33Z dnsseed thread start
bitcoin | 2021-06-17T07:59:33Z 0 addresses found from DNS seeds
bitcoin | 2021-06-17T07:59:33Z dnsseed thread exit
bitcoin | 2021-06-17T07:59:33Z addcon thread start
bitcoin | 2021-06-17T07:59:33Z msghand thread start
bitcoin | 2021-06-17T07:59:33Z opencon thread start
bitcoin | 2021-06-17T07:59:33Z init message: Done loading
stacks-node-follower | INFO [1623916774.343527] [testnet/stacks-node/src/run_loop/neon.rs:147] [main] Miner node: checking UTXOs at address: mtFzK54XtpktHj7fKonFExEPEGkUMsiXdy
stacks-node-follower | WARN [1623916774.344113] [testnet/stacks-node/src/run_loop/neon.rs:150] [main] Error when creating wallet: Network("Bitcoin RPC: connection failed - Os { code: 111, kind: ConnectionRefused, message: \"Connection refused\" }")
stacks-node-follower | ERRO [1623916774.344360] [testnet/stacks-node/src/burnchains/bitcoin_regtest_controller.rs:619] [main] Bitcoin RPC failure: error listing utxos Network("Bitcoin RPC: connection failed - Os { code: 111, kind: ConnectionRefused, message: \"Connection refused\" }")
stacks-node-api | {"message":"Stacks 1.0 token offering data import started","level":"info","timestamp":"2021-06-17T07:59:35.132Z"}
bitcoin | Bootstrapping chain
bitcoin | INFO: detected DYNAMIC_GENESIS_TIMESTAMP, will set the genesis timestamp to 1623916775
bitcoin | 2021-06-17T07:59:35Z CreateNewBlock(): block weight: 896 txs: 0 fees: 0 sigops 400
bitcoin | 2021-06-17T07:59:35Z Pre-allocating up to position 0x100000 in rev00000.dat
bitcoin | 2021-06-17T07:59:35Z Leaving InitialBlockDownload (latching to false)
bitcoin | 2021-06-17T07:59:35Z UpdateTip: new best=47b75cc9fcf17f73556bcda2155f48ccc29e190fc92e00222caa05b16514f121 height=1 version=0x20000000 log2_work=2.000000 tx=2 date='2021-06-17T07:59:35Z' progress=1.000000 cache=0.0MiB(1txo)
The important thing here is that it seems that stacks-node-follower
attempts to create the wallet before bitcoin finishes creating blocks.
That gave me an idea:
I first started ONLY the bitcoin image
docker-compose -f ./configurations/common.yaml -f ./configurations/private-testnet.yaml up -d bitcoin
Waited a 30 seconds then started the rest of the stack:
docker-compose -f ./configurations/common.yaml -f ./configurations/private-testnet.yaml up
I can confirm this works now but it doesn't work if I start all the services at once
. That makes me think this is a timing issue. What do you think?
I'm thinking of adding a healthcheck on bitcoin so I could properly use depends-on
docker-compose directive. Or is there any other way to fix the timing issue ?
Other than that, if I first start the bitcoin node, wait for it to initialize and then start the rest of the services, everything works smooth π―
Alright, @wileyj so this did the trick for me:
When specifying the bitcoin
image definition, I added a healthcheck:
bitcoin:
image: blockstack/bitcoind:puppet-chain
container_name: bitcoin
restart: on-failure
volumes:
- ./private-testnet:/etc/bitcoin
expose:
- 18443
- 28444
environment:
- DYNAMIC_GENESIS_TIMESTAMP=1
- RUST_BACKTRACE=1
networks:
- stacks-blockchain
healthcheck:
test: ["CMD", "curl", "-f", "-u", "blockstack:blockstacksystem", "http://bitcoin:18443"]
interval: 30s
timeout: 10s
retries: 10
And then made the stacks-node-follower
container dependent of bitcoin
container being healthy
like this:
stacks-node-follower:
image: blockstack/stacks-blockchain:feat-miner-control
container_name: stacks-node-follower
restart: on-failure
ports:
- ${STACKS_FOLLOWER_RPC_PORT_LOCAL:-20443}:20443
- ${STACKS_FOLLOWER_P2P_PORT_LOCAL:-20444}:20444
networks:
- stacks-blockchain
volumes:
- ./private-testnet:/src/stacks-node
depends_on:
- stacks-node-api
- bitcoin
This fixed it and is actually a best practice when dealing with timing issues between containers
I could prepare a PR for this, maybe someone else has this issue. Issues with containers starting sooner on some machines than others can happen due to difference in docker performance (how many cores are assigned to WSL2 and so on))
Later edit:
False alarm, healthcheck doesn't fix this issue completely. However, running bitcoin, waiting like 10 seconds and then running the rest of services does the trick π Is there a way to include some sort of retrying mechanism in stacks-blockchain
for when bitcoin responds with connection refused on wallet creation? π€
The important thing here is that it seems that stacks-node-follower attempts to create the wallet before bitcoin finishes creating blocks.
this is very interesting.....I think we can address this on our end, to be fair though it's something we thought possible but had never seen it in practice.
You are correct though, the timing absolutely matters - we'll have to look at what we can do here to slow things down for folks with faster systems, haha. I'll read through the rest of your comments in a bit, but i think we should loop in @MarvinJanssen as well here to spitball some ideas.
thanks again for your help diagnosing this!
Hey, glad to help.
I've attached the full docker-compose
logfile in case this would help provide more clarity π
Outstanding, thanks! i'll do my best to have a fix for this by monday, but i've been swamped as of late so don't hold me to it.
No worries and thank you, I'd also love to get involved, as I'm currently learning all I can about the platform
Hey guys, ended up fixing it from docker-compose
by using the wonderful wait
utility from here: https://github.com/ufoscout/docker-compose-wait
Essencially waiting for bitcoin:18443
to become available and then waiting a bit longer just to make sure
My container docker-compose configuration looks like:
stacks-blockchain:
build:
context: ./provisioning/stacks-blockchain
restart: on-failure
environment:
STACKS_EVENT_OBSERVER: stacks-blockchain-api:3700
XBLOCKSTACK_DEBUG: 1
RUST_BACKTRACE: 1
WAIT_HOST_CONNECT_TIMEOUT: 6
WAIT_BEFORE: 5
WAIT_HOSTS: bitcoin:18443
WAIT_AFTER: 15
WAIT_SLEEP_INTERVAL: 1
ports:
- "20443:20443"
- "20444:20444"
depends_on:
- stacks-blockchain-api
networks:
- stacks-blockchain
I ended up wrapping blockstack/stacks-blockchain:develop-stretch
into my own dockerfile so I could copy the wait utility. Confirmed this works. Looking forward to the fix on stacks-blockchain but for now I've got my functional setup π
Included an overly simplified private testnet setup here: https://github.com/dorucioclea/stacks-dev-env
Thanks, and nice work!
i'm not sure we want to have a modified stacks-blockchain image as part of this repo, i've been looking at ways to incorporate what that "wait" function does without having to modify the underlying images.
I have 2 possible solutions, but need time to test - essentially a docker health check, or loop over nc
until bitcoin is up (replicating what you've done, but without having to create a new image)
bitcoin:18443
@MarvinJanssen thoughts on using the changes here from @dorucioclea vs. what i've added here: https://github.com/blockstack/stacks-local-dev/commit/958e78f71883319a6e3382bd6d0025a8630717f1
Tbh this looks nice to me, minimally invasive @wileyj great work π
@wileyj I think your approach makes the most sense for now.
Thanks all! i'm closing this for now - the image used here is a bit old, but once a new master branch is merged in stacks-blockchain (coming soon), we can update to use the latest
image for this configuration.
Hi there, I'm trying to use your setup for private testnet but unfortunately it is not working.
Initially I had an issue related to a wallet not being created which I fixed by adding this container execution:
What this does is create a wallet called
testwallet
in order to avoid the wallet not being created error which this previously gave meNext, I slightly modified the
bitcoin configuration
to look like thisNotably, the only change I did was to add
disiablewallet=0
Now I'm getting the error below:
A couple of questions here:
puppet-chain
version of bitcoin node but I noticed that this is not the version used for thestack-blockchain-api
scriptsblockstack/stacks-blockchain:feat-miner-control
- is my understanding correct that this is based off of a feature branch? Couldn't find that specific branch on the github repository. Are there plans for this to be kept up-to-date with the main stacks-blockchain image?private testnet
a good choice for contracts development and integration testing ? Or ismocknet
better suited for this scenario?Thank you