Closed dcsan closed 2 years ago
dcsan
commented
2 years ago could this be caused by somehow my sync missing a few blocks? eg if my local machine had gone to sleep for a few blocks, do I now have a corrupted sync? eg it missed a chunk of incoming blocks, any future blocks are now invalid? in which case I can't really run this locally, it would have to be on a cloud machine that's always on... ?
dcsan
commented
2 years ago after a couple days running the logs look like this: I guess it's time to give up and nuke everything
➜ stacks-blockchain-docker git:(master) ✗ make logs
./manage.sh mainnet logs
Running: docker-compose --env-file .env -f ./configurations/common.yaml -f ./configurations/mainnet.yaml --profile stacks-blockchain logs -f
Attaching to stacks-blockchain, stacks-blockchain-api, postgres
postgres |
postgres | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres |
postgres | 2022-01-19 06:52:43.424 UTC [1] LOG: starting PostgreSQL 13.5 on aarch64-unknown-linux-musl, compiled by gcc (Alpine 10.3.1_git20211027) 10.3.1 20211027, 64-bit
postgres | 2022-01-19 06:52:43.424 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
postgres | 2022-01-19 06:52:43.424 UTC [1] LOG: listening on IPv6 address "::", port 5432
postgres | 2022-01-19 06:52:43.428 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres | 2022-01-19 06:52:43.443 UTC [23] LOG: database system was shut down at 2022-01-19 06:52:41 UTC
stacks-blockchain-api |
stacks-blockchain-api | > @blockstack/stacks-blockchain-api@1.0.0 start /app
stacks-blockchain-api | > node ./lib/index.js
stacks-blockchain-api |
stacks-blockchain-api | dotenv-flow: "PG_HOST" is already defined in `process.env` and will not be overwritten
stacks-blockchain-api | dotenv-flow: "PG_PORT" is already defined in `process.env` and will not be overwritten
stacks-blockchain-api | dotenv-flow: "PG_USER" is already defined in `process.env` and will not be overwritten
stacks-blockchain-api | dotenv-flow: "PG_PASSWORD" is already defined in `process.env` and will not be overwritten
postgres | 2022-01-19 06:52:43.459 UTC [1] LOG: database system is ready to accept connections
stacks-blockchain-api | dotenv-flow: "PG_DATABASE" is already defined in `process.env` and will not be overwritten
stacks-blockchain-api | dotenv-flow: "STACKS_CORE_EVENT_PORT" is already defined in `process.env` and will not be overwritten
stacks-blockchain-api | dotenv-flow: "STACKS_CORE_EVENT_HOST" is already defined in `process.env` and will not be overwritten
stacks-blockchain-api | dotenv-flow: "STACKS_BLOCKCHAIN_API_PORT" is already defined in `process.env` and will not be overwritten
stacks-blockchain-api | dotenv-flow: "STACKS_BLOCKCHAIN_API_HOST" is already defined in `process.env` and will not be overwritten
stacks-blockchain-api | dotenv-flow: "STACKS_CORE_RPC_HOST" is already defined in `process.env` and will not be overwritten
stacks-blockchain-api | dotenv-flow: "STACKS_CORE_RPC_PORT" is already defined in `process.env` and will not be overwritten
stacks-blockchain-api | dotenv-flow: "STACKS_CHAIN_ID" is already defined in `process.env` and will not be overwritten
stacks-blockchain-api | dotenv-flow: "GIT_TAG" is already defined in `process.env` and will not be overwritten
stacks-blockchain-api | {"message":"Notice: full BNS functionality requires 'BNS_IMPORT_DIR' to be set.","level":"warn","timestamp":"2022-01-19T06:52:52.159Z"}
stacks-blockchain-api | {"message":"Event observer listening at: http://0.0.0.0:3700","level":"info","timestamp":"2022-01-19T06:52:52.208Z"}
stacks-blockchain | INFO [1642575163.538754] [testnet/stacks-node/src/main.rs:81] [main] stacks-node 2.05.0.1.0-A (develop:c61a7da, release build, linux [x86_64])
stacks-blockchain | INFO [1642575163.540499] [testnet/stacks-node/src/main.rs:114] [main] Loading config at path /src/stacks-blockchain/Config.toml
stacks-blockchain | INFO [1642575163.854132] [testnet/stacks-node/src/run_loop/neon.rs:217] [main] Will run as a Follower node
stacks-blockchain | INFO [1642575163.855157] [testnet/stacks-node/src/run_loop/neon.rs:224] [main] Start syncing Bitcoin headers, feel free to grab a cup of coffee, this can take a while
stacks-blockchain | INFO [1642575166.196686] [src/burnchains/burnchain.rs:1271] [main] Syncing Bitcoin blocks: 77.4% (705233 to 707333 out of 719397)
stacks-blockchain | WARN [1642575687.852366] [src/chainstate/burn/operations/leader_block_commit.rs:328] [burnchain-db] Invalid commit tx: different output amounts for different PoX reward addresses
stacks-blockchain | WARN [1642575687.852708] [src/burnchains/burnchain.rs:704] [burnchain-db] Failed to parse leader block commit tx, txid: e3ea46f98358ab96da5c96c5ea352962545b6cde0e5007b259a76fa1846ce3c2, data: 2a47566f12c5ec2d9c32149aef1cbac4c0633abc39d1cf3620a74fd99c2bd666752010795bc3a64f756d3e26df2dcd54b7d6c21eeaae98f69cdb2b9045c50b06000ac59c000f000ac3d7008804, error: ParseErrorprogress?
stacks-blockchain | INFO [1642578907.540451] [src/chainstate/stacks/index/marf.rs:1185] [main] Batching insertions in MARF: 93% (311858 out of 335331)
stacks-blockchain | INFO [1642578909.783235] [src/chainstate/stacks/index/marf.rs:1185] [main] Batching insertions in MARF: 94% (315212 out of 335331)
stacks-blockchain | INFO [1642578911.983574] [src/chainstate/stacks/index/marf.rs:1185] [main] Batching insertions in MARF: 95% (318565 out of 335331)
stacks-blockchain | INFO [1642578914.128431] [src/chainstate/stacks/index/marf.rs:1185] [main] Batching insertions in MARF: 96% (321918 out of 335331)
stacks-blockchain | INFO [1642578916.280192] [src/chainstate/stacks/index/marf.rs:1185] [main] Batching insertions in MARF: 97% (325272 out of 335331)
stacks-blockchain | INFO [1642578918.434960] [src/chainstate/stacks/index/marf.rs:1185] [main] Batching insertions in MARF: 98% (328625 out of 335331)
stacks-blockchain | INFO [1642578920.590088] [src/chainstate/stacks/index/marf.rs:1185] [main] Batching insertions in MARF: 99% (331978 out of 335331)
stacks-blockchain | INFO [1642578963.949594] [src/chainstate/stacks/db/mod.rs:1414] [main] Computed Clarity state genesis, root_hash: 9653c92b1ad726e2dc17862a3786f7438ab9239c16dd8e7aaba8b0b5c34b52af
stacks-blockchain | WARN [1642579056.929387] [src/chainstate/burn/operations/leader_block_commit.rs:672] [chains-coordinator] Invalid block commit: missed target block, intended_modulus: 3, actual_modulus: 4, block_height: 666179, apparent_sender: 1DkfGWddhyqRQett2hzevqeS7EwRzaD1Vr
stacks-blockchain | WARN [1642579056.930244] [src/chainstate/burn/db/processing.rs:61] [chains-coordinator] REJECTED(666179) leader block commit 4b3cb35f2f6b30d998a0d02ff2d63715de6ddb20a74dd1c3147bb618e82f53d3 at 666179,384: BlockCommitNoParent
stacks-blockchain | WARN [1642579056.935003] [src/chainstate/burn/operations/leader_block_commit.rs:672] [chains-coordinator] Invalid block commit: missed target block, intended_modulus: 2, actual_modulus: 4, block_height: 666179, apparent_sender: 19F5dMCEJuBzfLRBbXj3Tk9FUG2kFwyCXn
stacks-blockchain | WARN [1642579056.936111] [src/chainstate/burn/db/processing.rs:61] [chains-coordinator] REJECTED(666179) leader block commit 1f69ba25379cd0ec3f77ac57486a4b4a97bc8d1f0faa3b897ed9be331743544f at 666179,557: BlockCommitNoParent
stacks-blockchain | INFO [1642579057.022293] [src/chainstate/burn/db/sortdb.rs:3699] [chains-coordinator] ACCEPTED(666179) leader block commit 2730f2ab06f5bc51a4114525fdbf0ba4e77b3d22784c6d10f54f58d8d4af93d1 at 666179,86, apparent_sender: 1BKKtxfH8ouwuQGeSS9XUzUNQS9hknSrQp
stacks-blockchain | INFO [1642579057.022713] [src/chainstate/burn/db/sortdb.rs:3699] [chains-coordinator] ACCEPTED(666179) leader block commit 7ee58bed330e5e55093f87a869f5b23629aab4fbbe22174482d37324fa93b963 at 666179,87, apparent_sender: 16sAXi1jxhxKCfY84hubDdnFNaAhqd5t49
stacks-blockchain | INFO [1642579057.023178] [src/chainstate/burn/db/sortdb.rs:3699] [chains-coordinator] ACCEPTED(666179) leader block commit 655c9b18eed30ef3be76f460afca9d18401694e3be7f71af8acf8cb48f77c27b at 666179,99, apparent_sender: 1Frf72kABGpEwEmPUGwsUbvJ2odenr3TRg
stacks-blockchain | INFO [1642579057.023596] [src/chainstate/burn/db/sortdb.rs:3699] [chains-coordinator] ACCEPTED(666179) leader block commit 4d35ef875a1e435c47ee5961aa5247872a8a547a6772b01e41deab41423f9b83 at 666179,100, apparent_sender: 17RfeXjpSQzqWrsEXd71eVWP4d28t1wKis
stacks-blockchain | INFO [1642579057.023949] [src/chainstate/burn/db/sortdb.rs:3699] [chains-coordinator] ACCEPTED(666179) leader block commit eb4f5b835929cf35c384bfe280cba0f164b0d0d44f8ba9e1ba191c2bbea1d22d at 666179,208, apparent_sender: 1K5YAWU9TejYXb2x5YfQD4TL6DPkRPP7KL
stacks-blockchain | WARN [1642579094.764102] [src/chainstate/burn/db/processing.rs:61] [chains-coordinator] REJECTED(666377) leader block commit b0b912ef2582c827072b022f47c188037762f3a47de3c6497268b48b6cd66a2d at 666377,313: BlockCommitNoParent
stacks-blockchain | WARN [1642579094.767453] [src/chainstate/burn/operations/leader_block_commit.rs:672] [chains-coordinator] Invalid block commit: missed target block, intended_modulus: 1, actual_modulus: 2, block_height: 666377, apparent_sender: 1K1rSXoxAaJPQdWpm6XgZ6xwzkikxmWonW
stacks-blockchain | WARN [1642579094.767820] [src/chainstate/burn/db/processing.rs:61] [chains-coordinator] REJECTED(666377) leader block commit d53763100712f0872ba7c5f171f1b87ee0927bb8379e22527d8a09b4130359b7 at 666377,660: BlockCommitNoParent
stacks-blockchain | INFO [1642579094.807685] [src/chainstate/burn/db/sortdb.rs:3699] [chains-coordinator] ACCEPTED(666377) leader block commit 6c1f46b285ed9f351ff7d0136557ad5ba32523f5ebcd3575b4b9236b0b5fa2d7 at 666377,7, apparent_sender: 1Frf72kABGpEwEmPUGwsUbvJ2odenr3TRg
stacks-blockchain | INFO [1642579094.808007] [src/chainstate/burn/db/sortdb.rs:3699] [chains-coordinator] ACCEPTED(666377) leader block commit 773436b103d615b24072ca087c4c5b652f0e75193a1a4d25083ceb309ca9e74b at 666377,112, apparent_sender: 1DkfGWddhyqRQett2hzevqeS7EwRzaD1Vrnot sure if all these REJECTED are normal or not...
dcsan
commented
2 years ago stacks-blockchain | INFO [1642579876.553165] [testnet/stacks-node/src/run_loop/neon.rs:217] [main] Will run as a Follower node
stacks-blockchain | INFO [1642579876.554080] [testnet/stacks-node/src/run_loop/neon.rs:224] [main] Start syncing Bitcoin headers, feel free to grab a cup of coffee, this can take a while
stacks-blockchain | INFO [1642579877.331406] [src/burnchains/burnchain.rs:1271] [main] Syncing Bitcoin blocks: 7.9% (668151 to 670251 out of 719405)7.9% - looks like its progressing...
dcsan
commented
2 years ago seems like maybe flaky wifi connection causes everything to stop
stacks-blockchain | ERRO [1642580466.073996] [src/burnchains/bitcoin/network.rs:208] [burnchain-downloader] Failed to connect to peer 34.207.108.183:8333: connection to peer could not be (re-)established
stacks-blockchain | ERRO [1642580468.216083] [src/burnchains/bitcoin/network.rs:208] [burnchain-downloader] Failed to connect to peer 34.207.108.183:8333: connection to peer could not be (re-)established
stacks-blockchain | ERRO [1642580472.825038] [src/burnchains/bitcoin/network.rs:208] [burnchain-downloader] Failed to connect to peer 34.207.108.183:8333: connection to peer could not be (re-)established
stacks-blockchain | WARN [1642580472.825491] [src/burnchains/bitcoin/network.rs:222] [burnchain-downloader] Connection broken; retrying in 10.293946065265517 sec...
stacks-blockchain | ERRO [1642580483.123755] [src/burnchains/bitcoin/network.rs:208] [burnchain-downloader] Failed to connect to peer 34.207.108.183:8333: connection to peer could not be (re-)established
stacks-blockchain | WARN [1642580483.123952] [src/burnchains/bitcoin/network.rs:222] [burnchain-downloader] Connection broken; retrying in 29.935134041694297 sec...OK back to this does not contain a parent block error
stacks-blockchain-api | {"level":"error","message":"Error processing core node block message DB does not contain a parent block at height 1 with index_hash 0x918697ef63f9d8bdf844c3312b299e72a231cde542f3173f7755bb8c1cdaf3a7","stack":"Error: DB does not contain a parent block at height 1 with index_hash 0x918697ef63f9d8bdf844c3312b299e72a231cde542f3173f7755bb8c1cdaf3a7\n at PgDataStore.handleReorg (/app/src/datastore/postgres-store.ts:2303:15)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at /app/src/datastore/postgres-store.ts:1048:7\n at /app/src/datastore/postgres-store.ts:716:24\n at PgDataStore.query (/app/src/datastore/postgres-store.ts:702:22)\n at PgDataStore.queryTx (/app/src/datastore/postgres-store.ts:713:12)\n at PgDataStore.update (/app/src/datastore/postgres-store.ts:1046:5)\n at handleBlockMessage (/app/src/event-stream/event-server.ts:322:3)\n at run (/app/node_modules/p-queue/dist/index.js:163:29)","timestamp":"2022-01-19T09:22:13.461Z"}from the README
If the Stacks blockchain is no longer syncing blocks, and the API reports an error similar to this:
Error processing core node block message DB does not contain a parent block at height 1970 with index_hash 0x3367f1abe0ee35b10e77fbcaa00d3ca452355478068a0662ec492bb30ee0f13e",The API (and by extension the DB) is out of sync with the blockchain. The only known method to recover is to resync from genesis (event-replay may work, but in all likliehood will restore data to the same broken state).
so, I've been going around in circles with this for a few days. It seems pointless to try and reset and resync and start the whole process from scratch again.
Interested why it's so troublesome and what other variables there are I could try to work with.
dcsan
commented
2 years ago getting pages of repeats of this error:
stacks-blockchain-api | {"txid":"0x5d28dd47bcd92716de43161104129e242f168608d64f96b54303966611683086","in_microblock":false,"stacks_height":2,"level":"info","message":"Transaction confirmed","timestamp":"2022-01-19T09:26:33.372Z"}
stacks-blockchain-api | {"level":"error","message":"Error processing core node block message DB does not contain a parent block at height 1 with index_hash 0x918697ef63f9d8bdf844c3312b299e72a231cde542f3173f7755bb8c1cdaf3a7","stack":"Error: DB does not contain a parent block at height 1 with index_hash 0x918697ef63f9d8bdf844c3312b299e72a231cde542f3173f7755bb8c1cdaf3a7\n at PgDataStore.handleReorg (/app/src/datastore/postgres-store.ts:2303:15)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at /app/src/datastore/postgres-store.ts:1048:7\n at /app/src/datastore/postgres-store.ts:716:24\n at PgDataStore.query (/app/src/datastore/postgres-store.ts:702:22)\n at PgDataStore.queryTx (/app/src/datastore/postgres-store.ts:713:12)\n at PgDataStore.update (/app/src/datastore/postgres-store.ts:1046:5)\n at handleBlockMessage (/app/src/event-stream/event-server.ts:322:3)\n at run (/app/node_modules/p-queue/dist/index.js:163:29)","timestamp":"2022-01-19T09:26:33.377Z"}
stacks-blockchain-api | {"level":"error","message":"error processing core-node /new_block: Error: DB does not contain a parent block at height 1 with index_hash 0x918697ef63f9d8bdf844c3312b299e72a231cde542f3173f7755bb8c1cdaf3a7 DB does not contain a parent block at height 1 with index_hash 0x918697ef63f9d8bdf844c3312b299e72a231cde542f3173f7755bb8c1cdaf3a7","stack":"Error: DB does not contain a parent block at height 1 with index_hash 0x918697ef63f9d8bdf844c3312b299e72a231cde542f3173f7755bb8c1cdaf3a7\n at PgDataStore.handleReorg (/app/src/datastore/postgres-store.ts:2303:15)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at /app/src/datastore/postgres-store.ts:1048:7\n at /app/src/datastore/postgres-store.ts:716:24\n at PgDataStore.query (/app/src/datastore/postgres-store.ts:702:22)\n at PgDataStore.queryTx (/app/src/datastore/postgres-store.ts:713:12)\n at PgDataStore.update (/app/src/datastore/postgres-store.ts:1046:5)\n at handleBlockMessage (/app/src/event-stream/event-server.ts:322:3)\n at run (/app/node_modules/p-queue/dist/index.js:163:29)","timestamp":"2022-01-19T09:26:33.377Z"}
stacks-blockchain | ERRO [1642584393.379840] [testnet/stacks-node/src/event_dispatcher.rs:151] [chains-coordinator] Event dispatcher: POST stacks-blockchain-api:3700/http://stacks-blockchain-api:3700/new_block failed with error Response { status: InternalServerError, headers: Headers { headers: {HeaderName("content-length"): [HeaderValue { inner: "12" }], HeaderName("x-powered-by"): [HeaderValue { inner: "Express" }], HeaderName("content-type"): [HeaderValue { inner: "application/json; charset=utf-8" }], HeaderName("date"): [HeaderValue { inner: "Wed, 19 Jan 2022 09:26:33 GMT" }], HeaderName("connection"): [HeaderValue { inner: "keep-alive" }], HeaderName("keep-alive"): [HeaderValue { inner: "timeout=5" }], HeaderName("etag"): [HeaderValue { inner: "W/\"c-3Rk1bge0s6VuJbi+S2m2iU0UGdY\"" }]} }, version: None, sender: Some(Sender { .. }), receiver: Receiver { .. }, body: Body { reader: "<hidden>", length: Some(12) }, local: TypeMap }
stacks-blockchain-api | {"req":{"url":"/new_block","headers":{"host":"stacks-blockchain-api:3700","content-length":"1780","date":"Wed, 19 Jan 2022 09:26:33 GMT","content-type":"application/json"},"method":"POST","httpVersion":"1.1","originalUrl":"/new_block","query":{}},"res":{"statusCode":500},"responseTime":27,"level":"error","message":"HTTP POST /new_block","timestamp":"2022-01-19T09:26:33.380Z"}
wileyj
commented
2 years ago a lot to unpack here - the first logs you shared don't indicate any sort of problem.
stacks-blockchain | WARN [1642535941.209026] [src/chainstate/burn/operations/leader_block_commit.rs:711]
[chains-coordinator] Invalid block-commit: bad PoX: BlockCommitBadOutputs, apparent_sender:
19W6NDuEN2PB1NZZc6e5hMzujQWu9d68jthese are normal log output.
the second message -again, that's normal output for the startup sequence. without knowing the uptime of the instances, it's impossible to tell what's going on here other than a normal startup at Wednesday, January 19, 2022 7:01:27.852 AM GMT.
The stacks-blockchain startup can take a little while, and if the bitcoin headers are already synced - it won't log much unless you have DEBUG env var turned on for the container.
Syncing the bitcoin headers, you can see logs like this:
stacks-blockchain | WARN [1642580472.825491] [src/burnchains/bitcoin/network.rs:222] [burnchain-downloader] Connection broken; retrying in 10.293946065265517 sec...
usually not a problem, just means the bitcoin host didn't respond in time and will be retried- this function will run until it's done. nothing you can do here outside of using your own local bitcoin node.
this error is the main issue:
stacks-blockchain-api | {"level":"error","message":"Error processing core node block message DB does not contain a parent block at height 1 with index_hash 0x918697ef63f9d8bdf844c3312b299e72a231cde542f3173f7755bb8c1cdaf3a7","stack":"Error: DB does not contain a parent block at height 1 with index_hash 0x918697ef63f9d8bdf844c3312b299e72a231cde542f3173f7755bb8c1cdaf3a7\n at PgDataStore.handleReorg (/app/src/datastore/postgres-store.ts:2303:15)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at /app/src/datastore/postgres-store.ts:1048:7\n at /app/src/datastore/postgres-store.ts:716:24\n at PgDataStore.query (/app/src/datastore/postgres-store.ts:702:22)\n at PgDataStore.queryTx (/app/src/datastore/postgres-store.ts:713:12)\n at PgDataStore.update (/app/src/datastore/postgres-store.ts:1046:5)\n at handleBlockMessage (/app/src/event-stream/event-server.ts:322:3)\n at run (/app/node_modules/p-queue/dist/index.js:163:29)","timestamp":"2022-01-19T09:22:13.461Z"}the only reason this can happen is if data was partially written to the database by the api, usually with a shutdown event (stopping the APi before the stacks-blockchain can produce this more reliably).
when this happens, the only recourse is to ./manage.sh mainnet reset followed by resync ./manage.sh mainnet start
wileyj
commented
2 years ago if, after resetting the persistent data and resyncing, you still see issues - ping me , we may need to look more closely at your setup
dcsan
commented
2 years ago I've done that last give up and start again step so many times now I made a makefile for it 👿
FYI I was using ./manage.sh mainnet restart and not start as you had above.
I'll run it one more time
FYI I'm running on a shiny M1 mac with:
$ docker -v
Docker version 20.10.8, build 3967b7d
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b949a8438443 blockstack/stacks-blockchain:2.05.0.1.0-A "/bin/stacks-node st…" 3 minutes ago Up 3 minutes 0.0.0.0:20443-20444->20443-20444/tcp stacks-blockchain
30ea82b72b8b blockstack/stacks-blockchain-api:0.71.2 "docker-entrypoint.s…" 3 minutes ago Up 3 minutes 0.0.0.0:3999->3999/tcp stacks-blockchain-api
6ef9d3b889d7 postgres:13-alpine "docker-entrypoint.s…" 3 minutes ago Up 3 minutes 0.0.0.0:6543->5432/tcp postgres
$ docker stats
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
b949a8438443 stacks-blockchain 103.76% 20.51MiB / 1.942GiB 1.03% 64.5MB / 1.53MB 9.81MB / 0B 6
30ea82b72b8b stacks-blockchain-api 0.55% 319.2MiB / 1.942GiB 16.05% 1.86MB / 3.67MB 50.6MB / 16.4kB 24
6ef9d3b889d7 postgres 0.13% 48.1MiB / 1.942GiB 2.42% 3.64MB / 55.1kB 22.6MB / 8.19kB 8
down:
./manage.sh mainnet down
reset:
./manage.sh mainnet reset
start:
./manage.sh mainnet start
restart:
./manage.sh mainnet restart
logs:
./manage.sh mainnet logs
start-again: down reset start logstrying this:
$ make start-again
wileyj
commented
2 years ago The problem is the M1. https://github.com/blockstack/stacks-blockchain-docker#macos-with-an-m1-processor-is-not-recommended-for-this-repo
Docker for Mac on an M1 has extremely poor I/O, there are a lot of issues filed about this, but so far nothing reliably works. It works fine on other ARM chips, it's all in how docker for mac handles the overlay fs.
Best option is to run from source, or use another x86_64 machine
wileyj
commented
2 years ago wileyj
commented
2 years ago you can try this, but the results from my tests were hit or miss: https://github.com/docker/roadmap/issues/7#issuecomment-975447079
dcsan
commented
2 years ago ok so i had set it up earlier today, seems to be in the same situation as all the other times, just a ton of errors spewing out. so your suggestions above are:
don't use this dockerized repo - try again from source repos, which probably means going down to the metal and updating XCode and digging through other requirements to get a system that fits the right profile. Given (2) and that I'm on a Mac M1 ... if it doesn't work with dockerized VM why would it work with raw OS access? seems like a way to spend hours repeating the same stuff and hoping for a different result.
you think it might be a docker issue for M1 macs?
docker/M1 was definitely a problem just after the apple silicon shipped, but i thought docker had stabilized a bit.
This stuff with does not contain a parent block does that really seem like a docker issue?
I've seen people in STX discord trying to run on an M1 with other problems. Maybe it just doesn't work?
If thats the case its probably better to just give up on a mac, and try to get this all working on a DO/GCP/AWS linux instance, and give up messing around with docker.
What's the preferred distro that is most likely to work?
stacks-blockchain-api | {"txid":"0x5d28dd47bcd92716de43161104129e242f168608d64f96b54303966611683086","in_microblock":false,"stacks_height":2,"level":"info","message":"Transaction confirmed","timestamp":"2022-01-20T08:36:07.472Z"}
stacks-blockchain-api | {"level":"error","message":"Error processing core node block message DB does not contain a parent block at height 1 with index_hash 0x918697ef63f9d8bdf844c3312b299e72a231cde542f3173f7755bb8c1cdaf3a7","stack":"Error: DB does not contain a parent block at height 1 with index_hash 0x918697ef63f9d8bdf844c3312b299e72a231cde542f3173f7755bb8c1cdaf3a7\n at PgDataStore.handleReorg (/app/src/datastore/postgres-store.ts:2303:15)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at /app/src/datastore/postgres-store.ts:1048:7\n at /app/src/datastore/postgres-store.ts:716:24\n at PgDataStore.query (/app/src/datastore/postgres-store.ts:702:22)\n at PgDataStore.queryTx (/app/src/datastore/postgres-store.ts:713:12)\n at PgDataStore.update (/app/src/datastore/postgres-store.ts:1046:5)\n at handleBlockMessage (/app/src/event-stream/event-server.ts:322:3)\n at run (/app/node_modules/p-queue/dist/index.js:163:29)","timestamp":"2022-01-20T08:36:07.479Z"}
stacks-blockchain-api | {"level":"error","message":"error processing core-node /new_block: Error: DB does not contain a parent block at height 1 with index_hash 0x918697ef63f9d8bdf844c3312b299e72a231cde542f3173f7755bb8c1cdaf3a7 DB does not contain a parent block at height 1 with index_hash 0x918697ef63f9d8bdf844c3312b299e72a231cde542f3173f7755bb8c1cdaf3a7","stack":"Error: DB does not contain a parent block at height 1 with index_hash 0x918697ef63f9d8bdf844c3312b299e72a231cde542f3173f7755bb8c1cdaf3a7\n at PgDataStore.handleReorg (/app/src/datastore/postgres-store.ts:2303:15)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at /app/src/datastore/postgres-store.ts:1048:7\n at /app/src/datastore/postgres-store.ts:716:24\n at PgDataStore.query (/app/src/datastore/postgres-store.ts:702:22)\n at PgDataStore.queryTx (/app/src/datastore/postgres-store.ts:713:12)\n at PgDataStore.update (/app/src/datastore/postgres-store.ts:1046:5)\n at handleBlockMessage (/app/src/event-stream/event-server.ts:322:3)\n at run (/app/node_modules/p-queue/dist/index.js:163:29)","timestamp":"2022-01-20T08:36:07.479Z"}
stacks-blockchain | ERRO [1642667767.481651] [testnet/stacks-node/src/event_dispatcher.rs:151] [chains-coordinator] Event dispatcher: POST stacks-blockchain-api:3700/http://stacks-blockchain-api:3700/new_block failed with error Response { status: InternalServerError, headers: Headers { headers: {HeaderName("date"): [HeaderValue { inner: "Thu, 20 Jan 2022 08:36:07 GMT" }], HeaderName("connection"): [HeaderValue { inner: "keep-alive" }], HeaderName("keep-alive"): [HeaderValue { inner: "timeout=5" }], HeaderName("etag"): [HeaderValue { inner: "W/\"c-3Rk1bge0s6VuJbi+S2m2iU0UGdY\"" }], HeaderName("content-length"): [HeaderValue { inner: "12" }], HeaderName("content-type"): [HeaderValue { inner: "application/json; charset=utf-8" }], HeaderName("x-powered-by"): [HeaderValue { inner: "Express" }]} }, version: None, sender: Some(Sender { .. }), receiver: Receiver { .. }, body: Body { reader: "<hidden>", length: Some(12) }, local: TypeMap }
stacks-blockchain-api | {"req":{"url":"/new_block","headers":{"host":"stacks-blockchain-api:3700","content-length":"1780","date":"Thu, 20 Jan 2022 08:36:07 GMT","content-type":"application/json"},"method":"POST","httpVersion":"1.1","originalUrl":"/new_block","query":{}},"res":{"statusCode":500},"responseTime":32,"level":"error","message":"HTTP POST /new_block","timestamp":"2022-01-20T08:36:07.482Z"}
^CERROR: Aborting.for others that have this working, I'm wondering did it "just work" or did you spend days messing around with parameters or editing docker files? I don't really see a lot of options here. It either works, or not. Maybe the last person to run this was before the 2.05 upgrade? I've definitely seen stuff in the stx ecosystem that was just completely broken but I was the first person to trip up along the road less travelled
wileyj
commented
2 years ago I think you're missing the point - the problem with this is not the docker-compose or the scripts etc. if you create an x86 VM, clone this repo and run it....it will work without updating any files.
The issue is not docker, it's how M1 macs implement the overlayfs for their virtualization. It's a known limitation for docker for mac on M1, many issues have been filed but as of today there is simply no answer for it.
These are the exact same issues i was facing using an M1 mac until i started digging in to understand the overlayfs issues - there are possible workarounds, but they're not pretty, and not simple.
Updating Xcode shouldn't be necessary, to run the stacks-blockchain you can grab a binary using the markdown i linked above, or compile your own from the source repo https://github.com/blockstack/stacks-blockchain/
if you really want to use Docker (and have a reliable and working setup), today Mac M1 is simply not an option due to the overlayfs issues in the current releases and the need for a lot of I/O.
Alternatively, you can run stacks/postgres/api using docker, but you'd have to write to ephemeral storage which would honestly not be a good idea.
dcsan
commented
2 years ago It's a known limitation for docker for mac on M1 These are the exact same issues i was facing using an M1 mac
Got it. we might want to add that to the README as most devs use macs, and recently M1s I personally knew there were M1 docker issues months back but I thought most were resolved now... I guess I need to track all the docker issues more closely. The error logs don't seem to give any clue that's the issue.
If i had known in advance maybe I would have spun up a DO droplet or something, now I'm 3 days of sunk costs in ...
wileyj
commented
2 years ago It's a known limitation for docker for mac on M1 These are the exact same issues i was facing using an M1 mac
Got it. we might want to add that to the README as most devs use macs, and recently M1s I personally knew there were M1 docker issues months back but I thought most were resolved now... I guess I need to track all the docker issues more closely. The error logs don't seem to give any clue that's the issue.
If i had known in advance maybe I would have spun up a DO droplet or something, now I'm 3 days of sunk costs in ...
yea, i get that - i'll take from your experience that the text in the readme isn't good enough and will make it more prominent with a link to some issues/workarounds (if someone's feeling bold)
dcsan
commented
2 years ago After your comment I just looked at the README again and it has the M1 warning at the start. I never noticed that and I must have read it a few times... ugh. Maybe I did the first checkout over two weeks ago
Perhaps we can do a cpu detection/warning inside the shell file?
wileyj
commented
2 years ago After your comment I just looked at the README again and it has the M1 warning at the start. I never noticed that and I must have read it a few times... ugh. Maybe I did the first checkout over two weeks ago
Perhaps we can do a cpu detection/warning inside the shell file?
yea, that could work for sure. i'll take a look, but PR's also welcome
dcsan
commented
2 years ago https://github.com/stacks-network/stacks-blockchain-docker/pull/54
OK PR submitted. hope the warning isn't too obnoxious but better than than miss it 🥴
Not sure where is the best place to post - just a question here.
I'm trying to get a local stacks node syncing. I've been running it for a day or so and it was erroring this AM on waking up. I just get a spew of errors:
and way back I saw one of these:
It's a drag to just nuke the whole thing and wait another 24 hours without any idea what the problem cause is... I've poked around the postgres DB and there are all the tables created, but mostly empty tables.
I saw #52 but not sure what else I can do. https://github.com/blockstack/stacks-blockchain-docker#workarounds-to-potential-issues
Any suggestions on debugging a local node sync?