search

stacks-network / sbtc-developer-release

sBTC primitives, signer components, helper tools
https://sbtc.tech
MIT License
1.98k stars 23 forks source link

[Devenv] Stacks node fails after running for some hours #342

Open radicleart opened 1 year ago

radicleart commented 1 year ago

Expected Behavior

The devenv and the stacks node keeps running.

Actual Behavior

The devenv environment bootstraps as expected. I'm able to see the deployed sBTC asset contract. I'm able to deploy contracts. But after some number of hours the stacks/postgres node fails. Contracts that had been deployed are no longer available. The postgres logs reports non-existent relationships;

docker logs postgres 2>&1 | grep -A 4 FATAL | more

$ docker logs postgres 2>&1 | grep -A 3 FATAL | more
2023-11-01 04:07:49.221 UTC [5227] FATAL:  expected SASL response, got message type 0
2023-11-01 04:07:49.488 UTC [5225] ERROR:  relation "burnchain_rewards" does not exist at character 17
2023-11-01 04:07:49.488 UTC [5225] STATEMENT:
            UPDATE burnchain_rewards
--
2023-11-01 05:37:56.247 UTC [5670] FATAL:  unsupported frontend protocol 1234.5632: server supports 3.0 to 3.0
2023-11-01 05:37:56.315 UTC [5666] ERROR:  relation "chain_tip" does not exist at character 119
2023-11-01 05:37:56.315 UTC [5666] STATEMENT:  SELECT block_height, index_block_hash, block_hash, microblock_hash, microblock_sequence, burn_block_height
          FROM chain_tip
--
2023-11-01 05:37:56.577 UTC [5671] FATAL:  password authentication failed for user "postgres"
2023-11-01 05:37:56.577 UTC [5671] DETAIL:  Connection matched pg_hba.conf line 100: "host all all all scram-sha-256"
2023-11-01 05:37:56.727 UTC [5667] ERROR:  relation "burnchain_rewards" does not exist at character 17
2023-11-01 05:37:56.727 UTC [5667] STATEMENT:

Steps to Reproduce

Environment is Ubuntu

./build.sh

leave running several hours.

Note this behaviour doesn't exist with clarinet integrate.

Note running down.sh / up.sh seems to bring it back to correct state.

Note: all docker containers remain running (stacks/postgres is erroring internally)

1c3cb8baadfa   sbtc-bridge-web:latest    "docker-entrypoint.s…"   14 hours ago   Up 14 hours   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp                                                                              sbtc-bridge-web
6c4074dc6eaa   sbtc-bridge-api:latest    "docker-entrypoint.s…"   14 hours ago   Up 14 hours   0.0.0.0:3010->3010/tcp, :::3010->3010/tcp, 7010/tcp                                                                    sbtc-bridge-api
2f76ec65340d   sbtc:latest               "entrypoint"             14 hours ago   Up 14 hours                                                                                                                          sbtc
99c04219eea9   stacks-explorer           "docker-entrypoint.s…"   14 hours ago   Up 14 hours   3020/tcp, 0.0.0.0:3020->3000/tcp, :::3020->3000/tcp                                                                    stacks-explorer
4a6edb0aa31b   mempool/frontend:latest   "/patch/entrypoint.s…"   14 hours ago   Up 14 hours   80/tcp, 0.0.0.0:8083->8083/tcp, :::8083->8083/tcp                                                                      mempool-web
79e80809df0e   stacks-api:latest         "docker-entrypoint.s…"   14 hours ago   Up 14 hours   0.0.0.0:3700->3700/tcp, :::3700->3700/tcp, 0.0.0.0:3999->3999/tcp, :::3999->3999/tcp                                   stacks-api
31938fcef5e7   mempool/backend:latest    "docker-entrypoint.s…"   14 hours ago   Up 14 hours   0.0.0.0:8999->8999/tcp, :::8999->8999/tcp                                                                              mempool-api
d77cd8b12c01   stacks:latest             "/bin/entrypoint.sh"     14 hours ago   Up 14 hours   0.0.0.0:20443-20444->20443-20444/tcp, :::20443-20444->20443-20444/tcp                                                  stacks
56f1ed6fdfdc   electrs:latest            "./entrypoint.sh"        14 hours ago   Up 14 hours   0.0.0.0:3002->3002/tcp, :::3002->3002/tcp, 0.0.0.0:60401->60401/tcp, :::60401->60401/tcp                               electrs
c74beb94badd   miner:latest              "/bin/entrypoint.sh"     14 hours ago   Up 14 hours                                                                                                                          miner
7e22043eb89e   mariadb:10.5.21           "docker-entrypoint.s…"   14 hours ago   Up 14 hours   0.0.0.0:3306->3306/tcp, :::3306->3306/tcp                                                                              mempool-db
9ded2e669b1a   postgres:15-alpine        "docker-entrypoint.s…"   14 hours ago   Up 14 hours   0.0.0.0:5432->5432/tcp, :::5432->5432/tcp                                                                              postgres
ce48b41881f8   mongo:6.0                 "docker-entrypoint.s…"   14 hours ago   Up 14 hours   0.0.0.0:27017->27017/tcp, :::27017->27017/tcp                                                                          mongodb
9048f42f8d5b   bitcoin:latest            "entrypoint.sh"          14 hours ago   Up 14 hours   0.0.0.0:18433->18433/tcp, :::18433->18433/tcp, 0.0.0.0:18443-18444->18443-18444/tcp, :::18443-18444->18443-18444/tcp   bitcoin
radicleart commented 1 year ago

Some discussion

Trying PG_DATABASE=postgres?sslmode=disable" on the postgres container.

radicleart commented 11 months ago

@wileyj, wondering if you've seen an issue like this before or if there may be a known fix? cc @AshtonStephens