node halts [mainnet]

[deantchi]

Stacks node: 2.05.0.6.0

When syncing from genesis, the stacks node abruptly comes to a halt. Upon looking at the logs, it looks like there are some Atlas or BNS related errors. While I think simply restarting the node will get past this issue, I want to make sure this issue is reported so someone can take a look.

12/21/2022, 10:36:21 AM | Atlas: 1 attachment instances emitted from events
-- | --
  |   |   | 12/21/2022, 10:36:22 AM | Atlas: could not find a peer including attachment in its inventory
  |   |   | 12/21/2022, 10:36:22 AM | Atlas: re-enqueuing batch AttachmentsBatch { stacks_block_height: 88066, canonical_stacks_tip_height: Some(88066), index_block_hash: c6b9aff11a7a2c7a2d3b30a7878eb3a27e1291e6e203bc890f8c44fee19bdeb5, attachments_instances: {QualifiedContractIdentifier { issuer: StandardPrincipalData(SP000000000000000000002Q6VF78), name: ContractName("bns") }: {223943: be4b34eedd34bc6034fc6f2aa4466174e58a378d}}, retry_count: 1, retry_deadline: 1671647784 } for retry
2x |   |   | 12/21/2022, 10:36:26 AM | Handle HTTPRequest
  |   |   | 12/21/2022, 10:36:31 AM | Atlas: could not find a peer including attachment in its inventory
  |   |   | 12/21/2022, 10:36:31 AM | Atlas: re-enqueuing batch AttachmentsBatch { stacks_block_height: 88066, canonical_stacks_tip_height: Some(88066), index_block_hash: c6b9aff11a7a2c7a2d3b30a7878eb3a27e1291e6e203bc890f8c44fee19bdeb5, attachments_instances: {QualifiedContractIdentifier { issuer: StandardPrincipalData(SP000000000000000000002Q6VF78), name: ContractName("bns") }: {223943: be4b34eedd34bc6034fc6f2aa4466174e58a378d}}, retry_count: 2, retry_deadline: 1671647795 } for retry

[jcnelson]

Is the node still running? Can you attach gdb and give me stack trace for each thread?

[pseudozach]

@deantchi did restart help in your case? I'm having similar issue with same signature and I've restarted the node multiple times but it only advances 1-2 blocks before halting again.

stacks-blockchain      | WARN [1672979450.044097] [src/net/atlas/download.rs:436] [p2p-(0.0.0.0:20444,0.0.0.0:20443)] Atlas: could not find a peer including attachment in its inventory
stacks-blockchain      | INFO [1672979450.324735] [src/net/atlas/download.rs:218] [p2p-(0.0.0.0:20444,0.0.0.0:20443)] Atlas: re-enqueuing batch AttachmentsBatch { stacks_block_height: 89992, canonical_stacks_tip_height: Some(89992), index_block_hash: 349b0ce86939fbfdb24073868be450287bbd0fe014bcb6bbfae18b39d8d11157, attachments_instances: {QualifiedContractIdentifier { issuer: StandardPrincipalData(SP000000000000000000002Q6VF78), name: ContractName("bns") }: {227050: e454ab15ec9aee2bb3695a8db4e86a1074357bef}}, retry_count: 8, retry_deadline: 1672979751 } for retry

I can provide logs as needed just need instructions. Here are my stacks-blockchain docker logs: https://transfer.sh/fl0Ipb/sb.tar.gz

[deantchi]

@pseudozach I'm seeing the same behavior with reboots. I'm still trying to get thread traces from my affected node. We are running our node in a more restricted environment so getting GDB traces on the node is a challenge.

[pseudozach]

@pseudozach I'm seeing the same behavior with reboots. I'm still trying to get thread traces from my affected node. We are running our node in a more restricted environment so getting GDB traces on the node is a challenge.

in my case the root cause turned out to be foundation's btc node being overloaded. so I switched to my own btc node as burnchain node and issue seems to have resolved for now.

[jcnelson]

I think this is blocked on #3086. Assigning to @deantchi for now. Please feel free to re-assign or close as you see fit.

[planbetterHQ]

We're experiencing the same issue. Still no clue why the node halts from time to time.