WAL connection_manager got stuck receiving no updates from storage broker

neondatabase / neon

Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, code-like database branching, and scale to zero.

https://neon.tech

Apache License 2.0

14.78k stars 429 forks source link

WAL connection_manager got stuck receiving no updates from storage broker #6838

Closed arssher closed 4 months ago

arssher commented 8 months ago

We've seen in the wild one case where $subject. We had checked that there had been updates, but pageserver wasn't receiving them. Need to find the cause and fix it. Probably a good first idea is to add more debugging to manager state: whether it sits in main select! loop or deeper.

Relevant ps logs: https://neonprod.grafana.net/goto/AEZ5V4TSg?orgId=1 https://neondb.slack.com/archives/C06KREAH31S

problame commented 4 months ago

This was because of the duplicate storage broker https://neondb.slack.com/archives/C06KREAH31S/p1708505829756859?thread_ts=1708429624.315409&cid=C06KREAH31S