search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.13k stars 3.81k forks source link

log: noisy logging during a drain operation #102339

Open nicktrav opened 1 year ago

nicktrav commented 1 year ago

Is your feature request related to a problem? Please describe.

A little UX papercut. When we’re draining, we log a TON of these, which just spams the logs and makes spotting issues during the drain difficult:

I230426 00:51:00.257800 536056919 1@sql/pgwire/server.go:745 ⋮ [T1,n9,client=‹10.142.1.109:37376›] 12674  rejecting new connection while server is draining
E230426 00:51:00.257890 536056919 1@server/server_sql.go:1750 ⋮ [T1,n9,client=‹10.142.1.109:37376›] 12675  serving SQL client conn: server is not accepting clients, try another node

Describe the solution you'd like

Consider logging less, e.g. using a log.Every.

Jira issue: CRDB-27401

rafiss commented 1 year ago

I think we can make the change, though one thing is that in the expected production setup, there is a load balancer in front of the cluster, and it should not direct any new connections to a node while it's draining. So in that case, seeing this log at all would actually be unexpected and a sign of misconfiguration.