neondatabase / autoscaling

Postgres vertical autoscaling in k8s
Apache License 2.0
144 stars 18 forks source link

neonvm-controller: Rework zap setup & disable sampling #947

Closed sharnoff closed 1 month ago

sharnoff commented 1 month ago

By default, zap enables "log sampling" when not in development mode. Unsurprisingly, we're finding that neonvm-controller logs are being dropped in production as a result of this.

I couldn't find a clean way to disable sampling without entirely constructing the zap.Logger from scratch, so that's what I did.

Notable changes:

More context here: https://neondb.slack.com/archives/C03TN5G758R/p1716775758112349?thread_ts=1716765298.436929 See also: #475

sharnoff commented 1 month ago

This is actually not quite working as intended. See e.g. stacktrace is included on error in this PR here, but not included on main here.

Will merge after that's working as expected. Hopefully it's not too bad.

sharnoff commented 1 month ago

Fixed in 887dea7f6ba6d058f0696cabe748cbaefa248879 + validated via CI logs that it's working as expected.