Open knz opened 3 years ago
cc @logston I think you'll want to learn this
In fact as per @tbg we may need to do so urgently, because in multi-tenant individual tenants cannot set cluster settings. So if we want these logs they can't be controlled by cluster settings for now.
What we can always do is override individual cluster settings from the env in start-sql
.
What we can always do is override individual cluster settings from the env in start-sql.
Is this override mechanism already implemented? If so, where can I find more information about it?
Settings have a .Override
method, which is what I was referring to.
This isn't the right option if we start letting tenants listen to their settings table, of course. Then, nothing else would be required.
ok so if nothing else we can implement a new override mechanism in the mt start-sql
CLI flag handling. That'd be a temporary workaround, but it doesn't hurt.
Logging filters sound like a good strategy. That could also potentially allow end users to tune it such that events we don't traditionally think of as security events go to a custom sink for their own consumption.
Echoing Aaron's sentiment, moving towards logging filters and away from setting clusters sounds like a great approach. The usability improvement around managing logs in a single place (at log configuration) instead of managing knobs at multiple places will lead to a quality of life improvement for users. I also like your idea of an exclusion list too because it gives the end user more flexibility and control around what events go where.
We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to CockroachDB!
Today we have certain notable events for which we use a cluster setting to enable/disable the logging. This includes:
create_statistics
, controlled bysql.stats.post_events.enabled
(see #57877)client_connection_start
/client_connection_end
, controlled byserver.auth_log.sql_connections.enabled
(see #57839)client_session_end
,client_authorization_*
, controlled byserver.auth_log.sql_sessions.enabled
(see #57839 )These logging events are only logged conditionally because they are otherwise rather frequent, and we did not want them to "pollute" logs or incur excessive disk IOPS when logging to files.
However, this is inconvenient for CC logging, where we likely do want to see these events collected and instead filter them "elsewhere" (in a network log collector). For CC clusters, this is extra inconvenient because it requires some additional cluster settings to be set before a CC cluster becomes operational, which is error-prone (both due to human mistakes and additional software errors).
Instead, we could consider have the internal CockroachDB components log them unconditionally, and have a single general-purpose filtering option in the logging system:
The filter could be like a list of logging events to exclude, per sink. For example in YAML:
This additional mechanism would also make it possible for end-users to opt in and out of logging certain notable events without requiring us to add cluster settings for each of them every time.
@aaron-crl thoughts? @thtruo ?
Jira issue: CRDB-3472