Persist Insights to help users correlate problematic signals over time

[koorosh]

Is your feature request related to a problem? Please describe. Currently insights for SQL statements and transactions stored in memory only and this:

• doesn't allow to track historical data for insights as far as it has eviction policy that cleans up cache periodically
• after cluster restarts (relevant to tenant clusters), Insights are not available then;
• querying insights for large clusters becomes expensive as far as it should fan out request for every node in the cluster.

Describe the solution you'd like

• Periodically preserve insights in system table. It should use the same logic of flashing data to system table as for system.transaction_statistics and system.statement_activity tables;
• Fallback to in-memory option in case requested insights haven't been flashed to system table yet.

Jira issue: CRDB-28612

Epic CC-28354

[blathers-crl[bot]]

Hello, I am Blathers. I am here to help you get the issue triaged.

I have CC'd a few people who may be able to assist you:

• @cockroachdb/sql-foundations (found keywords: SQL statement)

If we have not gotten back to your issue within a few business days, you can try the following:

• Join our community slack channel and ask on #cockroachdb.
• Try find someone from here if you know they worked closely on the area and CC them.

_{:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}