scoutapp / roadmap

The public roadmap for Scout application monitoring.
https://scoutapp.com
16 stars 2 forks source link

Database Monitoring #8

Closed itsderek23 closed 6 years ago

itsderek23 commented 7 years ago

Complete this form for early access to database monitoring.


Databases are a common controllable bottleneck for many applications. Poor query performance can cascade into impacting much of an application.

This is an attempt to provide a database-centric view of an app's health within Scout.

Click the image below for a full-size screenshot:

database_ui

The fundamentals

The special sauce

It's easy to unintentionally commit a change to application code that harms db performance. Constantly analyze DB performance for events like the following:

These should connect directly to the source code location and the developers behind them so it's easy to identify someone to fix a problem.

Early Access

Complete this form for early access to database monitoring.

sixfeetover commented 7 years ago

This looks awesome. I'd love to see support for SQL Server (direct db adapter: https://github.com/rails-sqlserver/tiny_tds) as well! Any chance of adding that, or providing instructions on how to add a new database for PR purposes?

schneems commented 7 years ago

When showing slow queries it would be helpful to show an example of what inputs that query takes. In Rails all queries will show with params placeholders ? and the actual expensive query might only trigger on a specific set of params.

On that same note it could be cool if we could also get some kind of EXPLAIN output as well. Raw explain output would help me not have to go back and forth between two tools (scout & the database CLI). Bonus points if you have some kind of visualization. I've not played with it but this looks promising http://tatiyants.com/postgres-query-plan-visualization/.

itsderek23 commented 7 years ago

I'd love to see support for SQL Server (direct db adapter: https://github.com/rails-sqlserver/tiny_tds) as well! Any chance of adding that, or providing instructions on how to add a new database for PR purposes?

It's likely we'll provide the fundamentals for most db systems (it's basically pivoting our data on the metric type vs. a web endpoint or background job). Additional metrics (like cache hit rates) that are db-specific would likely be added based on db popularity.

When showing slow queries it would be helpful to show an example of what inputs that query takes. In Rails all queries will show with params placeholders ? and the actual expensive query might only trigger on a specific set of params.

Yes - that's a pet peeve of @dlanderson :).

I think it's mostly a security question: VividCortex, for example, does collect raw queries but encrypts the queries before writing to their DB.

New Relic defaults to obfuscated queries, but you can optionally capture queries in raw form (docs).

On that same note it could be cool if we could also get some kind of EXPLAIN output as well. Raw explain output would help me not have to go back and forth between two tools (scout & the database CLI).

Yes. One challenge w/the Heroku Expensive queries and pg_stat_statements is that the queries are obfuscated: you need a raw query to run an explain.

Auto-running EXPLAIN on a repeated slow query would be pretty helpful. We'd want to avoid using EXPLAIN ANALYZE:

Important: Keep in mind that the statement is actually executed when the ANALYZE option is used. Although EXPLAIN will discard any output that a SELECT would return, other side effects of the statement will happen as usual. If you wish to use EXPLAIN ANALYZE on an INSERT, UPDATE, DELETE, CREATE TABLE AS, or EXECUTE statement without letting the command affect your data, use this approach:

Bonus points if you have some kind of visualization. I've not played with it but this looks promising http://tatiyants.com/postgres-query-plan-visualization/.

Yes - that's a beautiful tool. The Postgres EXPLAIN output can be a quite a bit to digest.

Related: Dexter is pretty fascinating. Just being able to suggest to the user to create an index based on measuring potential performance would be an incredible time-saver. I haven't ran this against a database yet - would be interested to hear from folks that have.

itsderek23 commented 6 years ago

Completed: http://blog.scoutapp.com/articles/2017/11/04/scout-database-monitoring