search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.99k stars 8.24k forks source link

'Admin View' of save object relationships #8804

Closed LeeDr closed 3 years ago

LeeDr commented 8 years ago

Describe the feature: I've had a few occasions where it would have been very useful to be able to quickly see the hierarchy of saved objects in a dashboard. For example, I open a dashboard while logged in as a user who has access to logstash-* but not metricbeat-* and, as expected, I get an error about the metricbeat data. But which visualization in the dashboard is using metricbeat data? The visualization name doesn't always give it away.

What I'd like to see is something like this;

Everything in the table would be a link to jump directly to that view.

It might make sense to support sorting on the columns to make it easy to find, for example, that some visualization is used on 3 different dashboards.

It might also be useful to allow filtering at least on some columns. Maybe just on Dashbaords?

Dashboard Visualization Search Index
Packetbeat Cassandra Cassandra ResponseKeyspace Cassandra search packetbeat-*
Packetbeat Cassandra Cassandra ResponseType Cassandra search packetbeat-*
Packetbeat Cassandra Cassandra ResponseTime Cassandra search packetbeat-*
Packetbeat Cassandra Cassandra RequestCount Cassandra search packetbeat-*
Packetbeat Cassandra Cassandra Ops packetbeat-*
Packetbeat Cassandra Cassandra RequestCountStackByType packetbeat-*
Packetbeat Cassandra Cassandra ResponseCountStackByType packetbeat-*
Packetbeat Cassandra Cassandra RequestCountByType packetbeat-*
Packetbeat Cassandra Cassandra ResponseCountByType packetbeat-*
Packetbeat Cassandra Navigation
Packetbeat Cassandra Cassandra QueryView packetbeat-*
Packetbeat Thrift performance Navigation packetbeat-*
Packetbeat Thrift performance Thrift requests per minute packetbeat-*
Packetbeat Thrift performance Thrift RPC Errors packetbeat-*
Packetbeat Thrift performance Slowest Thrift RPC methods packetbeat-*
Packetbeat Thrift performance Thrift response times percentiles packetbeat-*
Top Thrift RPC methods packetbeat-*
Top Thrift RPC calls with errors packetbeat-*

I started a little experiment in the selenium test framework just because I'm getting familiar with the elasticsearch client there. My first attempt started with getting the dashboards and parsing out the panels (which could be visualizations or saved searches) and listing them out. 

    return esClient.search('.kibana', 'dashboard')
    .then((response) => {
      PageObjects.common.debug('found ' + response.hits.total + ' dashboards');

I parsed the visualization ids out of each dashboard object like this;

JSON.parse(response.hits.hits._source.panelsJSON).id

And then I tried doing a GET on each visualization.id. This was sort of working, but I wasn't getting the console output how I wanted and probably needs to be re-thought if it were to end up in Kibana or as a plugin.

I chatted with @cjcenizal about it and agreed it would be much more efficient to just do 3 searches, 1 for Dashboards, 1 for Visualizations, and 1 for Searches, and join those results afterwards.

cjcenizal commented 8 years ago

I think we should also explore alternative ways to visualize this data. For example, how about a parallel coordinates chart?

image

Interactive examples:

Another option is a Sankey diagram:

image

Interactive examples:

LeeDr commented 8 years ago

That looks pretty cool, but I'm not sure how well it would scale to a large number of visualizations. If it were interactive and mousing over a object would highlight it's line and dim the others like our other charts do that might help a lot. And/or if you could apply filtering on some columns like Dashboards so that you're only looking at a subset of the data it might be great.

LeeDr commented 8 years ago

I have a script that generates this table below (except the header).
Picture these each as a link to that Dashboard, Visualization, Saved Search, or Index pattern. The code does a search on .kibana for index-pattern, search, visualization, and dashboard and stores the result in a hashmap with the object id as the key. Then it iterates through dashboards, and through the panels of the dashboard.

  If the panel is a visualization
     if the visualization came from a saved search
        get the saved search, 
           get the index pattern
     if the visualization used a "new query" on the index pattern
          get the index pattern
     else (it was a markdown visualization)
  If the panel is a search
          get the index pattern

What's not included here, but probably should be listed, are Visualization and Saved Searches which are not used on any Dashboards. They should be listed at the end. I guess I'd have to set a flag on each Visualization and Saved Search object in the hashmap as it's used to know which ones aren't used. Same for Index patterns.

Dashboard Visualization Saved Search Index Pattern
Metricbeat-filesystem System Navigation
Metricbeat-filesystem Top hosts by disk size Fsstats metricbeat-*
Metricbeat-filesystem Disk space overview Fsstats metricbeat-*
Metricbeat-filesystem Free disk space over days Fsstats metricbeat-*
Metricbeat-filesystem Total files over days Fsstats metricbeat-*
Metricbeat-processes Top processes by memory usage Process stats metricbeat-*
Metricbeat-processes CPU usage per process Process stats metricbeat-*
Metricbeat-processes Memory usage per process Process stats metricbeat-*
Metricbeat-processes Top processes by CPU usage Process stats metricbeat-*
Metricbeat-processes System Navigation
Metricbeat-processes Number of Pids Process stats metricbeat-*
Metricbeat-processes Number of processes Process stats metricbeat-*
Packetbeat Cassandra Cassandra: ResponseKeyspace packetbeat-*
Packetbeat Cassandra Cassandra: ResponseType packetbeat-*
Packetbeat Cassandra Cassandra: ResponseTime packetbeat-*
Packetbeat Cassandra Cassandra: RequestCount packetbeat-*
Packetbeat Cassandra Cassandra: Ops packetbeat-*
Packetbeat Cassandra Cassandra: RequestCountStackByType packetbeat-*
Packetbeat Cassandra Cassandra: ResponseCountStackByType packetbeat-*
Packetbeat Cassandra Cassandra: RequestCountByType packetbeat-*
Packetbeat Cassandra Cassandra: ResponseCountByType packetbeat-*
Packetbeat Cassandra Navigation
Packetbeat Cassandra Cassandra: QueryView packetbeat-*
Packetbeat Thrift performance Navigation
Packetbeat Thrift performance Thrift requests per minute Thrift transactions packetbeat-*
Packetbeat Thrift performance Thrift RPC Errors Thrift errors packetbeat-*
Packetbeat Thrift performance Slowest Thrift RPC methods Thrift transactions packetbeat-*
Packetbeat Thrift performance Thrift response times percentiles Thrift transactions packetbeat-*
Packetbeat Thrift performance Top Thrift-RPC methods Thrift transactions packetbeat-*
Packetbeat Thrift performance Top Thrift-RPC calls with errors Thrift errors packetbeat-*
Metricbeat - Apache HTTPD server status Apache HTTPD - CPU Apache HTTPD metricbeat-*
Metricbeat - Apache HTTPD server status Apache HTTPD - Hostname list Apache HTTPD metricbeat-*
Metricbeat - Apache HTTPD server status Apache HTTPD - Load1/5/15 Apache HTTPD metricbeat-*
Metricbeat - Apache HTTPD server status Apache HTTPD - Scoreboard Apache HTTPD metricbeat-*
Metricbeat - Apache HTTPD server status Apache HTTPD - Total accesses and kbytes Apache HTTPD metricbeat-*
Metricbeat - Apache HTTPD server status Apache HTTPD - Uptime Apache HTTPD metricbeat-*
Metricbeat - Apache HTTPD server status Apache HTTPD - Workers Apache HTTPD metricbeat-*
Metricbeat-cpu System Navigation
Metricbeat-cpu CPU usage over time Cpu stats metricbeat-*
Metricbeat-cpu System Load Load stats metricbeat-*
Metricbeat-cpu System Load over time Load stats metricbeat-*
Metricbeat-cpu Top hosts by CPU usage Cpu-Load stats metricbeat-*
Metricbeat-cpu CPU Usage Cpu stats metricbeat-*
Packetbeat HTTP Web transactions Web transactions packetbeat-*
Packetbeat HTTP HTTP error codes packetbeat-*
Packetbeat HTTP HTTP error codes evolution packetbeat-*
Packetbeat HTTP Navigation
Packetbeat HTTP Total number of HTTP transactions Web transactions packetbeat-*
Packetbeat HTTP HTTP codes for the top queries Web transactions packetbeat-*
Packetbeat HTTP Top 10 HTTP requests Web transactions packetbeat-*
Winlogbeat Dashboard Number of Events Over Time By Event Log winlogbeat-*
Winlogbeat Dashboard Number of Events winlogbeat-*
Winlogbeat Dashboard Top Event IDs winlogbeat-*
Winlogbeat Dashboard Event Levels winlogbeat-*
Winlogbeat Dashboard Event Sources winlogbeat-*
Metricbeat-memory System Navigation
Metricbeat-memory Top hosts by memory usage Memory stats metricbeat-*
Metricbeat-memory Memory usage over time Memory stats metricbeat-*
Metricbeat-memory Swap usage over time Memory stats metricbeat-*
Metricbeat-memory Total Memory Memory stats metricbeat-*
Metricbeat-memory Available Memory Memory stats metricbeat-*
Metricbeat-memory Memory usage Memory stats metricbeat-*
Metricbeat-memory Swap usage Memory stats metricbeat-*
Metricbeat-overview Servers overview System stats metricbeat-*
Metricbeat-overview System Navigation
Packetbeat Dashboard Web transactions Web transactions packetbeat-*
Packetbeat Dashboard DB transactions DB transactions packetbeat-*
Packetbeat Dashboard Cache transactions Cache transactions packetbeat-*
Packetbeat Dashboard RPC transactions RPC transactions packetbeat-*
Packetbeat Dashboard Response times percentiles Packetbeat Search packetbeat-*
Packetbeat Dashboard Errors count over time Transactions errors packetbeat-*
Packetbeat Dashboard Errors vs successful transactions Packetbeat Search packetbeat-*
Packetbeat Dashboard Latency histogram Packetbeat Search packetbeat-*
Packetbeat Dashboard Client locations Packetbeat Search packetbeat-*
Packetbeat Dashboard Response times repartition Packetbeat Search packetbeat-*
Packetbeat Dashboard Navigation
Packetbeat MongoDB performance Navigation
Packetbeat MongoDB performance MongoDB errors MongoDB errors packetbeat-*
Packetbeat MongoDB performance MongoDB commands MongoDB transactions packetbeat-*
Packetbeat MongoDB performance MongoDB errors per collection MongoDB errors packetbeat-*
Packetbeat MongoDB performance MongoDB in/out throughput MongoDB transactions packetbeat-*
Packetbeat MongoDB performance MongoDB response times by collection MongoDB transactions packetbeat-*
Packetbeat MongoDB performance Top slowest MongoDB queries MongoDB transactions packetbeat-*
Packetbeat MongoDB performance Number of MongoDB transactions with writeConcern w=0 MongoDB transactions with write concern 0 packetbeat-*
Packetbeat MySQL performance MySQL Errors MySQL errors packetbeat-*
Packetbeat MySQL performance MySQL Methods MySQL Transactions packetbeat-*
Packetbeat MySQL performance Navigation
Packetbeat MySQL performance MySQL throughput MySQL Transactions packetbeat-*
Packetbeat MySQL performance Most frequent MySQL queries MySQL Transactions packetbeat-*
Packetbeat MySQL performance Slowest MySQL queries MySQL Transactions packetbeat-*
Packetbeat MySQL performance Mysql response times percentiles MySQL Transactions packetbeat-*
Packetbeat MySQL performance MySQL Reads vs Writes MySQL Transactions packetbeat-*
Packetbeat NFS NFS clients pie chart nfs packetbeat-*
Packetbeat NFS NFS operations area chart nfs packetbeat-*
Packetbeat NFS NFS top group pie chart nfs packetbeat-*
Packetbeat NFS NFS top users pie chart nfs packetbeat-*
Packetbeat NFS NFS response times nfs packetbeat-*
Packetbeat NFS NFS errors nfs packetbeat-*
Packetbeat NFS NFS operation table nfs packetbeat-*
Packetbeat NFS NFS bytes in / out nfs packetbeat-*
Packetbeat NFS Navigation
Metricbeat system overview Network Bytes Network data metricbeat-*
Metricbeat system overview Network Packetloss Network data metricbeat-*
Metricbeat system overview System Navigation
Metricbeat system overview Total Memory Memory stats metricbeat-*
Metricbeat system overview Available Memory Memory stats metricbeat-*
Metricbeat system overview System overview by host System stats metricbeat-*
Metricbeat system overview System Load Load stats metricbeat-*
Metricbeat system overview CPU Usage Cpu stats metricbeat-*
Metricbeat filesystem per Host Top disks by memory usage Filesystem stats metricbeat-*
Metricbeat filesystem per Host Disk utilization over time Filesystem stats metricbeat-*
Metricbeat filesystem per Host System Navigation
Metricbeat filesystem per Host Disk space distribution Filesystem stats metricbeat-*
Metricbeat-network In vs Out Network Bytes Network data metricbeat-*
Metricbeat-network Top 10 interfaces Network data metricbeat-*
Metricbeat-network Network Packetloss Network data metricbeat-*
Metricbeat-network Packet loss on interfaces Network data metricbeat-*
Metricbeat-network System Navigation
Metricbeat-network Network Bytes Network data metricbeat-*
Packetbeat PgSQL performance Navigation
Packetbeat PgSQL performance PgSQL Errors PgSQL errors packetbeat-*
Packetbeat PgSQL performance PgSQL Methods PgSQL transactions packetbeat-*
Packetbeat PgSQL performance PgSQL response times percentiles PgSQL transactions packetbeat-*
Packetbeat PgSQL performance PgSQL throughput PgSQL transactions packetbeat-*
Packetbeat PgSQL performance PgSQL Reads vs Writes PgSQL transactions packetbeat-*
Packetbeat PgSQL performance Most frequent PgSQL queries PgSQL transactions packetbeat-*
Packetbeat PgSQL performance Slowest PgSQL queries PgSQL transactions packetbeat-*

cc @rasroh

LeeDr commented 8 years ago

Some crappy code that generated the table above;

import {
  bdd,
  esClient
} from '../../support';

import PageObjects from '../../support/page_objects';

var expect = require('expect.js');

bdd.describe('dashboard sources', function () {
  const watchId = 'cluster_health_watch_' + new Date().getTime();
  const kbnInternVars = global.__kibana__intern__;
  const config = kbnInternVars.intern.config;

  bdd.it('should show the each dashboard with its visualizations, searches, and indexes', function () {
    var dashMap = {};
    var visMap = {};
    var searchMap = {};
    var indexMap = {};
    var w1 = 40;
    var w2 = 55;
    var w3 = 50;
    var w4 = 20;

    return esClient.search('.kibana', 'index-pattern')
    .then((response) => {

      function getIndexSource(index) {
        indexMap[index._id] = index._source;
      }

      var getChartTypesPromises4 = response.hits.hits.map(getIndexSource);
      return Promise.all(getChartTypesPromises4)
      .then((response) => {
        return esClient.search('.kibana', 'visualization')
        .then((response) => {

          function getVisualizationSource(vis) {
            visMap[vis._id] = vis._source;
          }

          var getChartTypesPromises2 = response.hits.hits.map(getVisualizationSource);
          return Promise.all(getChartTypesPromises2);
        });
      })
      .then((response) => {
        return esClient.search('.kibana', 'search')
        .then((response) => {

          function getSearchSource(search) {
            searchMap[search._id] = search._source;
          }
          var getChartTypesPromises3 = response.hits.hits.map(getSearchSource);
          return Promise.all(getChartTypesPromises3);
        })
        .then((response) => {
          return esClient.search('.kibana', 'dashboard')
          .then((response) => {

            function getDashboardSource(dashboard) {
              dashMap[dashboard._id] = dashboard._source;
              // console.log('| ' + formatString('*Dashboard*',w1) + ' | '
              //   + formatString('*Visualization*', w2) + ' | '
              //   + formatString('*Search*', w3) + ' | '
              //   + formatString('*Index Pattern*', w4) + ' | ');

              function printVis(vis) {
                if (vis.type === 'visualization') {
                  if (visMap[vis.id].savedSearchId) {
                    console.log('| ' + formatString(dashboard._source.title,w1) + ' | '
                      + formatString(visMap[vis.id].title, w2) + ' | '
                      + formatString(searchMap[visMap[vis.id].savedSearchId].title, w3) + ' | '
                      + formatString(JSON.parse(searchMap[visMap[vis.id].savedSearchId].kibanaSavedObjectMeta.searchSourceJSON).index, w4) + ' | ');
                  } else if (JSON.parse(visMap[vis.id].kibanaSavedObjectMeta.searchSourceJSON).index) {
                    // it's built directly on an index, or is Markdown
                    console.log('| ' + formatString(dashboard._source.title,w1) + ' | '
                      + formatString(visMap[vis.id].title, w2) + ' | '
                      + formatString('      ', w3) + ' | '
                      + formatString(JSON.parse(visMap[vis.id].kibanaSavedObjectMeta.searchSourceJSON).index, w4) + ' | ');
                  } else {
                    // markdown ?
                    console.log('| ' + formatString(dashboard._source.title,w1) + ' | '
                      + formatString(visMap[vis.id].title, w2) + ' | '
                      + formatString('      ', w3) + ' | '
                      + formatString('      ', w4) + ' | ');
                  }
                } else if (vis.type === 'search') {
                  console.log('| ' + formatString(dashboard._source.title,w1) + ' | '
                  + formatString('      ', w2) + ' | '
                    + formatString(searchMap[vis.id].title, w3) + ' | '
                    + formatString(JSON.parse(searchMap[vis.id].kibanaSavedObjectMeta.searchSourceJSON).index, w4) + ' | ');
                }
              }

              var vis = JSON.parse(dashboard._source.panelsJSON).map(printVis);
              return Promise.all(vis);

            }

            var getChartTypesPromises = response.hits.hits.map(getDashboardSource);
            return Promise.all(getChartTypesPromises);
          });
        });

      });
    });

  });

  function formatString(str, width) {
    return (str + '                                                    ').substring(0, width);
  }

});
LeeDr commented 6 years ago

We now have a relationship API that should make this easier?

elasticmachine commented 6 years ago

Pinging @elastic/kibana-platform